dbader.org - Pythonhttps://dbader.org/blog/tags/pythonenTue, 27 Jun 2017 00:00:00 GMTTue, 27 Jun 2017 17:51:49 GMTPyRSS2Gen-1.1.0http://blogs.law.harvard.edu/tech/rsshttps://dbader.org/logodbader.orghttps://dbader.org/Linked Lists in Pythonhttps://dbader.org/blog/python-linked-list<article><header><h1>Linked Lists in Python</h1> <p>Learn how to implement a linked list data structure in Python, using only built-in data types and functionality from the standard library.</p> </header><section><figure><img alt="Python Linked Lists" src="/blog/figures/python-linked-lists.png" width="2560" height="1440"></figure> <p>Every Python programmer should know about linked lists:</p> <p>They are among the simplest and most common data structures used in programming.</p> <p>So, if you ever found yourself wondering, <strong>“Does Python have a built-in or ‘native’ linked list data structure?”</strong> or, <strong>“How do I write a linked list in Python?”</strong> then this tutorial will help you.</p> <p>Python doesn’t ship with a built-in linked list data type in the “classical” sense. Python’s <code>list</code> type is implemented as a dynamic array—which means it doesn’t suit the typical scenarios where you’d want to use a “proper” linked list data structure for performance reasons.</p> <p>Please note that this tutorial only considers linked list implementations that work on a “plain vanilla” Python install. I’m leaving out third-party packages intentionally. They don’t apply during coding interviews and it’s difficult to keep an up-to-date list that considers all packages available on Python packaging repositories.</p> <p>Before we get into the weeds and look at linked list implementations in Python, let’s do a quick recap of what a linked list data structure is—and how it compares to an array.</p> </section><section><h2>What are the characteristics of a linked list?</h2> <p>A linked list is an ordered collection of values. Linked lists are similar to arrays in the sense that they contain objects in a linear order. However they differ from arrays in their memory layout.</p> <p><strong>Arrays</strong> are <em>contiguous</em> data structures and they’re composed of fixed-size data records stored in adjoining blocks of memory. In an array, data is tightly packed—and we know the size of each data record which allows us to quickly look up an element given its index in the array:</p> <figure><img alt="Array Visualization" src="/blog/figures/python-linked-list-array-visualization.jpeg" width="1920" height="1080"></figure> <p><strong>Linked lists</strong>, however, are made up of data records <em>linked</em> together by pointers. This means that the data records that hold the actual “data payload” can be stored anywhere in memory—what creates the linear ordering is how each data record “points” to the next one:</p> <figure><img alt="Linked List Visualization" src="/blog/figures/python-linked-list-visualization.jpeg" width="1920" height="1080"></figure> <p>There are two different kinds of linked lists: <strong>singly-linked lists</strong> and <strong>doubly-linked lists</strong>. What you saw in the previous example was a singly-linked list—each element in it has a reference to (a “pointer”) to the <em>next</em> element in the list.</p> <p>In a doubly-linked list, each element has a reference to both the <em>next and the previous</em> element. Why is this useful? Having a reference to the previous element can speed up some operations, like removing (“unlinking”) an element from a list or traversing the list in reverse order.</p> </section><section><h2>How do linked lists and arrays compare performance-wise?</h2> <p>You just saw how linked lists and arrays use different data layouts behind the scenes to store information. This data layout difference reflects in the performance characteristics of linked lists and arrays:</p> <ul> <li><strong>Element Insertion &amp; Removal</strong>: Inserting and removing elements from a (doubly) linked list has time complexity <em>O(1)</em>, whereas doing the same on an array requires an <em>O(n)</em> copy operation in the worst case. On a linked list we can simply “hook in” a new element anywhere we want by adjusting the pointers from one data record to the next. On an array we have to allocate a bigger storage area first and copy around the existing elements, leaving a blank space to insert the new element into.</li> <li><strong>Element Lookup</strong>: Similarly, looking up an element given its index is a slow <em>O(n)</em> time operation on a linked list—but a fast <em>O(1)</em> lookup on an array. With a linked list we must jump from element to element and search the structure from the “head” of the list to find the index we want. But with an array we can calculate the exact address of an element in memory based on its index and the (fixed) size of each data record.</li> <li><strong>Memory Efficiency</strong>: Because the data stored in arrays is tightly packed they’re generally more space-efficient than linked lists. This mostly applies to static arrays, however. <a href="https://en.wikipedia.org/wiki/Dynamic_array" target="_blank">Dynamic arrays</a> typically over-allocate their backing store slightly to speed up element insertions in the average case, thus increasing the memory footprint.</li> </ul> <p>Now, how does this performance difference come into play with Python? Remember that <a href="http://www.laurentluce.com/posts/python-list-implementation/" target="_blank">Python’s built-in <code>list</code> type is in fact a dynamic array</a>. This means the performance differences we just discussed apply to it. Likewise, Python’s immutable <code>tuple</code> data type can be considered a static array in this case—with similar performance trade-offs compared to a proper linked list.</p> </section><section><h2>Does Python have a built-in or “native” linked list data structure?</h2> <p>Let’s come back to the original question. If you want to use a linked list in Python, is there a built-in data type you can use directly?</p> <p>The answer is: “It depends.”</p> <p>As of Python 3.6 (CPython), doesn’t provide a dedicated linked list data type. There’s nothing like <a href="https://docs.oracle.com/javase/7/docs/api/java/util/LinkedList.html" target="_blank">Java’s <code>LinkedList</code></a> built into Python or into the Python standard library.</p> <p>Python does however include the <code>collections.deque</code> class which provides a <a href="https://dbader.org/blog/queues-in-python">double-ended queue</a> and is implemented as a doubly-linked list internally. Under some specific circumstances you might be able to use it as a “makeshift” linked list. If that’s not an option you’ll need to write your own linked list implementation from scratch.</p> </section><section><h2>How do I write a linked list using Python?</h2> <p>If you want to stick with functionality built into the core language and into the Python standard library you have two options for implementing a linked list:</p> <ul> <li> <p>You could either use the <code>collections.deque</code> class from the Python standard library and take advantage of the fact that it’s implemented as a doubly-linked list behind the scenes. But this will only work for some use cases—I’ll go into more details on that further down in the article.</p> </li> <li> <p>Alternatively, you could define your own linked list type in Python by writing it from scratch using other built-in data types. You’d implement your own custom linked list class or base your implementation of Lisp-style chains of <code>tuple</code> objects. Again, see below for more details.</p> </li> </ul> <p>Now that we’ve covered some general questions on linked lists and their availability in Python, read on for examples of how to make both of the above approaches work.</p> </section><section><h2>Option 1: Using <code>collections.deque</code> as a Linked List</h2> <p>This approach might seem a little odd at first because the <code>collections.deque</code> class implements a double-ended queue, and it’s typically used as the go-to <a href="https://dbader.org/blog/stacks-in-python">stack</a> or <a href="https://dbader.org/blog/queues-in-python">queue implementation in Python</a>.</p> <p>But using this class as a “makeshift” linked list might make sense under some circumstances. You see, CPython’s <code>deque</code> is <a href="https://github.com/python/cpython/blob/947629916a5ecb1f6f6792e9b9234e084c5bf274/Modules/_collectionsmodule.c#L24-L26" target="_blank">powered by a doubly-linked list</a> behind the scenes and it provides a full “list-like” set of functionality.</p> <p>Under <em>some</em> circumstances, this makes treating <code>deque</code> objects as linked list replacements a valid option. Here are some of the key performance characteristics of this approach:</p> <ul> <li><strong>Inserting and removing elements at the front and back of a <code>deque</code> is a fast <em>O(1)</em> operation.</strong> However, inserting or removing in the middle takes <em>O(n)</em> time because we don’t have access to the previous-element or next-element linked list pointers. That’s abstracted away by the <code>deque</code> interface.</li> <li><strong>Storage is <em>O(n)</em>—but not every element gets its own list node.</strong> The <code>deque</code> class uses blocks that hold multiple elements at once and then these blocks are linked together as a doubly-linked list. <a href="https://github.com/python/cpython/blob/947629916a5ecb1f6f6792e9b9234e084c5bf274/Modules/_collectionsmodule.c#L14-L22" target="_blank">As of CPython 3.6 the block size is 64 elements.</a> This incurs some space overhead but retains the general performance characteristics given a large enough number of elements.</li> <li><strong>In-place reversal</strong>: In Python 3.2+ the elements in a <code>deque</code> instance can be reversed in-place with the <code>reverse()</code> method. This takes <em>O(n)</em> time and no extra space.</li> </ul> <p>Using <code>collections.deque</code> as a linked list in Python can be a valid choice if you mostly care about insertion performance at the beginning or the end of the list, and you don’t need access to the previous-element and next-element pointers on each object directly.</p> <p>Don’t use a <code>deque</code> if you need <em>O(1)</em> performance when removing elements. Removing elements by key or by index requires an <em>O(n)</em> search, even if you have already have a reference to the element to be removed. This is the main downside of using a <code>deque</code> like a linked list.</p> <p>If you’re looking for a linked list in Python because you want to implement <a href="https://dbader.org/blog/queues-in-python">queues</a> or a <a href="https://dbader.org/blog/stacks-in-python">stacks</a> then a <code>deque</code> is a great choice, however.</p> <p>Here are some examples on how you can use Python’s <code>deque</code> class as a replacement for a linked list:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">collections</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span> <span class="o">=</span> <span class="n">collections</span><span class="o">.</span><span class="n">deque</span><span class="p">()</span> <span class="c1"># Inserting elements at the front</span> <span class="c1"># or back takes O(1) time:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'B'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'C'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">appendleft</span><span class="p">(</span><span class="s1">'A'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span> <span class="n">deque</span><span class="p">([</span><span class="s1">'A'</span><span class="p">,</span> <span class="s1">'B'</span><span class="p">,</span> <span class="s1">'C'</span><span class="p">])</span> <span class="c1"># However, inserting elements at</span> <span class="c1"># arbitrary indexes takes O(n) time:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'X'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span> <span class="n">deque</span><span class="p">([</span><span class="s1">'A'</span><span class="p">,</span> <span class="s1">'B'</span><span class="p">,</span> <span class="s1">'X'</span><span class="p">,</span> <span class="s1">'C'</span><span class="p">])</span> <span class="c1"># Removing elements at the front</span> <span class="c1"># or back takes O(1) time:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="s1">'C'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">popleft</span><span class="p">()</span> <span class="s1">'A'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span> <span class="n">deque</span><span class="p">([</span><span class="s1">'B'</span><span class="p">,</span> <span class="s1">'X'</span><span class="p">])</span> <span class="c1"># Removing elements at arbitrary</span> <span class="c1"># indexes or by key takes O(n) time again:</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">del</span> <span class="n">lst</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="s1">'B'</span><span class="p">)</span> <span class="c1"># Deques can be reversed in-place:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span> <span class="o">=</span> <span class="n">deque</span><span class="p">([</span><span class="s1">'A'</span><span class="p">,</span> <span class="s1">'B'</span><span class="p">,</span> <span class="s1">'X'</span><span class="p">,</span> <span class="s1">'C'</span><span class="p">])</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">reverse</span><span class="p">()</span> <span class="n">deque</span><span class="p">([</span><span class="s1">'C'</span><span class="p">,</span> <span class="s1">'X'</span><span class="p">,</span> <span class="s1">'B'</span><span class="p">,</span> <span class="s1">'A'</span><span class="p">])</span> <span class="c1"># Searching for elements takes</span> <span class="c1"># O(n) time:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="s1">'X'</span><span class="p">)</span> <span class="mi">1</span> </pre></div> </section><section><h2>Option 2: Writing Your Own Python Linked Lists</h2> <p>If you need full control over the layout of each linked list node then there’s no perfect solution available in the Python standard library. If you want to stick with the standard library and built-in data types then writing your own linked list is your best bet.</p> <p>You’ll have to make a choice between implementing a singly-linked or a doubly-linked list. I’ll give examples of both, including some of the common operations like how to search for elements, or how to reverse a linked list.</p> <p>Let’s take a look at two concrete Python linked list examples. One for a singly-linked list, and one for a double-linked list.</p> </section><section><h2>✅ A Singly-Linked List Class in Python</h2> <p>Here’s how you might implement <strong>a class-based singly-linked list in Python</strong>, including some of the standard algorithms:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">ListNode</span><span class="p">:</span> <span class="sd">"""</span> <span class="sd"> A node in a singly-linked list.</span> <span class="sd"> """</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="nb">next</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">data</span> <span class="o">=</span> <span class="n">data</span> <span class="bp">self</span><span class="o">.</span><span class="n">next</span> <span class="o">=</span> <span class="nb">next</span> <span class="k">def</span> <span class="fm">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="nb">repr</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">)</span> <span class="k">class</span> <span class="nc">SinglyLinkedList</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Create a new singly-linked list.</span> <span class="sd"> Takes O(1) time.</span> <span class="sd"> """</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="o">=</span> <span class="bp">None</span> <span class="k">def</span> <span class="fm">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Return a string representation of the list.</span> <span class="sd"> Takes O(n) time.</span> <span class="sd"> """</span> <span class="n">nodes</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">curr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="k">while</span> <span class="n">curr</span><span class="p">:</span> <span class="n">nodes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">repr</span><span class="p">(</span><span class="n">curr</span><span class="p">))</span> <span class="n">curr</span> <span class="o">=</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span> <span class="k">return</span> <span class="s1">'['</span> <span class="o">+</span> <span class="s1">', '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">nodes</span><span class="p">)</span> <span class="o">+</span> <span class="s1">']'</span> <span class="k">def</span> <span class="nf">prepend</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Insert a new element at the beginning of the list.</span> <span class="sd"> Takes O(1) time.</span> <span class="sd"> """</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="o">=</span> <span class="n">ListNode</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">data</span><span class="p">,</span> <span class="nb">next</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">head</span><span class="p">)</span> <span class="k">def</span> <span class="nf">append</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Insert a new element at the end of the list.</span> <span class="sd"> Takes O(n) time.</span> <span class="sd"> """</span> <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="o">=</span> <span class="n">ListNode</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">data</span><span class="p">)</span> <span class="k">return</span> <span class="n">curr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="k">while</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span><span class="p">:</span> <span class="n">curr</span> <span class="o">=</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span> <span class="o">=</span> <span class="n">ListNode</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">data</span><span class="p">)</span> <span class="k">def</span> <span class="nf">find</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Search for the first element with `data` matching</span> <span class="sd"> `key`. Return the element or `None` if not found.</span> <span class="sd"> Takes O(n) time.</span> <span class="sd"> """</span> <span class="n">curr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="k">while</span> <span class="n">curr</span> <span class="ow">and</span> <span class="n">curr</span><span class="o">.</span><span class="n">data</span> <span class="o">!=</span> <span class="n">key</span><span class="p">:</span> <span class="n">curr</span> <span class="o">=</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span> <span class="k">return</span> <span class="n">curr</span> <span class="c1"># Will be None if not found</span> <span class="k">def</span> <span class="nf">remove</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Remove the first occurence of `key` in the list.</span> <span class="sd"> Takes O(n) time.</span> <span class="sd"> """</span> <span class="c1"># Find the element and keep a</span> <span class="c1"># reference to the element preceding it</span> <span class="n">curr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="n">prev</span> <span class="o">=</span> <span class="bp">None</span> <span class="k">while</span> <span class="n">curr</span> <span class="ow">and</span> <span class="n">curr</span><span class="o">.</span><span class="n">data</span> <span class="o">!=</span> <span class="n">key</span><span class="p">:</span> <span class="n">prev</span> <span class="o">=</span> <span class="n">curr</span> <span class="n">curr</span> <span class="o">=</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span> <span class="k">if</span> <span class="n">curr</span><span class="p">:</span> <span class="c1"># Unlink it from the list</span> <span class="n">prev</span><span class="o">.</span><span class="n">next</span> <span class="o">=</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span> <span class="o">=</span> <span class="bp">None</span> <span class="k">def</span> <span class="nf">reverse</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Reverse the list in-place.</span> <span class="sd"> Takes O(n) time.</span> <span class="sd"> """</span> <span class="n">curr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="n">prev_node</span> <span class="o">=</span> <span class="bp">None</span> <span class="n">next_node</span> <span class="o">=</span> <span class="bp">None</span> <span class="k">while</span> <span class="n">curr</span><span class="p">:</span> <span class="n">next_node</span> <span class="o">=</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span> <span class="o">=</span> <span class="n">prev_node</span> <span class="n">prev_node</span> <span class="o">=</span> <span class="n">curr</span> <span class="n">curr</span> <span class="o">=</span> <span class="n">next_node</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="o">=</span> <span class="n">prev_node</span> </pre></div> <p>And here’s how you’d use this linked list class in practice:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span> <span class="o">=</span> <span class="n">SinglyLinkedList</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span> <span class="p">[]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">prepend</span><span class="p">(</span><span class="mi">23</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">prepend</span><span class="p">(</span><span class="s1">'a'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">prepend</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">prepend</span><span class="p">(</span><span class="s1">'X'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'the'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'end'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span> <span class="p">[</span><span class="s1">'X'</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'a'</span><span class="p">,</span> <span class="mi">23</span><span class="p">,</span> <span class="s1">'the'</span><span class="p">,</span> <span class="s1">'end'</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'X'</span><span class="p">)</span> <span class="s1">'X'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'y'</span><span class="p">)</span> <span class="bp">None</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">reverse</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span> <span class="p">[</span><span class="s1">'end'</span><span class="p">,</span> <span class="s1">'the'</span><span class="p">,</span> <span class="mi">23</span><span class="p">,</span> <span class="s1">'a'</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'X'</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span> <span class="p">[</span><span class="s1">'end'</span><span class="p">,</span> <span class="s1">'the'</span><span class="p">,</span> <span class="mi">23</span><span class="p">,</span> <span class="s1">'a'</span><span class="p">,</span> <span class="s1">'X'</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="s1">'not found'</span><span class="p">)</span> </pre></div> <p>Note that removing an element in this implementation is still an <em>O(n)</em> time operation, even if you already have a reference to a <code>ListNode</code> object.</p> <p>In a single linked list removing an element typically requires searching the list because we need to know the previous and the next element. With a double-linked list you could write a <code>remove_elem()</code> method that unlinks and removes a node from the list in <em>O(1)</em> time.</p> </section><section><h2>✅ A Doubly-Linked List Class in Python</h2> <p>Let’s have a look at <strong>how to implement a doubly-linked list in Python</strong>. The following <code>DoublyLinkedList</code> class should point you in the right direction:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">DListNode</span><span class="p">:</span> <span class="sd">"""</span> <span class="sd"> A node in a doubly-linked list.</span> <span class="sd"> """</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">prev</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="nb">next</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">data</span> <span class="o">=</span> <span class="n">data</span> <span class="bp">self</span><span class="o">.</span><span class="n">prev</span> <span class="o">=</span> <span class="n">prev</span> <span class="bp">self</span><span class="o">.</span><span class="n">next</span> <span class="o">=</span> <span class="nb">next</span> <span class="k">def</span> <span class="fm">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="nb">repr</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">)</span> <span class="k">class</span> <span class="nc">DoublyLinkedList</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Create a new singly linked list.</span> <span class="sd"> Takes O(1) time.</span> <span class="sd"> """</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="o">=</span> <span class="bp">None</span> <span class="k">def</span> <span class="fm">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Return a string representation of the list.</span> <span class="sd"> Takes O(n) time.</span> <span class="sd"> """</span> <span class="n">nodes</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">curr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="k">while</span> <span class="n">curr</span><span class="p">:</span> <span class="n">nodes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">repr</span><span class="p">(</span><span class="n">curr</span><span class="p">))</span> <span class="n">curr</span> <span class="o">=</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span> <span class="k">return</span> <span class="s1">'['</span> <span class="o">+</span> <span class="s1">', '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">nodes</span><span class="p">)</span> <span class="o">+</span> <span class="s1">']'</span> <span class="k">def</span> <span class="nf">prepend</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Insert a new element at the beginning of the list.</span> <span class="sd"> Takes O(1) time.</span> <span class="sd"> """</span> <span class="n">new_head</span> <span class="o">=</span> <span class="n">DListNode</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">data</span><span class="p">,</span> <span class="nb">next</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">head</span><span class="p">)</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span><span class="o">.</span><span class="n">prev</span> <span class="o">=</span> <span class="n">new_head</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="o">=</span> <span class="n">new_head</span> <span class="k">def</span> <span class="nf">append</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Insert a new element at the end of the list.</span> <span class="sd"> Takes O(n) time.</span> <span class="sd"> """</span> <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="o">=</span> <span class="n">DListNode</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">data</span><span class="p">)</span> <span class="k">return</span> <span class="n">curr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="k">while</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span><span class="p">:</span> <span class="n">curr</span> <span class="o">=</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span> <span class="o">=</span> <span class="n">DListNode</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">data</span><span class="p">,</span> <span class="n">prev</span><span class="o">=</span><span class="n">curr</span><span class="p">)</span> <span class="k">def</span> <span class="nf">find</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Search for the first element with `data` matching</span> <span class="sd"> `key`. Return the element or `None` if not found.</span> <span class="sd"> Takes O(n) time.</span> <span class="sd"> """</span> <span class="n">curr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="k">while</span> <span class="n">curr</span> <span class="ow">and</span> <span class="n">curr</span><span class="o">.</span><span class="n">data</span> <span class="o">!=</span> <span class="n">key</span><span class="p">:</span> <span class="n">curr</span> <span class="o">=</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span> <span class="k">return</span> <span class="n">curr</span> <span class="c1"># Will be None if not found</span> <span class="k">def</span> <span class="nf">remove_elem</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">node</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Unlink an element from the list.</span> <span class="sd"> Takes O(1) time.</span> <span class="sd"> """</span> <span class="k">if</span> <span class="n">node</span><span class="o">.</span><span class="n">prev</span><span class="p">:</span> <span class="n">node</span><span class="o">.</span><span class="n">prev</span><span class="o">.</span><span class="n">next</span> <span class="o">=</span> <span class="n">node</span><span class="o">.</span><span class="n">next</span> <span class="k">if</span> <span class="n">node</span><span class="o">.</span><span class="n">next</span><span class="p">:</span> <span class="n">node</span><span class="o">.</span><span class="n">next</span><span class="o">.</span><span class="n">prev</span> <span class="o">=</span> <span class="n">node</span><span class="o">.</span><span class="n">prev</span> <span class="k">if</span> <span class="n">node</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="o">=</span> <span class="n">node</span><span class="o">.</span><span class="n">next</span> <span class="n">node</span><span class="o">.</span><span class="n">prev</span> <span class="o">=</span> <span class="bp">None</span> <span class="n">node</span><span class="o">.</span><span class="n">next</span> <span class="o">=</span> <span class="bp">None</span> <span class="k">def</span> <span class="nf">remove</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Remove the first occurence of `key` in the list.</span> <span class="sd"> Takes O(n) time.</span> <span class="sd"> """</span> <span class="n">elem</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">elem</span><span class="p">:</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">remove_elem</span><span class="p">(</span><span class="n">elem</span><span class="p">)</span> <span class="k">def</span> <span class="nf">reverse</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Reverse the list in-place.</span> <span class="sd"> Takes O(n) time.</span> <span class="sd"> """</span> <span class="n">curr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="n">prev_node</span> <span class="o">=</span> <span class="bp">None</span> <span class="k">while</span> <span class="n">curr</span><span class="p">:</span> <span class="n">prev_node</span> <span class="o">=</span> <span class="n">curr</span><span class="o">.</span><span class="n">prev</span> <span class="n">curr</span><span class="o">.</span><span class="n">prev</span> <span class="o">=</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span> <span class="n">curr</span><span class="o">.</span><span class="n">next</span> <span class="o">=</span> <span class="n">prev_node</span> <span class="n">curr</span> <span class="o">=</span> <span class="n">curr</span><span class="o">.</span><span class="n">prev</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span> <span class="o">=</span> <span class="n">prev_node</span><span class="o">.</span><span class="n">prev</span> </pre></div> <p>Here are a few examples on how to use this class. Notice how we can now remove elements in <em>O(1)</em> time with the <code>remove_elem()</code> function if we already hold a reference to the list node representing the element:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span> <span class="o">=</span> <span class="n">DoublyLinkedList</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span> <span class="p">[]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">prepend</span><span class="p">(</span><span class="mi">23</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">prepend</span><span class="p">(</span><span class="s1">'a'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">prepend</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">prepend</span><span class="p">(</span><span class="s1">'X'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'the'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'end'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span> <span class="p">[</span><span class="s1">'X'</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'a'</span><span class="p">,</span> <span class="mi">23</span><span class="p">,</span> <span class="s1">'the'</span><span class="p">,</span> <span class="s1">'end'</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'X'</span><span class="p">)</span> <span class="s1">'X'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'y'</span><span class="p">)</span> <span class="bp">None</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">reverse</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span> <span class="p">[</span><span class="s1">'end'</span><span class="p">,</span> <span class="s1">'the'</span><span class="p">,</span> <span class="mi">23</span><span class="p">,</span> <span class="s1">'a'</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'X'</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">elem</span> <span class="o">=</span> <span class="n">lst</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">remove_elem</span><span class="p">(</span><span class="n">elem</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="s1">'X'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="s1">'not found'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">lst</span> <span class="p">[</span><span class="s1">'end'</span><span class="p">,</span> <span class="s1">'the'</span><span class="p">,</span> <span class="mi">23</span><span class="p">,</span> <span class="s1">'a'</span><span class="p">]</span> </pre></div> <p>Both example for Python linked lists you saw here were class-based. An alternative approach would be to implement a <em>Lisp-style</em> linked list in Python using tuples as the core building blocks (“cons pairs”). Here’s a tutorial that goes into more detail: <a href="https://dbader.org/blog/functional-linked-lists-in-python">Functional Linked Lists in Python</a>.</p> </section><section><h2>Python Linked Lists: Recap &amp; Recommendations</h2> <p>We just looked at a number of approaches to implement a singly- and doubly-linked list in Python. You also saw some code examples of the standard operations and algorithms, for example how to reverse a linked list in-place.</p> <p>You should only consider using a linked list in Python when you’ve determined that you absolutely need a linked data structure for performance reasons (or you’ve been asked to use one in a coding interview.)</p> <p>In many cases the same algorithm implemented on top of Python’s highly optimized <code>list</code> objects will be sufficiently fast. If you know a dynamic array won’t cut it and you need a linked list, then check first if you can take advantage of Python’s built-in <code>deque</code> class.</p> <p>If none of these options work for you, and you want to stay within the standard library, only then should you write your own Python linked list.</p> <p>In an interview situation I’d also advise you to write your own implementation from scratch because that’s usually what the interviewer wants to see. However it can be beneficial to mention that <code>collections.deque</code> offers similar performance under the right circumstances. Good luck and…Happy Pythoning!</p> <p><em><a href="fundamental-data-structures-in-python">Read the full “Fundamental Data Structures in Python” article series here</a>. This article is missing something or you found an error? Help a brother out and leave a comment below.</em></p></section><footer></footer></article>https://dbader.org/blog/python-linked-listTue, 27 Jun 2017 00:00:00 GMTEnriching Your Python Classes With Dunder (Magic, Special) Methodshttps://dbader.org/blog/python-dunder-methods<article><header><h1>Enriching Your Python Classes With Dunder (Magic, Special) Methods</h1> <p>What Python’s “magic methods” are and how you would use them to make a simple account class more Pythonic.</p> </header><section><figure><img alt="Magic Methods on Your Python Classes" src="/blog/figures/python-classes-magic-methods.png" width="2560" height="1440"></figure> </section><section><h2>What Are Dunder Methods?</h2> <p>In Python, special methods are a set of predefined methods you can use to enrich your classes. They are easy to recognize because they start and end with double underscores, for example <code>__init__</code> or <code>__str__</code>.</p> <p>As it quickly became tiresome to say under-under-method-under-under Pythonistas adopted the term “dunder methods”, a short form of “double under.”</p> <p>These “dunders” or “special methods” in Python are also sometimes called “magic methods.” But using this terminology can make them seem more complicated than they really are—at the end of the day there’s nothing “magical” about them. <a href="http://www.pixelmonkey.org/2013/04/11/python-double-under-double-wonder" target="_blank">You should treat these methods like a normal language feature.</a></p> <p>Dunder methods let you emulate the behavior of built-in types. For example, to get the length of a string you can call <code>len('string')</code>. But an empty class definition doesn’t support this behavior out of the box:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">NoLenSupport</span><span class="p">:</span> <span class="k">pass</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">obj</span> <span class="o">=</span> <span class="n">NoLenSupport</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span> <span class="ne">TypeError</span><span class="p">:</span> <span class="s2">"object of type 'NoLenSupport' has no len()"</span> </pre></div> <p>To fix this, you can add a <code>__len__</code> dunder method to your class:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">LenSupport</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__len__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="mi">42</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">obj</span> <span class="o">=</span> <span class="n">LenSupport</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span> <span class="mi">42</span> </pre></div> <p>Another example is slicing. You can implement a <code>__getitem__</code> method which allows you to use Python’s list slicing syntax: <code>obj[start:stop]</code>.</p> </section><section><h2>Special Methods and the Python Data Model</h2> <p>This elegant design is known as the <a href="https://docs.python.org/3/reference/datamodel.html" target="_blank">Python data model</a> and lets developers tap into rich language features like sequences, iteration, operator overloading, attribute access, etc.</p> <p>You can see Python’s data model as a powerful API you can interface with by implementing one or more dunder methods. If you want to write more Pythonic code, knowing how and when to use dunder methods is an important step.</p> <p>For a beginner this might be slightly overwhelming at first though. No worries, in this article I will guide you through the use of dunder methods using a simple <code>Account</code> class as an example.</p> </section><section><h2>Enriching a Simple Account Class</h2> <p>Throughout this article I will enrich a simple Python class with various dunder methods to unlock the following language features:</p> <ul> <li>Initialization of new objects</li> <li>Object representation</li> <li>Enable iteration</li> <li>Operator overloading (comparison)</li> <li>Operator overloading (addition)</li> <li>Method invocation</li> <li>Context manager support (<code>with</code> statement)</li> </ul> <p>You can find the final code example <a href="https://github.com/pybites/dunders/blob/master/account.py" target="_blank">here</a>. I’ve also put together a <a href="https://github.com/pybites/dunders/blob/master/richer-classes-with-dunders.ipynb" target="_blank">Jupyter notebook</a> so you can more easily play with the examples.</p> </section><section><h2>Object Initialization: <code>__init__</code></h2> <p>Right upon starting my class I already need a special method. To construct account objects from the <code>Account</code> class I need a constructor which in Python is the <code>__init__</code> dunder:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Account</span><span class="p">:</span> <span class="sd">"""A simple account class"""</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">owner</span><span class="p">,</span> <span class="n">amount</span><span class="o">=</span><span class="mi">0</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> This is the constructor that lets us create</span> <span class="sd"> objects from this class</span> <span class="sd"> """</span> <span class="bp">self</span><span class="o">.</span><span class="n">owner</span> <span class="o">=</span> <span class="n">owner</span> <span class="bp">self</span><span class="o">.</span><span class="n">amount</span> <span class="o">=</span> <span class="n">amount</span> <span class="bp">self</span><span class="o">.</span><span class="n">_transactions</span> <span class="o">=</span> <span class="p">[]</span> </pre></div> <p>The constructor takes care of setting up the object. In this case it receives the owner name, an optional start amount and defines an internal transactions list to keep track of deposits and withdrawals.</p> <p>This allows us to create new accounts like this:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span> <span class="o">=</span> <span class="n">Account</span><span class="p">(</span><span class="s1">'bob'</span><span class="p">)</span> <span class="c1"># default amount = 0</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span> <span class="o">=</span> <span class="n">Account</span><span class="p">(</span><span class="s1">'bob'</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> </pre></div> </section><section><h2>Object Representation: <code>__str__</code>, <code>__repr__</code></h2> <p>It’s common practice in Python to provide a representation of your object for the consumer of your class (a bit like API documentation.) There are two ways to do this using dunder methods:</p> <ol> <li> <p><strong><code>__repr__</code></strong>: The “official” string representation of an object. This is how you would make an object of the class. The goal of <code>__repr__</code> is to be unambiguous.</p> </li> <li> <p><strong><code>__str__</code></strong>: The “informal” or nicely printable string representation of an object. This is for the enduser.</p> </li> </ol> <p>Let’s implement these two methods on the <code>Account</code> class:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Account</span><span class="p">:</span> <span class="c1"># ... (see above)</span> <span class="k">def</span> <span class="fm">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="s1">'Account({!r}, {!r})'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">owner</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">amount</span><span class="p">)</span> <span class="k">def</span> <span class="fm">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="s1">'Account of {} with starting amount: {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span> <span class="bp">self</span><span class="o">.</span><span class="n">owner</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">amount</span><span class="p">)</span> </pre></div> <p>If you don’t want to hardcode <code>"Account"</code> as the name for the class you can also use <code>self.__class__.__name__</code> to access it programmatically.</p> <p>If you wanted to implement just one of these <em>to-string</em> methods on a Python class, <a href="https://stackoverflow.com/questions/1436703/difference-between-str-and-repr-in-python" target="_blank">make sure it’s <code>__repr__</code></a>.</p> <p>Now I can query the object in various ways and always get a nice string representation:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">str</span><span class="p">(</span><span class="n">acc</span><span class="p">)</span> <span class="s1">'Account of bob with starting amount: 10'</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="n">acc</span><span class="p">)</span> <span class="s2">"Account of bob with starting amount: 10"</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">repr</span><span class="p">(</span><span class="n">acc</span><span class="p">)</span> <span class="s2">"Account('bob', 10)"</span> </pre></div> </section><section><h2>Iteration: <code>__len__</code>, <code>__getitem__</code>, <code>__reversed__</code></h2> <p>In order to iterate over our account object I need to add some transactions. So first, I’ll define a simple method to add transactions. I’ll keep it simple because this is just setup code to explain dunder methods, and not a production-ready accounting system:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">add_transaction</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">amount</span><span class="p">):</span> <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">amount</span><span class="p">,</span> <span class="nb">int</span><span class="p">):</span> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'please use int for amount'</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">_transactions</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">amount</span><span class="p">)</span> </pre></div> <p>I also defined a <a href="https://pybit.es/property-decorator.html" target="_blank">property</a> to calculate the balance on the account so I can conveniently access it with <code>account.balance</code>. This method takes the start amount and adds a sum of all the transactions:</p> <div class="codehilite"><pre><span></span><span class="nd">@property</span> <span class="k">def</span> <span class="nf">balance</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">amount</span> <span class="o">+</span> <span class="nb">sum</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_transactions</span><span class="p">)</span> </pre></div> <p>Let’s do some deposits and withdrawals on the account:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span> <span class="o">=</span> <span class="n">Account</span><span class="p">(</span><span class="s1">'bob'</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span><span class="o">.</span><span class="n">add_transaction</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span><span class="o">.</span><span class="n">add_transaction</span><span class="p">(</span><span class="o">-</span><span class="mi">10</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span><span class="o">.</span><span class="n">add_transaction</span><span class="p">(</span><span class="mi">50</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span><span class="o">.</span><span class="n">add_transaction</span><span class="p">(</span><span class="o">-</span><span class="mi">20</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span><span class="o">.</span><span class="n">add_transaction</span><span class="p">(</span><span class="mi">30</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span><span class="o">.</span><span class="n">balance</span> <span class="mi">80</span> </pre></div> <p>Now I have some data and I want to know:</p> <ol> <li> <p>How many transactions were there?</p> </li> <li> <p>Index the account object to get transaction number …</p> </li> <li> <p>Loop over the transactions</p> </li> </ol> <p>With the class definition I have this is currently not possible. All of the following statements raise <code>TypeError</code> exceptions:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">acc</span><span class="p">)</span> <span class="ne">TypeError</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">acc</span><span class="p">:</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="ne">TypeError</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="ne">TypeError</span> </pre></div> <p>Dunder methods to the rescue! It only takes a little bit of code to make the class iterable:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Account</span><span class="p">:</span> <span class="c1"># ... (see above)</span> <span class="k">def</span> <span class="fm">__len__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_transactions</span><span class="p">)</span> <span class="k">def</span> <span class="fm">__getitem__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">position</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_transactions</span><span class="p">[</span><span class="n">position</span><span class="p">]</span> </pre></div> <p>Now the previous statements work:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">acc</span><span class="p">)</span> <span class="mi">5</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">acc</span><span class="p">:</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="mi">20</span> <span class="o">-</span><span class="mi">10</span> <span class="mi">50</span> <span class="o">-</span><span class="mi">20</span> <span class="mi">30</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">-</span><span class="mi">10</span> </pre></div> <p>To iterate over transactions in reversed order you can implement the <code>__reversed__</code> special method:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="fm">__reversed__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="p">[::</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">list</span><span class="p">(</span><span class="nb">reversed</span><span class="p">(</span><span class="n">acc</span><span class="p">))</span> <span class="p">[</span><span class="mi">30</span><span class="p">,</span> <span class="o">-</span><span class="mi">20</span><span class="p">,</span> <span class="mi">50</span><span class="p">,</span> <span class="o">-</span><span class="mi">10</span><span class="p">,</span> <span class="mi">20</span><span class="p">]</span> </pre></div> <p>To reverse the list of transactions I used <a href="https://www.youtube.com/watch?v=sGhY8dQdu4A" target="_blank">Python’s reverse list slice</a> syntax. I also had to wrapp the result of <code>reversed(acc)</code> in a <code>list()</code> call because <code>reversed()</code> returns a a reverse iterator, not a list object we can print nicely in the REPL.</p> <p>All in all, this account class is starting to look quite Pythonic to me now.</p> </section><section><h2>Operator Overloading for Comparing Accounts: <code>__eq__</code>, <code>__lt__</code></h2> <p>We all write dozens of statements daily to compare Python objects:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="mi">2</span> <span class="o">&gt;</span> <span class="mi">1</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="s1">'a'</span> <span class="o">&gt;</span> <span class="s1">'b'</span> <span class="bp">False</span> </pre></div> <p>This feels completely natural, but it’s actually quite amazing what happens behind the scenes here. Why does <code>&gt;</code> work equally well on integers, strings and other objects (as long as they are the same type)? This polymorphic behavior is possible because these objects implement one or more comparison dunder methods.</p> <p>An easy way to verify this is to use the <code>dir()</code> builtin:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">dir</span><span class="p">(</span><span class="s1">'a'</span><span class="p">)</span> <span class="p">[</span><span class="s1">'__add__'</span><span class="p">,</span> <span class="o">...</span> <span class="s1">'__eq__'</span><span class="p">,</span> <span class="o">&lt;---------------</span> <span class="s1">'__format__'</span><span class="p">,</span> <span class="s1">'__ge__'</span><span class="p">,</span> <span class="o">&lt;---------------</span> <span class="s1">'__getattribute__'</span><span class="p">,</span> <span class="s1">'__getitem__'</span><span class="p">,</span> <span class="s1">'__getnewargs__'</span><span class="p">,</span> <span class="s1">'__gt__'</span><span class="p">,</span> <span class="o">&lt;---------------</span> <span class="o">...</span><span class="p">]</span> </pre></div> <p>Let’s build a second account object and compare it to the first one (I am adding a couple of transactions for later use):</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">acc2</span> <span class="o">=</span> <span class="n">Account</span><span class="p">(</span><span class="s1">'tim'</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc2</span><span class="o">.</span><span class="n">add_transaction</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc2</span><span class="o">.</span><span class="n">add_transaction</span><span class="p">(</span><span class="mi">40</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc2</span><span class="o">.</span><span class="n">balance</span> <span class="mi">160</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc2</span> <span class="o">&gt;</span> <span class="n">acc</span> <span class="ne">TypeError</span><span class="p">:</span> <span class="s2">"'&gt;' not supported between instances of 'Account' and 'Account'"</span> </pre></div> <p>What happened here? We got a <code>TypeError</code> because I have not implemented any comparison dunders nor inherited them from a parent class.</p> <p>Let’s add them. To not have to implement all of the comparison dunder methods, I use the <a href="https://docs.python.org/2/library/functools.html#functools.total_ordering" target="_blank">functools.total_ordering</a> decorator which allows me to take a shortcut, only implementing <code>__eq__</code> and <code>__lt__</code>:</p> <div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="n">total_ordering</span> <span class="nd">@total_ordering</span> <span class="k">class</span> <span class="nc">Account</span><span class="p">:</span> <span class="c1"># ... (see above)</span> <span class="k">def</span> <span class="fm">__eq__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">balance</span> <span class="o">==</span> <span class="n">other</span><span class="o">.</span><span class="n">balance</span> <span class="k">def</span> <span class="fm">__lt__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">balance</span> <span class="o">&lt;</span> <span class="n">other</span><span class="o">.</span><span class="n">balance</span> </pre></div> <p>And now I can compare <code>Account</code> instances no problem:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">acc2</span> <span class="o">&gt;</span> <span class="n">acc</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc2</span> <span class="o">&lt;</span> <span class="n">acc</span> <span class="bp">False</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span> <span class="o">==</span> <span class="n">acc2</span> <span class="bp">False</span> </pre></div> </section><section><h2>Operator Overloading for Merging Accounts: <code>__add__</code></h2> <p>In Python, <a href="/blog/python-first-class-functions">everything is an object</a>. We are completely fine adding two integers or two strings with the <code>+</code> (plus) operator, it behaves in expected ways:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="mi">1</span> <span class="o">+</span> <span class="mi">2</span> <span class="mi">3</span> <span class="o">&gt;&gt;&gt;</span> <span class="s1">'hello'</span> <span class="o">+</span> <span class="s1">' world'</span> <span class="s1">'hello world'</span> </pre></div> <p>Again, we see polymorphism at play: Did you notice how <code>+</code> behaves different depending the type of the object? For integers it sums, for strings it concatenates. Again doing a quick <code>dir()</code> on the object reveals the corresponding “dunder” interface into the data model:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">dir</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="p">[</span><span class="o">...</span> <span class="s1">'__add__'</span><span class="p">,</span> <span class="o">...</span> <span class="s1">'__radd__'</span><span class="p">,</span> <span class="o">...</span><span class="p">]</span> </pre></div> <p>Our <code>Account</code> object does not support addition yet, so when you try to add two instances of it there’s a <code>TypeError</code>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span> <span class="o">+</span> <span class="n">acc2</span> <span class="ne">TypeError</span><span class="p">:</span> <span class="s2">"unsupported operand type(s) for +: 'Account' and 'Account'"</span> </pre></div> <p>Let’s implement <code>__add__</code> to be able to merge two accounts. The expected behavior would be to merge all attributes together: the owner name, as well as starting amounts and transactions. To do this we can benefit from the iteration support we implemented earlier:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="fm">__add__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span> <span class="n">owner</span> <span class="o">=</span> <span class="s1">'{}&amp;{}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">owner</span><span class="p">,</span> <span class="n">other</span><span class="o">.</span><span class="n">owner</span><span class="p">)</span> <span class="n">start_amount</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">amount</span> <span class="o">+</span> <span class="n">other</span><span class="o">.</span><span class="n">amount</span> <span class="n">acc</span> <span class="o">=</span> <span class="n">Account</span><span class="p">(</span><span class="n">owner</span><span class="p">,</span> <span class="n">start_amount</span><span class="p">)</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="n">other</span><span class="p">):</span> <span class="n">acc</span><span class="o">.</span><span class="n">add_transaction</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">return</span> <span class="n">acc</span> </pre></div> <p>Yes, it is a bit more involved than the other dunder implementations so far. It should show you though that you are in the driver’s seat. You can implement addition however you please. If we wanted to ignore historic transactions—fine, you can also implement it like this:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="fm">__add__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span> <span class="n">owner</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">owner</span> <span class="o">+</span> <span class="n">other</span><span class="o">.</span><span class="n">owner</span> <span class="n">start_amount</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">balance</span> <span class="o">+</span> <span class="n">other</span><span class="o">.</span><span class="n">balance</span> <span class="k">return</span> <span class="n">Account</span><span class="p">(</span><span class="n">owner</span><span class="p">,</span> <span class="n">start_amount</span><span class="p">)</span> </pre></div> <p>I think the former implementation would be more realistic though, in terms of what a consumer of this class would expect to happen.</p> <p>Now we have a new merged account with starting amount $110 (10 + 100) and balance of $240 (80 + 160):</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">acc3</span> <span class="o">=</span> <span class="n">acc2</span> <span class="o">+</span> <span class="n">acc</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc3</span> <span class="n">Account</span><span class="p">(</span><span class="s1">'tim&amp;bob'</span><span class="p">,</span> <span class="mi">110</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc3</span><span class="o">.</span><span class="n">amount</span> <span class="mi">110</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc3</span><span class="o">.</span><span class="n">balance</span> <span class="mi">240</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc3</span><span class="o">.</span><span class="n">_transactions</span> <span class="p">[</span><span class="mi">20</span><span class="p">,</span> <span class="mi">40</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="o">-</span><span class="mi">10</span><span class="p">,</span> <span class="mi">50</span><span class="p">,</span> <span class="o">-</span><span class="mi">20</span><span class="p">,</span> <span class="mi">30</span><span class="p">]</span> </pre></div> <p>Note this works in both directions because we’re adding objects of the same type. In general, if you would add your object to a builtin (<code>int</code>, <code>str</code>, …) the <code>__add__</code> method of the builtin wouldn’t know anything about your object. In that case you need to implement the reverse add method (<code>__radd__</code>) as well. You can see an example for that <a href="http://www.marinamele.com/2014/04/modifying-add-method-of-python-class.html" target="_blank">here</a>.</p> </section><section><h2>Callable Python Objects: <code>__call__</code></h2> <p>You can make an object callable like a regular function by adding the <code>__call__</code> dunder method. For our account class we could print a nice report of all the transactions that make up its balance:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Account</span><span class="p">:</span> <span class="c1"># ... (see above)</span> <span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">print</span><span class="p">(</span><span class="s1">'Start amount: {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">amount</span><span class="p">))</span> <span class="k">print</span><span class="p">(</span><span class="s1">'Transactions: '</span><span class="p">)</span> <span class="k">for</span> <span class="n">transaction</span> <span class="ow">in</span> <span class="bp">self</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="n">transaction</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">Balance: {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">balance</span><span class="p">))</span> </pre></div> <p>Now when I call the object with the double-parentheses <code>acc()</code> syntax, I get a nice account statement with an overview of all transactions and the current balance:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span> <span class="o">=</span> <span class="n">Account</span><span class="p">(</span><span class="s1">'bob'</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span><span class="o">.</span><span class="n">add_transaction</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span><span class="o">.</span><span class="n">add_transaction</span><span class="p">(</span><span class="o">-</span><span class="mi">10</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span><span class="o">.</span><span class="n">add_transaction</span><span class="p">(</span><span class="mi">50</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span><span class="o">.</span><span class="n">add_transaction</span><span class="p">(</span><span class="o">-</span><span class="mi">20</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span><span class="o">.</span><span class="n">add_transaction</span><span class="p">(</span><span class="mi">30</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">acc</span><span class="p">()</span> <span class="n">Start</span> <span class="n">amount</span><span class="p">:</span> <span class="mi">10</span> <span class="n">Transactions</span><span class="p">:</span> <span class="mi">20</span> <span class="o">-</span><span class="mi">10</span> <span class="mi">50</span> <span class="o">-</span><span class="mi">20</span> <span class="mi">30</span> <span class="n">Balance</span><span class="p">:</span> <span class="mi">80</span> </pre></div> <p>Please keep in mind that this is just a toy example. A “real” account class probably wouldn’t print to the console when you use the function call syntax on one of its instances. In general, the downside of having a <code>__call__</code> method on your objects is that it can be hard to see what the purpose of calling the object is.</p> <p>Most of the time it’s therefore better to add an explicit method to the class. In this case it probably would’ve been more transparent to have a separate <code>Account.print_statement()</code> method.</p> </section><section><h2>Context Manager Support and the <code>With</code> Statement: <code>__enter__</code>, <code>__exit__</code></h2> <p>My final example in this tutorial is about a slightly more advanced concept in Python: Context managers and adding support for the <code>with</code> statement.</p> <p>Now, <a href="/blog/python-context-managers-and-with-statement">what is a “context manager” in Python</a>? Here’s a quick overview:</p> <blockquote> <p>A context manager is a simple “protocol” (or interface) that your object needs to follow so it can be used with the <code>with</code> statement. Basically all you need to do is add <code>__enter__</code> and <code>__exit__</code> methods to an object if you want it to function as a context manager.</p> </blockquote> <p>Let’s use context manager support to add a rollback mechanism to our <code>Account</code> class. If the balance goes negative upon adding another transaction we rollback to the previous state.</p> <p>We can leverage the Pythonic <code>with</code> statement by adding two more dunder methods. I’m also adding some print calls to make the example clearer when we demo it:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Account</span><span class="p">:</span> <span class="c1"># ... (see above)</span> <span class="k">def</span> <span class="fm">__enter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">print</span><span class="p">(</span><span class="s1">'ENTER WITH: Making backup of transactions for rollback'</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">_copy_transactions</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_transactions</span><span class="p">)</span> <span class="k">return</span> <span class="bp">self</span> <span class="k">def</span> <span class="fm">__exit__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">exc_type</span><span class="p">,</span> <span class="n">exc_val</span><span class="p">,</span> <span class="n">exc_tb</span><span class="p">):</span> <span class="k">print</span><span class="p">(</span><span class="s1">'EXIT WITH:'</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">' '</span><span class="p">)</span> <span class="k">if</span> <span class="n">exc_type</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">_transactions</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_copy_transactions</span> <span class="k">print</span><span class="p">(</span><span class="s1">'Rolling back to previous transactions'</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s1">'Transaction resulted in {} ({})'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span> <span class="n">exc_type</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span> <span class="n">exc_val</span><span class="p">))</span> <span class="k">else</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s1">'Transaction OK'</span><span class="p">)</span> </pre></div> <p>As an exception has to be raised to trigger a rollback, I define a quick helper method to validate the transactions in an account:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">validate_transaction</span><span class="p">(</span><span class="n">acc</span><span class="p">,</span> <span class="n">amount_to_add</span><span class="p">):</span> <span class="k">with</span> <span class="n">acc</span> <span class="k">as</span> <span class="n">a</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s1">'Adding {} to account'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">amount_to_add</span><span class="p">))</span> <span class="n">a</span><span class="o">.</span><span class="n">add_transaction</span><span class="p">(</span><span class="n">amount_to_add</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s1">'New balance would be: {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">a</span><span class="o">.</span><span class="n">balance</span><span class="p">))</span> <span class="k">if</span> <span class="n">a</span><span class="o">.</span><span class="n">balance</span> <span class="o">&lt;</span> <span class="mi">0</span><span class="p">:</span> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'sorry cannot go in debt!'</span><span class="p">)</span> </pre></div> <p>Now I can use an <code>Account</code> object with the <code>with</code> statement. When I make a transaction to add a positive amount, all is good:</p> <div class="codehilite"><pre><span></span><span class="n">acc4</span> <span class="o">=</span> <span class="n">Account</span><span class="p">(</span><span class="s1">'sue'</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">Balance start: {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">acc4</span><span class="o">.</span><span class="n">balance</span><span class="p">))</span> <span class="n">validate_transaction</span><span class="p">(</span><span class="n">acc4</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">Balance end: {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">acc4</span><span class="o">.</span><span class="n">balance</span><span class="p">))</span> </pre></div> <p>Executing the above Python snippet produces the following printout:</p> <div class="codehilite"><pre><span></span>Balance start: 10 ENTER WITH: Making backup of transactions for rollback Adding 20 to account New balance would be: 30 EXIT WITH: Transaction OK Balance end: 30 </pre></div> <p>However when I try to withdraw too much money, the code in <code>__exit__</code> kicks in and rolls back the transaction:</p> <div class="codehilite"><pre><span></span><span class="n">acc4</span> <span class="o">=</span> <span class="n">Account</span><span class="p">(</span><span class="s1">'sue'</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">Balance start: {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">acc4</span><span class="o">.</span><span class="n">balance</span><span class="p">))</span> <span class="k">try</span><span class="p">:</span> <span class="n">validate_transaction</span><span class="p">(</span><span class="n">acc4</span><span class="p">,</span> <span class="o">-</span><span class="mi">50</span><span class="p">)</span> <span class="k">except</span> <span class="ne">ValueError</span> <span class="k">as</span> <span class="n">exc</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="n">exc</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">Balance end: {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">acc4</span><span class="o">.</span><span class="n">balance</span><span class="p">))</span> </pre></div> <p>In this case we get a different result:</p> <div class="codehilite"><pre><span></span>Balance start: 10 ENTER WITH: Making backup of transactions for rollback Adding -50 to account New balance would be: -40 EXIT WITH: Rolling back to previous transactions ValueError: sorry cannot go in debt! Balance end: 10 </pre></div> </section><section><h2>Conclusion</h2> <p>I hope you feel a little less afraid of dunder methods after reading this article. A strategic use of them makes your classes more Pythonic, because they emulate builtin types with Python-like behaviors.</p> <p>As with any feature, please don’t overuse it. Operator overloading, for example, can get pretty obscure. Adding “karma” to a person object with <code>+bob</code> or <code>tim &lt;&lt; 3</code> is definitely <em>possible</em> using dunders—but might not be the most obvious or appropriate way to use these special methods. However, for common operations like comparison and additions they can be an elegant approach.</p> <p>Showing each and every dunder method would make for a very long tutorial. If you want to learn more about dunder methods and the Python data model I recommend you go through the <a href="https://docs.python.org/3/reference/datamodel.html" target="_blank">Python reference documentation</a>.</p> <p>Also, be sure to check out our <a href="https://pybit.es/codechallenge24.html" target="_blank">dunder method coding challenge</a> where you can experiment and put your newfound “dunder skills” to practice.</p></section><footer></footer></article>https://dbader.org/blog/python-dunder-methodsTue, 20 Jun 2017 00:00:00 GMTUnderstanding Asynchronous Programming in Pythonhttps://dbader.org/blog/understanding-asynchronous-programming-in-python<article><header><h1>Understanding Asynchronous Programming in Python</h1> <p>How to use Python to write asynchronous programs, and why you’d want to do such a thing.</p> </header><section><figure><img alt="" src="/blog/figures/python-async-programming.png" width="1280" height="720"></figure> <p>A <strong>synchronous program</strong> is what most of us started out writing, and can be thought of as performing one execution step at a time, one after another.</p> <p>Even with conditional branching, loops and function calls, we can still think about the code in terms of taking one execution step at a time, and when complete, moving on to the next.</p> <p>Here are couple of example programs that would work this way:</p> <ul> <li> <p><strong>Batch processing programs</strong> are often created as synchronous programs: get some input, process it, create some output. One step logically follows another till we create the desired output. There’s really nothing else the program has to pay attention to besides those steps, and in that order.</p> </li> <li> <p><strong>Command-line programs</strong> are often small, quick processes to “transform” something into something else. This can be expressed as a series of program steps executed serially and done.</p> </li> </ul> <p>An <strong>asynchronous program</strong> behaves differently. It still takes one execution step at a time. However the difference is the system may not wait for an execution step to be complete before moving on.</p> <p>This means we are continuing onward through execution steps of the program, even though a previous execution step (or multiple steps) is running “elsewhere”. This also implies when one of those execution steps is running “elsewhere” completes, our program code somehow has to handle it.</p> <p>Why would we want to write a program in this manner? The simple answer is it helps us handle particular kinds of programming problems.</p> <p>Here’s a conceptual program that might be a candidate for asynchronous programming:</p> </section><section><h2 class="crosshead thin">Let’s Take a Look at a Simplistic Web Server</h2> <p>Its basic unit of work is the same as we described above for batch processing; get some input, process it, create the output. Written as a synchronous program this would create a working web server.</p> <p>It would also be an <em>absolutely terrible</em> web server.</p> <p><strong>Why?</strong> In the case of a web server one unit of work (input, process, output) is not its only purpose. Its real purpose is to handle hundreds, perhaps thousands, of units of work at the same time, and for long periods of time.</p> <p><strong>Can we make our synchronous web server better?</strong> Sure, we can optimize our execution steps to make them as fast as possible. Unfortunately there are very real limits to this approach that leads to a web server that can’t respond fast enough, and can’t handle enough current users.</p> <p><strong>What are the real limits of optimizing the above approach?</strong> The speed of the network, file IO speed, database query speed, the speed of other connected services, etc. The common feature of this list is they are all IO functions. All of these items are many orders of magnitude slower than our CPU’s processing speed.</p> <p>In a <em>synchronous program</em> if an execution step starts a database query (for example), the CPU is essentially idle for a long time before the query returns with some data and it can continue with the next execution step.</p> <p>For <em>batch oriented programs</em> this isn’t a priority, processing the results of that IO is the goal, and often takes far longer than the IO. Any optimization efforts would be focused on the processing work, not the IO.</p> <p>File, network and database IO are all pretty fast, but still way slower than the CPU. Asynchronous programming techniques allow our programs to take advantage of the relatively slow IO processes, and free the CPU to do other work.</p> <p>When I started trying to understand asynchronous programming, people I asked and documentation I read talked a lot about the importance of writing non-blocking code. Yeah, this never helped me either.</p> <p>What’s non-blocking code? What’s blocking code? That information was like having a reference manual without any practical context about how to use that technical detail in a meaningful way.</p> </section><section><h2 class="crosshead thin">The Real World is Asynchronous</h2> <p>Writing asynchronous programs is different, and kind of hard to get your head around. And that’s interesting because the world we live in, and how we interact with it, is almost entirely asynchronous.</p> <p><strong>Here’s an example a lot of you can relate to:</strong> being a parent trying to do several things at once; balance the checkbook, do some laundry and keep an eye on the kids.</p> <p>We do this without even thinking about it, but let’s break it down somewhat:</p> <ul> <li> <p>Balancing the checkbook is a task we’re trying to get done, and we could think of it as a synchronous task; one step follows another till it’s done.</p> </li> <li> <p>However, we can break away from it to do laundry, unloading the dryer, moving clothes from the washer to the dryer and starting another load in the washer. However, these tasks can be done asynchronously.</p> </li> <li> <p>While we’re actually working with the washer and dryer that’s a synchronous task and we’re working, but the bulk of the task happens after we start the washer and dryer and walk away to get back to work on the checkbook task. Now the task is asynchronous, the washer and dryer will run independently till the buzzer goes off, notifying us that one or the other needs attention.</p> </li> <li> <p>Watching the kids is another asynchronous task. Once they are set up and playing, they do so independently (sort of) until they need attention; someone’s hungry, someone gets hurt, someone yells in alarm, and as parents we react to it. The kids are a long running task with high priority, superceding any other task we might be doing, like the checkbook or laundry.</p> </li> </ul> <p>This example illustrates both blocking and non-blocking code. While we’re moving laudry around, for example, the CPU (the parent) is busy and blocked from doing other work.</p> <p>But it’s okay because the CPU is busy and the task is relatively quick. When we start the washer and dryer and go back to do something else, now the laundry task has become asynchronous because the CPU is doing something else, has changed context if you will, and will be notified when the laundry task is complete by the machine buzzers.</p> <p>As people this is how we work, we’re naturally always juggling multiple things at once, often without thinking about it. As programmers the trick is how to translate this kind of behavior into code that does kind of the same thing.</p> <p>Let’s try to “program” this using code ideas you might be familiar with:</p> </section><section><h2>Thought Experiment #1: The “Batching” Parent</h2> <p>Think about trying to do these tasks in a completely synchronous manner. If we’re a good parent in this scenario we just watch the kids, waiting for something to happen needing our attention. Nothing else, like the checkbook or laundry, would get done in this scenario.</p> <p>We could re-prioritize the tasks any way we want, but only one of them would happen at a time in a synchronous, one after another, manner. This would be like the synchronous web server described above, it would work, but it would be a terrible way to live.</p> <p>Nothing except watching the kids would get done till they were asleep, all other tasks would happen after that, well into the night. A couple of weeks of this and most parents would jump out the window.</p> </section><section><h2>Thought Experiment #2: The “Polling” Parent</h2> <p>Let’s change things up so mulitple things could get done by using polling. In this approach the parent periodically breaks away from any current task and checks to see if any of the other tasks need attention.</p> <p>Since we’re programming a parent, let’s make our polling interval something like fifteen minutes. So here every fifteen minutes the parent goes to check if the washer, dryer or kids need any attention, and then goes back to work on the checkbook. If any of those things do need attention, the work it gets done and the parent goes back to the checkbook task and continues on with the polling loop.</p> <p>This works, tasks are getting done, but has a couple of problems. The CPU (parent) is spending a lot of time checking on things that don’t need attention because they aren’t done, like the washer and dryer. Given the polling interval, it’s entirely possible for tasks to be finished, but they wouldn’t get attention for some time, upto fifteen minutes. And the high priority watching the kids task probably couldn’t tolerate a possible window of fifteen minutes with no attention when something might be going drastically wrong.</p> <p>We could address this by shortening our polling interval, but now the CPU is spending even more time context switching between tasks, and we start to hit a point of diminishing returns. And again, a couple of weeks of living like this and, well, see my previous comment about window and jumping.</p> </section><section><h2>Thought Experiment #3: The “Threading” Parent</h2> <p>As parents it’s often heard, “if I could only clone myself”. Since we’re pretending we can program parents, we can essentially do this by using threading.</p> <p>If we think of all the tasks as one “program”, we can break up the tasks and run them as threads, cloning the parent so to speak. Now there is a parent instance for each task; watching the kids, monitoring the dryer, monitoring the washer and doing the checkbook, all running independently. This sounds like a pretty nice solution to the program problem.</p> <p>But is it? Since we have to tell the parent instances (CPUs) explicitely what to do in a program, we can run into some problems because all instances share everything in the program space.</p> <p>For example, the parent monitoring the dryer sees the clothes are dry, takes control of the dryer and starts unloading. Let’s say that while the dryer parent is unloading clothes, the washer parent sees the washer is done, takes control of the washer, and then wants to take control of the dryer to move clothes from the washer to the dryer. When the dryer parent is finished unloading clothes that parent wants to take control of the washer and move clothes from the washer to the dryer.</p> <p>Now those two parents are <a href="https://en.wikipedia.org/wiki/Deadlock" target="_blank">deadlocked</a>.</p> <p>Both have control of their own resource, and want control of the other resource. They will wait forever for the other to release control. As programmers we’d have to write code to work this situation out.</p> <p>Here’s another issue that might arise from parent threading. Suppose that unfortunately a child gets hurt and that parent has to take the child to emergent care. That happens right away because that parent clone is dedicated to watching the kids. But at emergent care the parent has to write a fairly large check to cover the deductible.</p> <p>Meanwhile, the parent working on the checkbook is unaware of this large check being written, and suddenly the family account is overdrawn. Because the parent clones work within the same program, and the family money (checkbook) is a shared resource in that world, we’d have to work out a way to for the kid watching parent to inform the checkbook parent of what’s going on. Or provide some kind of locking mechanism so the resource can be used by only one parent at a time, with updates.</p> <p>All of these things are manageable in program threading code, but it’s difficult to get right, and hard to debug when it’s wrong.</p> </section><section><h2 class="crosshead thin">Let’s Write Some Python Code</h2> <p>Now we’re going to take some of the approaches outlined in these “thought experiments” and we’ll turn them into functioning Python programs.</p> <p>You can download all of the example code from <a href="https://github.com/writeson/async_python_programming_presentation" target="_blank">this GitHub repository</a>.</p> <p>All the examples in this article have been tested with Python 3.6.1, and the <a href="https://github.com/writeson/async_python_programming_presentation/blob/master/requirements.txt" target="_blank"><code>requirements.txt</code> file included with the code examples</a> indicates what modules you’ll need to run all the examples.</p> <p>I would strongly suggest setting up a <a href="https://www.youtube.com/watch?v=UqkT2Ml9beg" target="_blank">Python virtual environment</a> to run the code so as not to interfere with your system Python.</p> </section><section><h2 class="crosshead thin">Example 1: Synchronous Programming</h2> <p>This first example shows a somewhat contrived way of having a task pull “work” off a queue and do that work. In this case the work is just getting a number, and the task counts up to that number. It also prints it’s running at every count step, and prints the total at the end. The contrived part is this program provides a naive basis for multiple tasks to process the work on the queue.</p> <div class="codehilite"><pre><span></span><span class="sd">"""</span> <span class="sd">example_1.py</span> <span class="sd">Just a short example showing synchronous running of 'tasks'</span> <span class="sd">"""</span> <span class="kn">import</span> <span class="nn">queue</span> <span class="k">def</span> <span class="nf">task</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">):</span> <span class="k">if</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">empty</span><span class="p">():</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} nothing to do'</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="k">while</span> <span class="ow">not</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">empty</span><span class="p">():</span> <span class="n">count</span> <span class="o">=</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="n">total</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">count</span><span class="p">):</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} running'</span><span class="p">)</span> <span class="n">total</span> <span class="o">+=</span> <span class="mi">1</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} total: {total}'</span><span class="p">)</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span> <span class="sd">"""</span> <span class="sd"> This is the main entry point for the program</span> <span class="sd"> """</span> <span class="c1"># create the queue of 'work'</span> <span class="n">work_queue</span> <span class="o">=</span> <span class="n">queue</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span> <span class="c1"># put some 'work' in the queue</span> <span class="k">for</span> <span class="n">work</span> <span class="ow">in</span> <span class="p">[</span><span class="mi">15</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">2</span><span class="p">]:</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">work</span><span class="p">)</span> <span class="c1"># create some tasks</span> <span class="n">tasks</span> <span class="o">=</span> <span class="p">[</span> <span class="p">(</span><span class="n">task</span><span class="p">,</span> <span class="s1">'One'</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">),</span> <span class="p">(</span><span class="n">task</span><span class="p">,</span> <span class="s1">'Two'</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">)</span> <span class="p">]</span> <span class="c1"># run the tasks</span> <span class="k">for</span> <span class="n">t</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">q</span> <span class="ow">in</span> <span class="n">tasks</span><span class="p">:</span> <span class="n">t</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">q</span><span class="p">)</span> <span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span> <span class="n">main</span><span class="p">()</span> </pre></div> <p>The “task” in this program is just a function that accepts a string and a queue. When executed it looks to see if there is anything in the queue to process, and if so it pulls values off the queue, starts a for loop to count up to that value, and prints the total at the end. It continues this till there is nothing left in the queue, and exits.</p> <p>When we run this task we get a listing showing that task one does all the work. The loop within it consumes all the work on the queue, and performs it. When that loop exits, task two gets a chance to run, but finds the queue empty, so it prints a statement to that affect and exits. There is nothing in the code that allows task one and task two to play nice together and switch between them.</p> </section><section><h2 class="crosshead thin">Example 2: Simple Cooperative Concurrency</h2> <p>The next version of the program (<code>example_2.py</code>) adds the ability of the two tasks to play nice together through the use of generators. The addition of the yield statement in the task function means the loop exits at that point, but maintains its context so it can be restarted later. The “run the tasks” loop later in the program takes advantage of this when it calls <code>t.next()</code>. This statement restarts the task at the point where it previously yielded.</p> <p>This is a form of cooperative concurrency. The program is yielding control of its current context so something else can run. In this case it allows our primative “run the tasks” scheduler to run two instances of the task function, each one consuming work from the same queue. This is sort of clever, but a lot of work to get the same results as the first program.</p> <div class="codehilite"><pre><span></span><span class="sd">"""</span> <span class="sd">example_2.py</span> <span class="sd">Just a short example demonstrating a simple state machine in Python</span> <span class="sd">"""</span> <span class="kn">import</span> <span class="nn">queue</span> <span class="k">def</span> <span class="nf">task</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">queue</span><span class="p">):</span> <span class="k">while</span> <span class="ow">not</span> <span class="n">queue</span><span class="o">.</span><span class="n">empty</span><span class="p">():</span> <span class="n">count</span> <span class="o">=</span> <span class="n">queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="n">total</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">count</span><span class="p">):</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} running'</span><span class="p">)</span> <span class="n">total</span> <span class="o">+=</span> <span class="mi">1</span> <span class="k">yield</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} total: {total}'</span><span class="p">)</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span> <span class="sd">"""</span> <span class="sd"> This is the main entry point for the program</span> <span class="sd"> """</span> <span class="c1"># create the queue of 'work'</span> <span class="n">work_queue</span> <span class="o">=</span> <span class="n">queue</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span> <span class="c1"># put some 'work' in the queue</span> <span class="k">for</span> <span class="n">work</span> <span class="ow">in</span> <span class="p">[</span><span class="mi">15</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">2</span><span class="p">]:</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">work</span><span class="p">)</span> <span class="c1"># create some tasks</span> <span class="n">tasks</span> <span class="o">=</span> <span class="p">[</span> <span class="n">task</span><span class="p">(</span><span class="s1">'One'</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">),</span> <span class="n">task</span><span class="p">(</span><span class="s1">'Two'</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">)</span> <span class="p">]</span> <span class="c1"># run the tasks</span> <span class="n">done</span> <span class="o">=</span> <span class="bp">False</span> <span class="k">while</span> <span class="ow">not</span> <span class="n">done</span><span class="p">:</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">tasks</span><span class="p">:</span> <span class="k">try</span><span class="p">:</span> <span class="nb">next</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">except</span> <span class="ne">StopIteration</span><span class="p">:</span> <span class="n">tasks</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">tasks</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="n">done</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span> <span class="n">main</span><span class="p">()</span> </pre></div> <p>When this program is run the output shows that both task one and two are running, consuming work from the queue and processing it. This is what’s intended, both tasks are processing work, and each ends up processing two items from the queue. But again, quite a bit of work to achieve the results.</p> <p>The trick here is using the <code>yield</code> statement, which turns the task function into a generator, to perform a “context switch”. The program uses this context switch in order to run two instances of the task.</p> </section><section><h2 class="crosshead thin">Example 3: Cooperative Concurreny With Blocking Calls</h2> <p>The next version of the program (<code>example_3.py</code>) is exactly the same as the last, except for the addition of a <code>time.sleep(1)</code> call in the body of our task loop. This adds a one second delay to every iteration of the task loop. The delay was added to simulate the affect of a slow IO process occurring in our task.</p> <p>I’ve also included a simple Elapsed Time class to handle the start time/elapsed time features used in the reporting.</p> <div class="codehilite"><pre><span></span><span class="sd">"""</span> <span class="sd">example_3.py</span> <span class="sd">Just a short example demonstraing a simple state machine in Python</span> <span class="sd">However, this one has delays that affect it</span> <span class="sd">"""</span> <span class="kn">import</span> <span class="nn">time</span> <span class="kn">import</span> <span class="nn">queue</span> <span class="kn">from</span> <span class="nn">lib.elapsed_time</span> <span class="kn">import</span> <span class="n">ET</span> <span class="k">def</span> <span class="nf">task</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">queue</span><span class="p">):</span> <span class="k">while</span> <span class="ow">not</span> <span class="n">queue</span><span class="o">.</span><span class="n">empty</span><span class="p">():</span> <span class="n">count</span> <span class="o">=</span> <span class="n">queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="n">total</span> <span class="o">=</span> <span class="mi">0</span> <span class="n">et</span> <span class="o">=</span> <span class="n">ET</span><span class="p">()</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">count</span><span class="p">):</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} running'</span><span class="p">)</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="n">total</span> <span class="o">+=</span> <span class="mi">1</span> <span class="k">yield</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} total: {total}'</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} total elapsed time: {et():.1f}'</span><span class="p">)</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span> <span class="sd">"""</span> <span class="sd"> This is the main entry point for the program</span> <span class="sd"> """</span> <span class="c1"># create the queue of 'work'</span> <span class="n">work_queue</span> <span class="o">=</span> <span class="n">queue</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span> <span class="c1"># put some 'work' in the queue</span> <span class="k">for</span> <span class="n">work</span> <span class="ow">in</span> <span class="p">[</span><span class="mi">15</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">2</span><span class="p">]:</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">work</span><span class="p">)</span> <span class="n">tasks</span> <span class="o">=</span> <span class="p">[</span> <span class="n">task</span><span class="p">(</span><span class="s1">'One'</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">),</span> <span class="n">task</span><span class="p">(</span><span class="s1">'Two'</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">)</span> <span class="p">]</span> <span class="c1"># run the scheduler to run the tasks</span> <span class="n">et</span> <span class="o">=</span> <span class="n">ET</span><span class="p">()</span> <span class="n">done</span> <span class="o">=</span> <span class="bp">False</span> <span class="k">while</span> <span class="ow">not</span> <span class="n">done</span><span class="p">:</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">tasks</span><span class="p">:</span> <span class="k">try</span><span class="p">:</span> <span class="nb">next</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">except</span> <span class="ne">StopIteration</span><span class="p">:</span> <span class="n">tasks</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">tasks</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="n">done</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">print</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="s1">'Total elapsed time: {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">et</span><span class="p">()))</span> <span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span> <span class="n">main</span><span class="p">()</span> </pre></div> <p>When this program is run the output shows that both task one and two are running, consuming work from the queue and processing it as before. With the addition of the mock IO delay, we’re seeing that our cooperative concurrency hasn’t gotten us anything, the delay stops the processing of the entire program, and the CPU just waits for the IO delay to be over.</p> <p>This is exactly what’s meant by “blocking code” in asynchronous documentation. Notice the time it takes to the run the entire program, this is the cummulative time of the all the delays. This again shows running things this way is not a win.</p> </section><section><h2 class="crosshead thin">Example 4: Cooperative Concurrency With Non-Blocking Calls (gevent)</h2> <p>The next version of the program (<code>example_4.py</code>) has been modified quite a bit. It makes use of the <a href="http://www.gevent.org/" target="_blank">gevent asynchronous programming module</a> right at the top of the program. The module is imported, along with a module called <code>monkey</code>.</p> <p>Then a method of the <code>monkey</code> module is called, <code>patch_all()</code>. What in the world is that doing? The simple explanation is it sets the program up so any other module imported having blocking (synchronous) code in it is “patched” to make it asynchronous.</p> <p>Like most simple explanations, this isn’t very helpful. What it means in relation to our example program is the <code>time.sleep(1)</code> (our mock IO delay) no longer “blocks” the program. Instead it yields control cooperatively back to the system. Notice the “yield” statement from <code>example_3.py</code> is no longer present, it’s now part of the <code>time.sleep(1)</code> call.</p> <p>So, if the <code>time.sleep(1)</code> function has been patched by gevent to yield control, where is the control going? One of the effects of using gevent is that it starts an event loop thread in the program. For our purposes this is like the “run the tasks” loop from <code>example_3.py</code>. When the <code>time.sleep(1)</code> delay ends, it returns control to the next executable statement after the <code>time.sleep(1)</code> statement. The advantage of this behavior is the CPU is no longer blocked by the delay, but is free to execute other code.</p> <p>Our “run the tasks” loop no longer exists, instead our task array contains two calls to <code>gevent.spawn(...)</code>. These two calls start two gevent threads (called greenlets), which are lightweight microthreads that context switch cooperatively, rather than as a result of the system switching contexts like regular threads.</p> <p>Notice the <code>gevent.joinall(tasks)</code> right after our tasks are spawned. This statement causes our program to wait till task one and task two are both finished. Without this our program would have continued on through the print statements, but with essentially nothing to do.</p> <div class="codehilite"><pre><span></span><span class="sd">"""</span> <span class="sd">example_4.py</span> <span class="sd">Just a short example demonstrating a simple state machine in Python</span> <span class="sd">However, this one has delays that affect it</span> <span class="sd">"""</span> <span class="kn">import</span> <span class="nn">gevent</span> <span class="kn">from</span> <span class="nn">gevent</span> <span class="kn">import</span> <span class="n">monkey</span> <span class="n">monkey</span><span class="o">.</span><span class="n">patch_all</span><span class="p">()</span> <span class="kn">import</span> <span class="nn">time</span> <span class="kn">import</span> <span class="nn">queue</span> <span class="kn">from</span> <span class="nn">lib.elapsed_time</span> <span class="kn">import</span> <span class="n">ET</span> <span class="k">def</span> <span class="nf">task</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">):</span> <span class="k">while</span> <span class="ow">not</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">empty</span><span class="p">():</span> <span class="n">count</span> <span class="o">=</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="n">total</span> <span class="o">=</span> <span class="mi">0</span> <span class="n">et</span> <span class="o">=</span> <span class="n">ET</span><span class="p">()</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">count</span><span class="p">):</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} running'</span><span class="p">)</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="n">total</span> <span class="o">+=</span> <span class="mi">1</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} total: {total}'</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} total elapsed time: {et():.1f}'</span><span class="p">)</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span> <span class="sd">"""</span> <span class="sd"> This is the main entry point for the programWhen</span> <span class="sd"> """</span> <span class="c1"># create the queue of 'work'</span> <span class="n">work_queue</span> <span class="o">=</span> <span class="n">queue</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span> <span class="c1"># put some 'work' in the queue</span> <span class="k">for</span> <span class="n">work</span> <span class="ow">in</span> <span class="p">[</span><span class="mi">15</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">2</span><span class="p">]:</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">work</span><span class="p">)</span> <span class="c1"># run the tasks</span> <span class="n">et</span> <span class="o">=</span> <span class="n">ET</span><span class="p">()</span> <span class="n">tasks</span> <span class="o">=</span> <span class="p">[</span> <span class="n">gevent</span><span class="o">.</span><span class="n">spawn</span><span class="p">(</span><span class="n">task</span><span class="p">,</span> <span class="s1">'One'</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">),</span> <span class="n">gevent</span><span class="o">.</span><span class="n">spawn</span><span class="p">(</span><span class="n">task</span><span class="p">,</span> <span class="s1">'Two'</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">)</span> <span class="p">]</span> <span class="n">gevent</span><span class="o">.</span><span class="n">joinall</span><span class="p">(</span><span class="n">tasks</span><span class="p">)</span> <span class="k">print</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Total elapsed time: {et():.1f}'</span><span class="p">)</span> <span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span> <span class="n">main</span><span class="p">()</span> </pre></div> <p>When this program runs, notice both task one and two start at the same time, then wait at the mock IO call. This is an indication the <code>time.sleep(1)</code> call is no longer blocking, and other work is being done.</p> <p>At the end of the program notice the total elapsed time, it’s essentially half the time it took for <code>example_3.py</code> to run. Now we’re starting to see the advantages of an asynchronous program.</p> <p>Being able to run two, or more, things concurrently by running IO processes in a non-blocking manner. By using gevent greenlets and controlling the context switches, we’re able to multiplex between tasks without to much trouble.</p> </section><section><h2 class="crosshead thin">Example 5: Synchronous (Blocking) HTTP Downloads</h2> <p>The next version of the program (<code>example_5.py</code>) is kind of a step forward and step back. The program now is doing some actual work with real IO, making HTTP requests to a list of URLs and getting the page contents, but it’s doing so in a blocking (synchronous) manner.</p> <p>We’ve modified the program to import the wonderful <a href="https://requests.org/" target="_blank"><code>requests</code> module</a> to make the actual HTTP requests, and added a list of URLs to the queue rather than numbers. Inside the task, rather than increment a counter, we’re using the requests module to get the contents of a URL gotten from the queue, and printing how long it took to do so.</p> <div class="codehilite"><pre><span></span><span class="sd">"""</span> <span class="sd">example_5.py</span> <span class="sd">Just a short example demonstrating a simple state machine in Python</span> <span class="sd">This version is doing actual work, downloading the contents of</span> <span class="sd">URL's it gets from a queue</span> <span class="sd">"""</span> <span class="kn">import</span> <span class="nn">queue</span> <span class="kn">import</span> <span class="nn">requests</span> <span class="kn">from</span> <span class="nn">lib.elapsed_time</span> <span class="kn">import</span> <span class="n">ET</span> <span class="k">def</span> <span class="nf">task</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">):</span> <span class="k">while</span> <span class="ow">not</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">empty</span><span class="p">():</span> <span class="n">url</span> <span class="o">=</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} getting URL: {url}'</span><span class="p">)</span> <span class="n">et</span> <span class="o">=</span> <span class="n">ET</span><span class="p">()</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} got URL: {url}'</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} total elapsed time: {et():.1f}'</span><span class="p">)</span> <span class="k">yield</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span> <span class="sd">"""</span> <span class="sd"> This is the main entry point for the program</span> <span class="sd"> """</span> <span class="c1"># create the queue of 'work'</span> <span class="n">work_queue</span> <span class="o">=</span> <span class="n">queue</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span> <span class="c1"># put some 'work' in the queue</span> <span class="k">for</span> <span class="n">url</span> <span class="ow">in</span> <span class="p">[</span> <span class="s2">"http://google.com"</span><span class="p">,</span> <span class="s2">"http://yahoo.com"</span><span class="p">,</span> <span class="s2">"http://linkedin.com"</span><span class="p">,</span> <span class="s2">"http://shutterfly.com"</span><span class="p">,</span> <span class="s2">"http://mypublisher.com"</span><span class="p">,</span> <span class="s2">"http://facebook.com"</span> <span class="p">]:</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="n">tasks</span> <span class="o">=</span> <span class="p">[</span> <span class="n">task</span><span class="p">(</span><span class="s1">'One'</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">),</span> <span class="n">task</span><span class="p">(</span><span class="s1">'Two'</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">)</span> <span class="p">]</span> <span class="c1"># run the scheduler to run the tasks</span> <span class="n">et</span> <span class="o">=</span> <span class="n">ET</span><span class="p">()</span> <span class="n">done</span> <span class="o">=</span> <span class="bp">False</span> <span class="k">while</span> <span class="ow">not</span> <span class="n">done</span><span class="p">:</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">tasks</span><span class="p">:</span> <span class="k">try</span><span class="p">:</span> <span class="nb">next</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">except</span> <span class="ne">StopIteration</span><span class="p">:</span> <span class="n">tasks</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">tasks</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="n">done</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">print</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Total elapsed time: {et():.1f}'</span><span class="p">)</span> <span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span> <span class="n">main</span><span class="p">()</span> </pre></div> <p>As in an earlier version of the program, we’re using a <code>yield</code> to turn our task function into a generator, and perform a context switch in order to let the other task instance run.</p> <p>Each task gets a URL from the work queue, gets the contents of the page pointed to by the URL and reports how long it took to get that content.</p> <p>As before, the <code>yield</code> allows both our tasks to run, but because this program is running synchronously, each <code>requests.get()</code> call blocks the CPU till the page is retrieved. Notice the total time to run the entire program at the end, this will be meaningful for the next example.</p> </section><section><h2 class="crosshead thin">Example 6: Asynchronous (Non-Blocking) HTTP Downloads With gevent</h2> <p>This version of the program (<code>example_6.py</code>) modifies the previous version to use the gevent module again. Remember the gevent <code>monkey.patch_all()</code> call modifies any following modules so synchronous code becomes asynchronous, this includes <code>requests</code>.</p> <p>Now the tasks have been modified to remove the <code>yield</code> call because the <code>requests.get(url)</code> call is no longer blocking, but performs a context switch back to the gevent event loop. In the “run the task” section we use gevent to spawn two instance of the task generator, then use <code>joinall()</code> to wait for them to complete.</p> <div class="codehilite"><pre><span></span><span class="sd">"""</span> <span class="sd">example_6.py</span> <span class="sd">Just a short example demonstrating a simple state machine in Python</span> <span class="sd">This version is doing actual work, downloading the contents of</span> <span class="sd">URL's it gets from a queue. It's also using gevent to get the</span> <span class="sd">URL's in an asynchronous manner.</span> <span class="sd">"""</span> <span class="kn">import</span> <span class="nn">gevent</span> <span class="kn">from</span> <span class="nn">gevent</span> <span class="kn">import</span> <span class="n">monkey</span> <span class="n">monkey</span><span class="o">.</span><span class="n">patch_all</span><span class="p">()</span> <span class="kn">import</span> <span class="nn">queue</span> <span class="kn">import</span> <span class="nn">requests</span> <span class="kn">from</span> <span class="nn">lib.elapsed_time</span> <span class="kn">import</span> <span class="n">ET</span> <span class="k">def</span> <span class="nf">task</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">):</span> <span class="k">while</span> <span class="ow">not</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">empty</span><span class="p">():</span> <span class="n">url</span> <span class="o">=</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} getting URL: {url}'</span><span class="p">)</span> <span class="n">et</span> <span class="o">=</span> <span class="n">ET</span><span class="p">()</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} got URL: {url}'</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} total elapsed time: {et():.1f}'</span><span class="p">)</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span> <span class="sd">"""</span> <span class="sd"> This is the main entry point for the program</span> <span class="sd"> """</span> <span class="c1"># create the queue of 'work'</span> <span class="n">work_queue</span> <span class="o">=</span> <span class="n">queue</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span> <span class="c1"># put some 'work' in the queue</span> <span class="k">for</span> <span class="n">url</span> <span class="ow">in</span> <span class="p">[</span> <span class="s2">"http://google.com"</span><span class="p">,</span> <span class="s2">"http://yahoo.com"</span><span class="p">,</span> <span class="s2">"http://linkedin.com"</span><span class="p">,</span> <span class="s2">"http://shutterfly.com"</span><span class="p">,</span> <span class="s2">"http://mypublisher.com"</span><span class="p">,</span> <span class="s2">"http://facebook.com"</span> <span class="p">]:</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="c1"># run the tasks</span> <span class="n">et</span> <span class="o">=</span> <span class="n">ET</span><span class="p">()</span> <span class="n">tasks</span> <span class="o">=</span> <span class="p">[</span> <span class="n">gevent</span><span class="o">.</span><span class="n">spawn</span><span class="p">(</span><span class="n">task</span><span class="p">,</span> <span class="s1">'One'</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">),</span> <span class="n">gevent</span><span class="o">.</span><span class="n">spawn</span><span class="p">(</span><span class="n">task</span><span class="p">,</span> <span class="s1">'Two'</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">)</span> <span class="p">]</span> <span class="n">gevent</span><span class="o">.</span><span class="n">joinall</span><span class="p">(</span><span class="n">tasks</span><span class="p">)</span> <span class="k">print</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Total elapsed time: {et():.1f}'</span><span class="p">)</span> <span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span> <span class="n">main</span><span class="p">()</span> </pre></div> <p>At the end of this program run, take a look at the total time and the individual times to get the contents of the URL’s. You’ll see the total time is <em>less</em> than the cummulative time of all the <code>requests.get()</code> calls.</p> <p>This is because those calls are running asynchronously, so we’re effectively taking better advantage of the CPU by allowing it to make multiple requests at once.</p> </section><section><h2 class="crosshead thin">Example 7: Asynchronous (Non-Blocking) HTTP Downloads With Twisted</h2> <p>This version of the program (<code>example_7.py</code>) uses the <a href="https://twistedmatrix.com/" target="_blank">Twisted module</a> to do essentially the same thing as the gevent module, download the URL contents in a non-blocking manner.</p> <p>Twisted is a very powerful system, and takes a fundementally different approach to create asynchronous programs. Where gevent modifies modules to make their synchronous code asynchronous, Twisted provides it’s own functions and methods to reach the same ends.</p> <p>Where <code>example_6.py</code> used the patched <code>requests.get(url)</code> call to get the contents of the URLs, here we use the Twisted function <code>getPage(url)</code>.</p> <p>In this version the <code>@defer.inlineCallbacks</code> function decorator works together with the <code>yield getPage(url)</code> to perform a context switch into the Twisted event loop.</p> <p>In gevent the event loop was implied, but in Twisted it’s explicitely provided by the <code>reactor.run()</code> statement line near the bottom of the program.</p> <div class="codehilite"><pre><span></span><span class="sd">"""</span> <span class="sd">example_7.py</span> <span class="sd">Just a short example demonstrating a simple state machine in Python</span> <span class="sd">This version is doing actual work, downloading the contents of</span> <span class="sd">URL's it gets from a work_queue. This version uses the Twisted</span> <span class="sd">framework to provide the concurrency</span> <span class="sd">"""</span> <span class="kn">from</span> <span class="nn">twisted.internet</span> <span class="kn">import</span> <span class="n">defer</span> <span class="kn">from</span> <span class="nn">twisted.web.client</span> <span class="kn">import</span> <span class="n">getPage</span> <span class="kn">from</span> <span class="nn">twisted.internet</span> <span class="kn">import</span> <span class="n">reactor</span><span class="p">,</span> <span class="n">task</span> <span class="kn">import</span> <span class="nn">queue</span> <span class="kn">from</span> <span class="nn">lib.elapsed_time</span> <span class="kn">import</span> <span class="n">ET</span> <span class="nd">@defer.inlineCallbacks</span> <span class="k">def</span> <span class="nf">my_task</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">):</span> <span class="k">try</span><span class="p">:</span> <span class="k">while</span> <span class="ow">not</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">empty</span><span class="p">():</span> <span class="n">url</span> <span class="o">=</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} getting URL: {url}'</span><span class="p">)</span> <span class="n">et</span> <span class="o">=</span> <span class="n">ET</span><span class="p">()</span> <span class="k">yield</span> <span class="n">getPage</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} got URL: {url}'</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} total elapsed time: {et():.1f}'</span><span class="p">)</span> <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span> <span class="sd">"""</span> <span class="sd"> This is the main entry point for the program</span> <span class="sd"> """</span> <span class="c1"># create the work_queue of 'work'</span> <span class="n">work_queue</span> <span class="o">=</span> <span class="n">queue</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span> <span class="c1"># put some 'work' in the work_queue</span> <span class="k">for</span> <span class="n">url</span> <span class="ow">in</span> <span class="p">[</span> <span class="sa">b</span><span class="s2">"http://google.com"</span><span class="p">,</span> <span class="sa">b</span><span class="s2">"http://yahoo.com"</span><span class="p">,</span> <span class="sa">b</span><span class="s2">"http://linkedin.com"</span><span class="p">,</span> <span class="sa">b</span><span class="s2">"http://shutterfly.com"</span><span class="p">,</span> <span class="sa">b</span><span class="s2">"http://mypublisher.com"</span><span class="p">,</span> <span class="sa">b</span><span class="s2">"http://facebook.com"</span> <span class="p">]:</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="c1"># run the tasks</span> <span class="n">et</span> <span class="o">=</span> <span class="n">ET</span><span class="p">()</span> <span class="n">defer</span><span class="o">.</span><span class="n">DeferredList</span><span class="p">([</span> <span class="n">task</span><span class="o">.</span><span class="n">deferLater</span><span class="p">(</span><span class="n">reactor</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">my_task</span><span class="p">,</span> <span class="s1">'One'</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">),</span> <span class="n">task</span><span class="o">.</span><span class="n">deferLater</span><span class="p">(</span><span class="n">reactor</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">my_task</span><span class="p">,</span> <span class="s1">'Two'</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">)</span> <span class="p">])</span><span class="o">.</span><span class="n">addCallback</span><span class="p">(</span><span class="k">lambda</span> <span class="n">_</span><span class="p">:</span> <span class="n">reactor</span><span class="o">.</span><span class="n">stop</span><span class="p">())</span> <span class="c1"># run the event loop</span> <span class="n">reactor</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> <span class="k">print</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Total elapsed time: {et():.1f}'</span><span class="p">)</span> <span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span> <span class="n">main</span><span class="p">()</span> </pre></div> <p>Notice the end result is the same as the gevent version, the total program run time is less than the cummulative time for each URL to be retrieved.</p> </section><section><h2 class="crosshead thin">Example 8: Asynchronous (Non-Blocking) HTTP Downloads With Twisted Callbacks</h2> <p>This version of the program (<code>example_8.py</code>) also uses the Twisted library, but shows a more traditional approach to using Twisted.</p> <p>By this I mean rather than using the <code>@defer.inlineCallbacks</code> / <code>yield</code> style of coding, this version uses explicit callbacks. A “callback” is a function that is passed to the system and can be called later in reaction to an event. In the example below the <code>success_callback()</code> function is provided to Twisted to be called when the <code>getPage(url)</code> call completes.</p> <p>Notice in the program the <code>@defer.inlineCallbacks</code> decorator is no longer present on the <code>my_task()</code> function. In addition, the function is yielding a variable called <code>d</code>, shortand for something called a deferred, which is what is returned by the <code>getPage(url)</code> function call.</p> <p>A <em>deferred</em> is Twisted’s way of handling asynchronous programming, and is what the callback is attached to. When this deferred “fires” (when the <code>getPage(url)</code> completes), the callback function will be called with the variables defined at the time the callback was attached.</p> <div class="codehilite"><pre><span></span><span class="sd">"""</span> <span class="sd">example_8.py</span> <span class="sd">Just a short example demonstrating a simple state machine in Python</span> <span class="sd">This version is doing actual work, downloading the contents of</span> <span class="sd">URL's it gets from a queue. This version uses the Twisted</span> <span class="sd">framework to provide the concurrency</span> <span class="sd">"""</span> <span class="kn">from</span> <span class="nn">twisted.internet</span> <span class="kn">import</span> <span class="n">defer</span> <span class="kn">from</span> <span class="nn">twisted.web.client</span> <span class="kn">import</span> <span class="n">getPage</span> <span class="kn">from</span> <span class="nn">twisted.internet</span> <span class="kn">import</span> <span class="n">reactor</span><span class="p">,</span> <span class="n">task</span> <span class="kn">import</span> <span class="nn">queue</span> <span class="kn">from</span> <span class="nn">lib.elapsed_time</span> <span class="kn">import</span> <span class="n">ET</span> <span class="k">def</span> <span class="nf">success_callback</span><span class="p">(</span><span class="n">results</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">url</span><span class="p">,</span> <span class="n">et</span><span class="p">):</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} got URL: {url}'</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} total elapsed time: {et():.1f}'</span><span class="p">)</span> <span class="k">def</span> <span class="nf">my_task</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">queue</span><span class="p">):</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">queue</span><span class="o">.</span><span class="n">empty</span><span class="p">():</span> <span class="k">while</span> <span class="ow">not</span> <span class="n">queue</span><span class="o">.</span><span class="n">empty</span><span class="p">():</span> <span class="n">url</span> <span class="o">=</span> <span class="n">queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Task {name} getting URL: {url}'</span><span class="p">)</span> <span class="n">et</span> <span class="o">=</span> <span class="n">ET</span><span class="p">()</span> <span class="n">d</span> <span class="o">=</span> <span class="n">getPage</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="n">d</span><span class="o">.</span><span class="n">addCallback</span><span class="p">(</span><span class="n">success_callback</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">url</span><span class="p">,</span> <span class="n">et</span><span class="p">)</span> <span class="k">yield</span> <span class="n">d</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span> <span class="sd">"""</span> <span class="sd"> This is the main entry point for the program</span> <span class="sd"> """</span> <span class="c1"># create the queue of 'work'</span> <span class="n">work_queue</span> <span class="o">=</span> <span class="n">queue</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span> <span class="c1"># put some 'work' in the queue</span> <span class="k">for</span> <span class="n">url</span> <span class="ow">in</span> <span class="p">[</span> <span class="sa">b</span><span class="s2">"http://google.com"</span><span class="p">,</span> <span class="sa">b</span><span class="s2">"http://yahoo.com"</span><span class="p">,</span> <span class="sa">b</span><span class="s2">"http://linkedin.com"</span><span class="p">,</span> <span class="sa">b</span><span class="s2">"http://shutterfly.com"</span><span class="p">,</span> <span class="sa">b</span><span class="s2">"http://mypublisher.com"</span><span class="p">,</span> <span class="sa">b</span><span class="s2">"http://facebook.com"</span> <span class="p">]:</span> <span class="n">work_queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="c1"># run the tasks</span> <span class="n">et</span> <span class="o">=</span> <span class="n">ET</span><span class="p">()</span> <span class="c1"># create cooperator</span> <span class="n">coop</span> <span class="o">=</span> <span class="n">task</span><span class="o">.</span><span class="n">Cooperator</span><span class="p">()</span> <span class="n">defer</span><span class="o">.</span><span class="n">DeferredList</span><span class="p">([</span> <span class="n">coop</span><span class="o">.</span><span class="n">coiterate</span><span class="p">(</span><span class="n">my_task</span><span class="p">(</span><span class="s1">'One'</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">)),</span> <span class="n">coop</span><span class="o">.</span><span class="n">coiterate</span><span class="p">(</span><span class="n">my_task</span><span class="p">(</span><span class="s1">'Two'</span><span class="p">,</span> <span class="n">work_queue</span><span class="p">)),</span> <span class="p">])</span><span class="o">.</span><span class="n">addCallback</span><span class="p">(</span><span class="k">lambda</span> <span class="n">_</span><span class="p">:</span> <span class="n">reactor</span><span class="o">.</span><span class="n">stop</span><span class="p">())</span> <span class="c1"># run the event loop</span> <span class="n">reactor</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> <span class="k">print</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'Total elapsed time: {et():.1f}'</span><span class="p">)</span> <span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span> <span class="n">main</span><span class="p">()</span> </pre></div> <p>The end result of running this program is the same as the previous two examples, the total time of the program is less than the cummulative time of getting the URLs.</p> <p>Whether you use gevent or Twisted is a matter of personal preference and coding style. Both are powerful libaries that provide mechanisms allowing the programmer to create asynchronous code.</p> </section><section><h2 class="crosshead thin">Conclusion</h2> <p>I hope this has helped you see and understand where and how asynchronous programming can be useful. If you’re writing a program that’s calculating PI to the millionth decimal place, asynchronous code isn’t going to help at all.</p> <p>However, if you’re trying to implement a server, or a program that does a significant amount of IO, it could make a huge difference. It’s a powerful technique that can take your programs to the next level.</p></section><footer></footer></article>https://dbader.org/blog/understanding-asynchronous-programming-in-pythonTue, 13 Jun 2017 00:00:00 GMTHow to Speed Up Python Code Reviews With Lintinghttps://dbader.org/blog/speeding-up-python-code-reviews-with-linting<article><header><h1>How to Speed Up Python Code Reviews With Linting</h1> <p>Ever introduced code reviews to an existing Python code base? It can be awesome, or pure hell…</p> </header><section><figure><img alt="" src="/blog/figures/speeding-up-code-reviews-linting.png" width="1280" height="720"></figure> <p>One fateful Thursday morning I sat down with a fresh cup of coffee, ready to dig in and give some feedback on a fix we wanted to ship before the end of the sprint.</p> <p>When I loaded up the first set of changes into my trusty Sublime Text, my eyes nearly fell out—this was a serious “can’t see the forest for the trees” type of situation:</p> </section><section><h2 class="crosshead thin">The formatting for this Python code was…All. Over. The. Place.</h2> <p>There was no consistency whatsoever in how the code was indented, how braces were positioned… even the spacing between operators inside expressions was seemingly randomized:</p> <div class="codehilite"><pre><span></span><span class="c1">#the worst code ever .</span> <span class="n">value</span> <span class="o">+=</span><span class="mi">10</span><span class="o">*</span> <span class="n">othervalue</span> </pre></div> <p>Ugh.</p> <p>It just seemed so sloppy! And the inconsistent formatting made it really hard to see what the code did, what the intention behind it was.</p> <p>It felt like my brain was 90% occupied with parsing out the code, instead of being able to focus on the bigger picture and to hunt for actual bugs.</p> <p>I must’ve spent at least an hour cleaning up the formatting, before I was able to give any substantial feedback on these changes. It was the most tedious code review of my dev career.</p> <p>My busywork was of little value to the company, too:</p> </section><section><h2 class="crosshead thin">They paid me a software engineer’s salary for nudging around braces and juggling whitespace…</h2> <p>That same day I pulled the whole team together to discuss the mandatory use of a code style checker before code reviews.</p> <p>And guess what? It worked out great.</p> <p>Most developers on the team were using Sublime Text so we all installed the <a href="http://www.sublimelinter.com/en/latest/" target="_blank">SublimeLinter package</a>. It’s the most popular code linting framework for Sublime Text and I like it for its focus, simplicity, and performance.</p> <p>A code linter is a program that analyses your source code for potential errors. Code linters are great at finding “mechanical” issues like syntax errors, structural problems, such as the use of undefined variables, and also best practice or code style violations.</p> <p>SublimeLinter let’s you integrate code linting feedback into your editing environment. Setting up SublimeLinter gives you immediate feedback on your code right when you type it:</p> <figure><img alt="" src="/blog/figures/python-code-linting.jpg" width="958" height="220"></figure> <p>When you install SublimeLinter it doesn’t actually include any linter engines. It’s more like a “meta linter” that lets you integrate various command-line linter binaries like <a href="https://github.com/SublimeLinter/SublimeLinter-flake8" target="_blank">Flake8 (Python)</a> or <a href="https://github.com/SublimeLinter/SublimeLinter-jshint" target="_blank">JSHint (JavaScript)</a> under one roof.</p> <p>The linter binaries do the real work. And that way, SublimeLinter can support more than just one programming language. If you’re doing any kind of full-stack web development, for example, you could install code linters for JavaScript, CSS, Ruby, Go, and Python.</p> <p>SublimeLinter will then pick the right code linter to run on each file you’re editing. Any errors or warnings found by these separate linters would all be integrated with the same look and feel into your Sublime Text editor window by SublimeLinter.</p> <p>And because we were using command line tools through SublimeLinter we were able to set up the same set of code style checks on our CI build server very easily. That way no badly formatted code could slip through the cracks ever again.</p> <p>It made the whole team more productive. And it was great for morale: No more time wasted on nudging braces or juggling whitespace 🙂</p> </section><section><h2>Additional Resources &amp; Links</h2> <p>Here are a couple of extra links to help you get set up with SublimeLinter. I listed the most common linter binaries and linter plugins so you can get started right away:</p> <ul> <li><a href="/products/sublime-python-guide/">My Sublime Python course</a></li> <li><a href="http://www.sublimelinter.com/en/latest/" target="_blank">SublimeLinter docs</a></li> <li><a href="https://github.com/SublimeLinter" target="_blank">All official linter plugins for SublimeLinter</a></li> <li>JavaScript: <a href="https://github.com/SublimeLinter/SublimeLinter-jshint" target="_blank">JSHint</a>, <a href="https://github.com/SublimeLinter/SublimeLinter-flow" target="_blank">Flow</a>, <a href="https://github.com/SublimeLinter/SublimeLinter-jsl" target="_blank">JSL</a>, <a href="https://github.com/SublimeLinter/SublimeLinter-jsxhint" target="_blank">JSXHint</a>, <a href="https://github.com/SublimeLinter/SublimeLinter-jscs" target="_blank">JSCS</a></li> <li>Ruby: <a href="https://github.com/SublimeLinter/SublimeLinter-ruby" target="_blank">Ruby (built-in)</a>, <a href="https://github.com/SublimeLinter/SublimeLinter-rubocop" target="_blank">Rubocop</a></li> <li>Python: <a href="https://github.com/SublimeLinter/SublimeLinter-flake8" target="_blank">Flake8</a>, <a href="https://github.com/SublimeLinter/SublimeLinter-pylint" target="_blank">Pylint</a>, <a href="https://github.com/SublimeLinter/SublimeLinter-pep8" target="_blank">Pep8</a>, <a href="https://github.com/SublimeLinter/SublimeLinter-pyflakes" target="_blank">Pyflakes</a></li> <li>PHP: <a href="https://github.com/SublimeLinter/SublimeLinter-php" target="_blank">PHP</a>, <a href="https://github.com/SublimeLinter/SublimeLinter-phplint" target="_blank">PHPLint</a></li> <li>Go: <a href="https://packagecontrol.io/packages/SublimeLinter-contrib-golint" target="_blank">GoLint</a>, <a href="https://packagecontrol.io/packages/SublimeLinter-contrib-gotype" target="_blank">GoType</a></li> <li>Lua: <a href="https://github.com/SublimeLinter/SublimeLinter-luacheck" target="_blank">LuaCheck</a>, <a href="https://github.com/SublimeLinter/SublimeLinter-lua" target="_blank">Lua (built-in)</a></li> <li>Haskell: <a href="https://github.com/SublimeLinter/SublimeLinter-ghc" target="_blank">Ghc</a></li> <li>C++: <a href="https://github.com/SublimeLinter/SublimeLinter-cppcheck" target="_blank">CppCheck</a>, <a href="https://github.com/SublimeLinter/SublimeLinter-cpplint" target="_blank">CppLint</a></li> <li>CSS: <a href="https://github.com/SublimeLinter/SublimeLinter-csslint" target="_blank">CssLint</a></li> <li>HTML: <a href="https://github.com/SublimeLinter/SublimeLinter-html-tidy" target="_blank">HTMLTidy</a></li> <li>Java: <a href="https://github.com/SublimeLinter/SublimeLinter-javac" target="_blank">Java (built-in)</a></li> <li>Plaintext: <a href="https://github.com/amperser/SublimeLinter-contrib-proselint" target="_blank">Proselint</a></li> </ul></section><footer></footer></article>https://dbader.org/blog/speeding-up-python-code-reviews-with-lintingThu, 08 Jun 2017 00:00:00 GMTA Story About Python Masteryhttps://dbader.org/blog/a-story-about-python-mastery<article><header><h1>A Story About Python Mastery</h1> <p>A couple of years ago I’d become quite interested in martial arts. Hours upon hours of watching “The Karate Kid” growing up must’ve taken their toll on me…</p> </header><section><figure><img alt="" src="/blog/figures/story-about-python-mastery.png" width="1280" height="720"></figure> <p>And so, I found myself at this smelly little gym, joining my first couple of karate practice sessions.</p> <p>(By the way, my “Mr. Miyagi” wasn’t the fatherly philosopher from the Karate Kid movies—our sensei was a complete geek, working a day job as a Borland Delphi programmer somewhere. I liked him.)</p> <p>So anyway, here I was at this dingy gym, working hard to learn how to count in Japanese and getting my hand-eye coordination under control…</p> <p>(You know, karate practice actually feels more like learning to dance than learning how to fight. At least when you’re a beginner.)</p> <p>Moments later my friend kicks me in the face because I turned left when I should’ve turned right—</p> <p>My interested in karate waned quickly after that.</p> <p>Yeah…I’m a lover, not a fighter.</p> <p>Why am I telling you this? Well, the question came up in a recent email exchange:</p> </section><section><h2 class="crosshead thin">“How does one MASTER the skill of programming Python?”</h2> <p>I like to think mastering programming as a skill is quite similar to mastering a physical skill like karate. (Although I’ve had more success with the former.)</p> <p>Here, let me explain.</p> <p>With both, it takes a long time to build up the right foundation. But once “muscle memory” starts kicking in, your progress can skyrocket. It’s all about making it through that first rough patch of slow learning progress without losing your motivation.</p> <p>Mastering a programming language means lifelong learning. The topic is fractal—there’s always a way to expand your knowledge in some obscure way. One can hit critical mass in terms of knowledge and be called an expert, but it’s unlikely a single person will “know it all.”</p> <p>A seasoned programmer acts deliberately and with an economy of movement that a beginner can’t yet understand. Biological differences like age, “IQ”, play less of a role. The more experienced dev still codes circles around the eager newcomer.</p> <p>There’s road maps but no “one true path” to mastery. Learning progress will depend highly on the motivation and drive of the individual, and the peers they surround themselves with. Mentorship and community play the biggest role in becoming successful.</p> <p>Like martial “arts” programming is more of an art than a science. It’s a creative endeavour rather than a strictly mechanical affair. Brute force and applying 10,000 “<code>IF</code> this <code>THEN</code> that” rules might get one a job, but doesn’t lead to the true joy of programming.</p> <p>(I swear one day I <em>will</em> create a Bob Ross-like show called <em>The Joy of Programming</em>: “Let’s put some little curly braces over here…and here…and there.”)</p> <p>Mastering a skill like programming seeps into all areas of your life. Just like building physical skills will increase confidence, so will mastering programming. It leads to a sense of accomplishment, a deep satisfaction, and confidence through recognition.</p> <p>Alright, that’s my (philosophical) update for the week.</p> <p>If you’d like to avoid getting kicked in the head learning Python, then check out <a href="/products/">some of the Python training products I offer here on dbader.org</a>.</p></section><footer></footer></article>https://dbader.org/blog/a-story-about-python-masteryThu, 08 Jun 2017 00:00:00 GMTRecords, Structs, and Data Transfer Objects in Pythonhttps://dbader.org/blog/records-structs-and-data-transfer-objects-in-python<article><header><h1>Records, Structs, and Data Transfer Objects in Python</h1> <p>How to implement records, structs, and “plain old data objects” in Python using only built-in data types and classes from the standard library.</p> </header><section><figure><img alt="" src="/blog/figures/records-structs-in-python.png" width="1280" height="720"></figure> <p>Compared to arrays, <a href="https://en.wikipedia.org/wiki/Record_(computer_science)" target="_blank">record data structures</a> provide a fixed number of fields, each field can have a name, and may have a different type.</p> <p>I’m using the definition of a “record” loosely in this article. For example, I’m also going to discuss types like Python’s built-in <code>tuple</code> that may or may not be considered “records” in a strict sense because they don’t provide named fields.</p> <p>Python provides several data types you can use to implement records, structs, and data transfer objects. In this article you’ll get a quick look at each implementation and its unique characteristics. At the end you’ll find a summary and a decision making guide that will help you make your own pick.</p> <p>Alright, let’s get started:</p> </section><section><h2>✅ The <a href="https://docs.python.org/3.6/library/stdtypes.html#dict" target="_blank"><code>dict</code></a> Built-in</h2> <p>Python <a href="https://dbader.org/blog/python-dictionaries-maps-and-hashtables">dictionaries</a> store an arbitrary number of objects, each identified by a unique key. Dictionaries are often also called “maps” or “associative arrays” and allow the efficient lookup, insertion, and deletion of any object associated with a given key.</p> <p>Using dictionaries as a record data type or data object in Python is possible. Dictionaries are easy to create in Python as they have their own syntactic sugar built into the language in the form of dictionary literals. The dictionary syntax is concise and quite convenient to type.</p> <p>Data objects created using dictionaries are mutable and there’s little protection against misspelled field names, as fields can be added and removed freely at any time. Both of these properties can introduce surprising bugs and there’s always a trade-off to be made between convenience and error resilience.</p> <div class="codehilite"><pre><span></span><span class="n">car1</span> <span class="o">=</span> <span class="p">{</span> <span class="s1">'color'</span><span class="p">:</span> <span class="s1">'red'</span><span class="p">,</span> <span class="s1">'mileage'</span><span class="p">:</span> <span class="mf">3812.4</span><span class="p">,</span> <span class="s1">'automatic'</span><span class="p">:</span> <span class="bp">True</span><span class="p">,</span> <span class="p">}</span> <span class="n">car2</span> <span class="o">=</span> <span class="p">{</span> <span class="s1">'color'</span><span class="p">:</span> <span class="s1">'blue'</span><span class="p">,</span> <span class="s1">'mileage'</span><span class="p">:</span> <span class="mf">40231.0</span><span class="p">,</span> <span class="s1">'automatic'</span><span class="p">:</span> <span class="bp">False</span><span class="p">,</span> <span class="p">}</span> <span class="c1"># Dicts have a nice repr:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car2</span> <span class="p">{</span><span class="s1">'color'</span><span class="p">:</span> <span class="s1">'blue'</span><span class="p">,</span> <span class="s1">'automatic'</span><span class="p">:</span> <span class="bp">False</span><span class="p">,</span> <span class="s1">'mileage'</span><span class="p">:</span> <span class="mf">40231.0</span><span class="p">}</span> <span class="c1"># Get mileage:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car2</span><span class="p">[</span><span class="s1">'mileage'</span><span class="p">]</span> <span class="mf">40231.0</span> <span class="c1"># Dicts are mutable:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car2</span><span class="p">[</span><span class="s1">'mileage'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">12</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car2</span><span class="p">[</span><span class="s1">'windshield'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'broken'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car2</span> <span class="p">{</span><span class="s1">'windshield'</span><span class="p">:</span> <span class="s1">'broken'</span><span class="p">,</span> <span class="s1">'color'</span><span class="p">:</span> <span class="s1">'blue'</span><span class="p">,</span> <span class="s1">'automatic'</span><span class="p">:</span> <span class="bp">False</span><span class="p">,</span> <span class="s1">'mileage'</span><span class="p">:</span> <span class="mi">12</span><span class="p">}</span> <span class="c1"># No protection against wrong field names,</span> <span class="c1"># or missing/extra fields:</span> <span class="n">car3</span> <span class="o">=</span> <span class="p">{</span> <span class="s1">'colr'</span><span class="p">:</span> <span class="s1">'green'</span><span class="p">,</span> <span class="s1">'automatic'</span><span class="p">:</span> <span class="bp">False</span><span class="p">,</span> <span class="s1">'windshield'</span><span class="p">:</span> <span class="s1">'broken'</span><span class="p">,</span> <span class="p">}</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3.6/library/stdtypes.html#tuple" target="_blank"><code>tuple</code></a> Built-in</h2> <p>Python’s tuples are a simple data structure for grouping arbitrary objects. <a href="https://www.youtube.com/watch?v=p9ppfvHv2Us" target="_blank">Tuples are immutable</a>—they cannot be modified once they’ve been created.</p> <p>Performancewise, tuples take up <a href="https://github.com/python/cpython/blob/1a5856bf9295fa73995898d576e0bedf016aee1f/Include/tupleobject.h#L10-L34" target="_blank">slightly less memory</a> than <a href="https://github.com/python/cpython/blob/b879fe82e7e5c3f7673c9a7fa4aad42bd05445d8/Include/listobject.h#L4-L41" target="_blank">lists in CPython</a> and they’re faster to construct at instantiation time. As you can see in the bytecode disassembly below, constructing a tuple constant takes a single <code>LOAD_CONST</code> opcode while constructing a list object with the same contents requires several more operations:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">dis</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dis</span><span class="o">.</span><span class="n">dis</span><span class="p">(</span><span class="nb">compile</span><span class="p">(</span><span class="s2">"(23, 'a', 'b', 'c')"</span><span class="p">,</span> <span class="s1">''</span><span class="p">,</span> <span class="s1">'eval'</span><span class="p">))</span> <span class="mi">1</span> <span class="mi">0</span> <span class="n">LOAD_CONST</span> <span class="mi">4</span> <span class="p">((</span><span class="mi">23</span><span class="p">,</span> <span class="s1">'a'</span><span class="p">,</span> <span class="s1">'b'</span><span class="p">,</span> <span class="s1">'c'</span><span class="p">))</span> <span class="mi">3</span> <span class="n">RETURN_VALUE</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dis</span><span class="o">.</span><span class="n">dis</span><span class="p">(</span><span class="nb">compile</span><span class="p">(</span><span class="s2">"[23, 'a', 'b', 'c']"</span><span class="p">,</span> <span class="s1">''</span><span class="p">,</span> <span class="s1">'eval'</span><span class="p">))</span> <span class="mi">1</span> <span class="mi">0</span> <span class="n">LOAD_CONST</span> <span class="mi">0</span> <span class="p">(</span><span class="mi">23</span><span class="p">)</span> <span class="mi">3</span> <span class="n">LOAD_CONST</span> <span class="mi">1</span> <span class="p">(</span><span class="s1">'a'</span><span class="p">)</span> <span class="mi">6</span> <span class="n">LOAD_CONST</span> <span class="mi">2</span> <span class="p">(</span><span class="s1">'b'</span><span class="p">)</span> <span class="mi">9</span> <span class="n">LOAD_CONST</span> <span class="mi">3</span> <span class="p">(</span><span class="s1">'c'</span><span class="p">)</span> <span class="mi">12</span> <span class="n">BUILD_LIST</span> <span class="mi">4</span> <span class="mi">15</span> <span class="n">RETURN_VALUE</span> </pre></div> <p>However you shouldn’t place too much emphasis on these differences. In practice the performance difference will often be negligible and trying to squeeze out extra performance out of a program by switching from lists to tuples will likely be the wrong approach.</p> <p>A potential downside of plain tuples is that the data you store in them can only be pulled out by accessing it through integer indexes. You can’t give names to individual properties stored in a tuple. This can impact code readability.</p> <p>Also, a tuple is always an ad-hoc structure. It’s difficult to ensure that two tuples have the same number of fields and the same properties stored on them.</p> <p>This makes it easy to introduce “slip-of-the-mind” bugs by mixing up the field order, for example. Therefore I would recommend you keep the number of fields stored in a tuple as low as possible.</p> <div class="codehilite"><pre><span></span><span class="c1"># Fields: color, mileage, automatic</span> <span class="n">car1</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'red'</span><span class="p">,</span> <span class="mf">3812.4</span><span class="p">,</span> <span class="bp">True</span><span class="p">)</span> <span class="n">car2</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'blue'</span><span class="p">,</span> <span class="mf">40231.0</span><span class="p">,</span> <span class="bp">False</span><span class="p">)</span> <span class="c1"># Tuple instances have a nice repr:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car1</span> <span class="p">(</span><span class="s1">'red'</span><span class="p">,</span> <span class="mf">3812.4</span><span class="p">,</span> <span class="bp">True</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car2</span> <span class="p">(</span><span class="s1">'blue'</span><span class="p">,</span> <span class="mf">40231.0</span><span class="p">,</span> <span class="bp">False</span><span class="p">)</span> <span class="c1"># Get mileage:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car2</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="mf">40231.0</span> <span class="c1"># Tuples are immutable:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car2</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="mi">12</span> <span class="ne">TypeError</span><span class="p">:</span> <span class="s2">"'tuple' object does not support item assignment"</span> <span class="c1"># No protection against missing/extra fields</span> <span class="c1"># or a wrong order:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car3</span> <span class="o">=</span> <span class="p">(</span><span class="mf">3431.5</span><span class="p">,</span> <span class="s1">'green'</span><span class="p">,</span> <span class="bp">True</span><span class="p">,</span> <span class="s1">'silver'</span><span class="p">)</span> </pre></div> </section><section><h2>✅ Writing a Custom Class</h2> <p>Classes allow you to define reusable “blueprints” for data objects to ensure each object provides the same set of fields.</p> <p>Using regular Python classes as record data types is feasible, but it also takes manual work to get the convenience features of other implementations. For example, adding new fields to the <code>__init__</code> constructor is verbose and takes time.</p> <p>Also, the <a href="https://www.youtube.com/watch?v=aIdzBGzaxUo" target="_blank">default string representation</a> for objects instantiated from custom classes is not very helpful. To fix that you may have to add your own <code>__repr__</code> method, which again is usually quite verbose and must be updated every time you add a new field.</p> <p>Fields stored on classes are mutable and new fields can be added freely, which may or may not be what you intend. It’s possible to provide more access control and to create read-only fields using the <a href="https://docs.python.org/3/library/functions.html#property" target="_blank">@property decorator</a>, but this requires writing more glue code.</p> <p>Writing a custom class is a great option whenever you’d like to add business logic and <em>behavior</em> to your record objects using methods. But this means these objects are technically no longer plain data objects.</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Car</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">color</span><span class="p">,</span> <span class="n">mileage</span><span class="p">,</span> <span class="n">automatic</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">color</span> <span class="o">=</span> <span class="n">color</span> <span class="bp">self</span><span class="o">.</span><span class="n">mileage</span> <span class="o">=</span> <span class="n">mileage</span> <span class="bp">self</span><span class="o">.</span><span class="n">automatic</span> <span class="o">=</span> <span class="n">automatic</span> <span class="n">car1</span> <span class="o">=</span> <span class="n">Car</span><span class="p">(</span><span class="s1">'red'</span><span class="p">,</span> <span class="mf">3812.4</span><span class="p">,</span> <span class="bp">True</span><span class="p">)</span> <span class="n">car2</span> <span class="o">=</span> <span class="n">Car</span><span class="p">(</span><span class="s1">'blue'</span><span class="p">,</span> <span class="mf">40231.0</span><span class="p">,</span> <span class="bp">False</span><span class="p">)</span> <span class="c1"># Get the mileage:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car2</span><span class="o">.</span><span class="n">mileage</span> <span class="mf">40231.0</span> <span class="c1"># Classes are mutable:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car2</span><span class="o">.</span><span class="n">mileage</span> <span class="o">=</span> <span class="mi">12</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car2</span><span class="o">.</span><span class="n">windshield</span> <span class="o">=</span> <span class="s1">'broken'</span> <span class="c1"># String representation is not very useful</span> <span class="c1"># (must add a manually written __repr__ method):</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car1</span> <span class="o">&lt;</span><span class="n">Car</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x1081e69e8</span><span class="o">&gt;</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/collections.html#collections.namedtuple" target="_blank">collections.namedtuple</a> Class</h2> <p>The <code>namedtuple</code> class available in Python 2.6+ provides an extension of the built-in <code>tuple</code> data type. Similarly to defining a custom class, using <code>namedtuple</code> allows you to define reusable “blueprints” for your records that ensure the correct field names are used.</p> <p>Namedtuples are immutable just like regular tuples. This means you cannot add new fields or modify existing fields after the namedtuple instance was created.</p> <p>Besides that, namedtuples are, well…named tuples. Each object stored in them can be accessed through a unique identifier. This frees you from having to remember integer indexes, or resorting to workarounds like defining integer constants as mnemonics for your indexes.</p> <p>Namedtuple objects are implemented as regular Python classes internally. When it comes to memory usage they are also “better” than regular classes and just as memory efficient as regular tuples:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">namedtuple</span> <span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">sys</span> <span class="kn">import</span> <span class="n">getsizeof</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">p1</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span><span class="s1">'Point'</span><span class="p">,</span> <span class="s1">'x y z'</span><span class="p">)(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">p2</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">getsizeof</span><span class="p">(</span><span class="n">p1</span><span class="p">)</span> <span class="mi">72</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">getsizeof</span><span class="p">(</span><span class="n">p2</span><span class="p">)</span> <span class="mi">72</span> </pre></div> <p>Namedtuples can be an easy way to clean up your code and to make it more readable by enforcing a better structure for your data.</p> <p>I find that going from ad-hoc data types like dictionaries with a fixed format to namedtuples helps me express the intent of my code more clearly. Often when I apply this refactoring I magically come up with a better solution for the problem I’m facing.</p> <p>Using namedtuples over unstructured tuples and dicts can also make my coworkers’ lives easier because namedtuples make the data passed around “self-documenting”, at least to a degree.</p> <p>For more information and code examples, check out <a href="writing-clean-python-with-namedtuples">my tutorial on namedtuples here on dbader.org</a>.</p> <div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">namedtuple</span> <span class="n">Car</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span><span class="s1">'Car'</span> <span class="p">,</span> <span class="s1">'color mileage automatic'</span><span class="p">)</span> <span class="n">car1</span> <span class="o">=</span> <span class="n">Car</span><span class="p">(</span><span class="s1">'red'</span><span class="p">,</span> <span class="mf">3812.4</span><span class="p">,</span> <span class="bp">True</span><span class="p">)</span> <span class="c1"># Instances have a nice repr:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car1</span> <span class="n">Car</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="s1">'red'</span><span class="p">,</span> <span class="n">mileage</span><span class="o">=</span><span class="mf">3812.4</span><span class="p">,</span> <span class="n">automatic</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="c1"># Accessing fields</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car1</span><span class="o">.</span><span class="n">mileage</span> <span class="mf">3812.4</span> <span class="c1"># Fields are immtuable:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car1</span><span class="o">.</span><span class="n">mileage</span> <span class="o">=</span> <span class="mi">12</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="s2">"can't set attribute"</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car1</span><span class="o">.</span><span class="n">windshield</span> <span class="o">=</span> <span class="s1">'broken'</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="s2">"'Car' object has no attribute 'windshield'"</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/typing.html#typing.NamedTuple" target="_blank">typing.NamedTuple</a> Class</h2> <p>This class added in Python 3.6 is the younger sibling of <code>collections.namedtuple</code>. It is very similar to <code>namedtuple</code>, the main difference is an updated syntax for defining new record types and added support for type hints.</p> <p>Please note that type annotations are not enforced without a separate <a href="https://www.youtube.com/watch?v=2xWhaALHTvU" target="_blank">type checking tool like mypy</a>—but even without tool support they can provide useful hints to other programmers (or be terribly confusing if the type hints get out of date.)</p> <div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">NamedTuple</span> <span class="k">class</span> <span class="nc">Car</span><span class="p">(</span><span class="n">NamedTuple</span><span class="p">):</span> <span class="n">color</span><span class="p">:</span> <span class="nb">str</span> <span class="n">mileage</span><span class="p">:</span> <span class="nb">float</span> <span class="n">automatic</span><span class="p">:</span> <span class="nb">bool</span> <span class="n">car1</span> <span class="o">=</span> <span class="n">Car</span><span class="p">(</span><span class="s1">'red'</span><span class="p">,</span> <span class="mf">3812.4</span><span class="p">,</span> <span class="bp">True</span><span class="p">)</span> <span class="c1"># Instances have a nice repr</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car1</span> <span class="n">Car</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="s1">'red'</span><span class="p">,</span> <span class="n">mileage</span><span class="o">=</span><span class="mf">3812.4</span><span class="p">,</span> <span class="n">automatic</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="c1"># Accessing fields</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car1</span><span class="o">.</span><span class="n">mileage</span> <span class="mf">3812.4</span> <span class="c1"># Fields are immutable</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car1</span><span class="o">.</span><span class="n">mileage</span> <span class="o">=</span> <span class="mi">12</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="s2">"can't set attribute"</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car1</span><span class="o">.</span><span class="n">windshield</span> <span class="o">=</span> <span class="s1">'broken'</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="s2">"'Car' object has no attribute 'windshield'"</span> <span class="c1"># Type annotations are not enforced without</span> <span class="c1"># a separate type checking tool like mypy:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">Car</span><span class="p">(</span><span class="s1">'red'</span><span class="p">,</span> <span class="s1">'NOT_A_FLOAT'</span><span class="p">,</span> <span class="mi">99</span><span class="p">)</span> <span class="n">Car</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="s1">'red'</span><span class="p">,</span> <span class="n">mileage</span><span class="o">=</span><span class="s1">'NOT_A_FLOAT'</span><span class="p">,</span> <span class="n">automatic</span><span class="o">=</span><span class="mi">99</span><span class="p">)</span> </pre></div> </section><section><h2>⚠️ The <a href="https://docs.python.org/3/library/struct.html#struct.Struct" target="_blank">struct.Struct</a> Class</h2> <p>This class performs conversions between Python values and C structs serialized into Python <code>bytes</code> objects. It can be used to handle binary data stored in files or from network connections, for example.</p> <p>Structs are defined using a <a href="https://docs.python.org/3/library/struct.html#format-characters" target="_blank">format strings-like mini language</a> that allows you to define the arrangement of various C data types, like <code>char</code>, <code>int</code>, and <code>long</code>, as well as their <code>unsigned</code> variants.</p> <p>The <code>struct</code> module is seldom used to represent data objects that are meant to be handled purely inside Python code. They’re intended primarily as a data exchange format, rather than a way of holding data in memory that’s only used by Python code.</p> <p>In some cases packing primitive data into structs may use less memory than keeping it in other data types—but that would be a quite advanced (and probably unnecessary) optimization.</p> <div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">struct</span> <span class="kn">import</span> <span class="n">Struct</span> <span class="n">MyStruct</span> <span class="o">=</span> <span class="n">Struct</span><span class="p">(</span><span class="s1">'i?f'</span><span class="p">)</span> <span class="n">data</span> <span class="o">=</span> <span class="n">MyStruct</span><span class="o">.</span><span class="n">pack</span><span class="p">(</span><span class="mi">23</span><span class="p">,</span> <span class="bp">False</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">)</span> <span class="c1"># All you get is a blob of data:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">data</span> <span class="sa">b</span><span class="s1">'</span><span class="se">\x17\x00\x00\x00\x00\x00\x00\x00\x00\x00</span><span class="s1">(B'</span> <span class="c1"># Data blobs can be unpacked again:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">MyStruct</span><span class="o">.</span><span class="n">unpack</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="p">(</span><span class="mi">23</span><span class="p">,</span> <span class="bp">False</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">)</span> </pre></div> </section><section><h2>⚠️ The <a href="https://docs.python.org/3/library/types.html#types.SimpleNamespace" target="_blank">types.SimpleNamespace</a> Class</h2> <p>Here’s one more “esoteric” choice for implementing data objects in Python. This class was added in Python 3.3 and it provides attribute access to its namespace. It also includes a meaningful <code>__repr__</code> by default.</p> <p>As its name proclaims, <code>SimpleNamespace</code> is simple—it’s basically a glorified dictionary that allows attribute access and prints nicely. Attributes can be added, modified, and deleted freely.</p> <div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">types</span> <span class="kn">import</span> <span class="n">SimpleNamespace</span> <span class="n">car1</span> <span class="o">=</span> <span class="n">SimpleNamespace</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="s1">'red'</span><span class="p">,</span> <span class="n">mileage</span><span class="o">=</span><span class="mf">3812.4</span><span class="p">,</span> <span class="n">automatic</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="c1"># The default repr:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car1</span> <span class="n">namespace</span><span class="p">(</span><span class="n">automatic</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">color</span><span class="o">=</span><span class="s1">'red'</span><span class="p">,</span> <span class="n">mileage</span><span class="o">=</span><span class="mf">3812.4</span><span class="p">)</span> <span class="c1"># Instances are mutable</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car1</span><span class="o">.</span><span class="n">mileage</span> <span class="o">=</span> <span class="mi">12</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car1</span><span class="o">.</span><span class="n">windshield</span> <span class="o">=</span> <span class="s1">'broken'</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">del</span> <span class="n">car1</span><span class="o">.</span><span class="n">automatic</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">car1</span> <span class="n">namespace</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="s1">'red'</span><span class="p">,</span> <span class="n">mileage</span><span class="o">=</span><span class="mi">12</span><span class="p">,</span> <span class="n">windshield</span><span class="o">=</span><span class="s1">'broken'</span><span class="p">)</span> </pre></div> </section><section><h2>Which type should I use for data objects in Python?</h2> <p>As you’ve seen there’s quite a number of different options to implement records or data objects in Python. Generally your decision will depend on your use case:</p> <ul> <li><strong>You only have a few (2-3) fields</strong>: Using a plain tuple object may be okay because the field order is easy to remember or field names are superfluous. For example, think of an <code>(x, y, z)</code> point in 3D space.</li> <li><strong>You need immutable fields</strong>: In this case plain tuples, <code>collections.namedtuple</code>, <code>typing.NamedTuple</code> would all make good options for implementing this type of data object.</li> <li><strong>You need to lock down field names to avoid typos</strong>: <code>collections.namedtuple</code> and <code>typing.NamedTuple</code> are your friends.</li> <li><strong>You want to keep things simple</strong>: A plain dictionary object might be a good choice due to the convenient syntax that closely resembles JSON.</li> <li><strong>You need full control over your data structure</strong>: It’s time to write a custom class with <code>@property</code> setters and getters.</li> <li><strong>You need to add behavior (methods) to the object</strong>: You should write a custom class. Either from scratch or by extending <code>collections.namedtuple</code> or <code>typing.NamedTuple</code>.</li> <li><strong>You need to pack data tightly to serialize it to disk or send it over the network</strong>: Time to bust out <code>struct.Struct</code>, this is a great use case for it.</li> </ul> <p>If you’re looking for a safe default choice, my general recommendation for implementing a plain record, struct, or data object in Python would be to:</p> <ul> <li>use <code>collections.namedtuple</code> in Python 2.x; and</li> <li>its younger sibling <code>typing.NamedTuple</code> in Python 3.</li> </ul> <p><em><a href="fundamental-data-structures-in-python">Read the full “Fundamental Data Structures in Python” article series here</a>. This article is missing something or you found an error? Help a brother out and leave a comment below.</em></p></section><footer></footer></article>https://dbader.org/blog/records-structs-and-data-transfer-objects-in-pythonTue, 06 Jun 2017 00:00:00 GMTAn Overview of Python’s “ipaddress” Modulehttps://dbader.org/blog/python-ipaddress-module<article><header><h1>An Overview of Python’s “ipaddress” Module</h1> <p>An introduction to the ipaddress module available on Python 3.3+ for manipulation of IPv4 and IPv6 addresses.</p> </header><section><figure><img alt="Python 3 ipaddress Module Overview" src="/blog/figures/ipaddress-module-overview.png" width="1280" height="720"></figure> <p>In this article we’ll take a look at the <a href="https://docs.python.org/3/library/ipaddress.html" target="_blank"><code>ipaddress</code> module</a> that is available on Python 3.3 and above. This tutorial is intended to serve as a handy reference for any network engineer wondering how to parse and work with IP addresses in Python.</p> <p>In this overview article you’ll learn:</p> <ul> <li>What the differnce between IPv4 and IPv6 addresses is.</li> <li>How to work with IPv4 addresses using Python’s <code>ipaddress</code> module.</li> <li>How to work with IPv6 addresses using Python’s <code>ipaddress</code> module.</li> </ul> </section><section><h2>IPv4 vs IPv6 Addresses – A Primer</h2> <p>At a high level, IPv4 and IPv6 addresses are used for like purposes and functions. However, since there are major differences in the address structure for each protocol, this tutorial has separated into separate sections, one each for IPv4 and IPv6.</p> <p>In today’s Internet, the IPv4 protocol controls the majority of IP processing and will remain so for the near future. The enhancements in scale and functionality that come with IPv6 are necessary for the future of the Internet and adoption is progressing. The adoption rate, however, remains slow to this date.</p> <p>An <strong>IPv4 address</strong> is composed of 32 bits, organized into four eight bit groupings referred to as “octets”. The word “octet” is used to identify an eight-bit structure in place of the more common term “byte”, but they carry the same definition. The four octets are referred to as octet1, octet2, octet3, and octet4. This is a “dotted decimal” format where each eight-bit octet can have a decimal value based on eight bits from zero to 255.</p> <div class="callout-box"> <p><strong>IPv4 address example</strong>: <code>192.168.100.10</code></p> <p><strong>IPv4 address example (CIDR notation)</strong>: <code>192.168.100.10/24</code></p> <p>The <code>/24</code> is CIDR notation to indicate that leading 24 of the 32 bits are used to identify the network portion of the address. Remembering that each octet is 8 bits long, this means that the first three octets (3 × 8 = 24) identify the network (192.168.100.x) and the remaining eight bits of the address identify the node (x.x.x.10).</p> <p>CIDR notation can be anything from <code>/8</code> bits through to <code>/30</code> bits, with an occasional <code>/32</code> bits (<code>/31</code> is invalid), but <code>/24</code> is often used. For example, your home network, or your school or company network is most likely identified with a <code>/24</code> CIDR.</p> <p>An older format for expressing the network identification is a network mask where the CIDR is expressed as a separate dotted decimal number. For example, a <code>/24</code> CIDR equates to a network mask of <code>255.255.255.0</code>.</p> </div> <p>An <strong>IPv6 address</strong> is 128 bits long, which is a significant increase over the 32 bits in an IPv4 address. There are many differences between IPv4 and IPv6, but the notable difference is in the addressing structure. The additional length provides an exponential increase in the number of networks and host that can be supported.</p> <div class="callout-box"> <p><strong>IPv6 address example</strong>: <code>2001:db8:abcd:100::1/64</code></p> <p>Where the IPv4 address uses a dotted decimal format, the IPv6 protocol uses hexadecimal notation. Each position in an IPv6 address represents four bits with a value from <code>0</code> to <code>f</code>, organized as follows:</p> <ul> <li>The 128 bits are divided into 8 groupings of 16 bits each separated by colons. A group is referred to as a “quartet” or “hextet” each with four hexadecimal characters (4 hex characters times 4 bits = 16 bits). In the above example, the first quartet is “<code>2001</code>”.</li> <li>Leading zeros in any quartet are suppressed/condensed. In the above example, the second quartet is “<code>db8</code>”, which is actually “<code>0db8</code>”” with the leading zero suppressed. The last quartet is “<code>1</code>”, which is actually “<code>0001</code>”” with three leading zeros suppressed.</li> <li>If a quartet contains all zeros, it is suppressed to a single zero. For example: a quartet with “<code>:0000:</code>” would be compressed to “<code>:0:</code>”.</li> <li>If an address contains a contiguous string of quartets that are all zeros, the contiguous string of zeros is condensed and represented with double colons. In the above example, the double colon represents three all zero quartets, or “<code>:0000:0000:0000:</code>” condensed to “<code>::</code>”. Since the example address has five quartets with values, the number of condensed quartets must be three (eight total minus five populated).</li> </ul> <p>All IPv6 address structures used CIDR notation to determine how many of the leading bits are used for network identification with the balance used for host/interface identification. Given 128 bits, many options are available.</p> </div> </section><section><h2>Python’s <code>ipaddress</code> Module and IPv4 Addresses</h2> <p>The <code>ipaddress</code> module is designed around CIDR notation, which is recommended because of its brevity and ease of use. The <code>ipaddress</code> module also includes methods to revert to a network mask if required.</p> <p>The original definition of IPv4 addresses includes a “class” that is defined by address ranges in the first octet. The <code>ipaddress</code> module does not recognize IPv4 classes and is therefore not included in this tutorial.</p> <p>The <code>ipaddress</code> module includes three specific IPv4 address object types:</p> <ol> <li>a “host” or an individual address object that does not include CIDR notation,</li> <li>an individual interface address object that includes CIDR notation, and</li> <li>and a network address object that refers to the range of IP addresses for the entire network.</li> </ol> <p>The major difference between a “host” and an “interface” is that a host or <code>ip_address</code> object does not include CIDR notation, whereas an <code>ip_interface</code> object includes the CIDR notation:</p> <ul> <li>The <code>ip_address</code> object is most useful when working with IP packets that do not need nor use CIDR notation.</li> <li>The <code>ip_interface</code> object is most useful when working with node and interface identification for connection to an IP network which must include network/subnet identification.</li> <li>The <code>ip_network</code> object includes all addresses within a network and is most useful for network identification.</li> </ul> <p><strong>Creating IPv4 Host Address Objects with <code>ipaddress</code></strong>:</p> <p>The <code>ipaddress.ip_address()</code> factory function is used to create an <code>ip_address</code> object. This automatically determines whether to create an IPv4 or IPv6 address based on the passed-in value (IPv6 addressing will be discussed at a latter point in this tutorial). As noted above, this object represents an IP Address as found in a packet traversing a network where CIDR is not required.</p> <p>In many cases, the value used to create an ip_address object will be a string in the IPv4 dotted decimal format as per this example:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">ipaddress</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_ip</span> <span class="o">=</span> <span class="n">ipaddress</span><span class="o">.</span><span class="n">ip_address</span><span class="p">(</span><span class="s1">'192.168.100.10'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_ip</span> <span class="n">IPv4Address</span><span class="p">(</span><span class="s1">'192.168.100.10'</span><span class="p">)</span> </pre></div> <p>Alternatively, the IPv4 address may be entered in binary, as a decimal value of the full 32 bit binary value, or in hexadecimal format as per this example:</p> <div class="codehilite"><pre><span></span><span class="c1"># All 32 binary bits can be used to create an IPv4 address:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ipaddress</span><span class="o">.</span><span class="n">ip_address</span><span class="p">(</span><span class="mb">0b11000000101010000110010000001010</span><span class="p">)</span> <span class="n">IPv4Address</span><span class="p">(</span><span class="s1">'192.168.100.10'</span><span class="p">)</span> <span class="c1"># The decimal value of the 32 bit binary number can also be used:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ipaddress</span><span class="o">.</span><span class="n">ip_address</span><span class="p">(</span><span class="mi">3232261130</span><span class="p">)</span> <span class="n">IPv4Address</span><span class="p">(</span><span class="s1">'192.168.100.10'</span><span class="p">)</span> <span class="c1"># As can the hexadecimal value of the 32 bits:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ipaddress</span><span class="o">.</span><span class="n">ip_address</span><span class="p">(</span><span class="mh">0xC0A8640A</span><span class="p">)</span> <span class="n">IPv4Address</span><span class="p">(</span><span class="s1">'192.168.100.10'</span><span class="p">)</span> </pre></div> <p>The first example uses the full 32 bit address, and the second example is the decimal value of the 32 bit address. Both are unwieldy, error-prone and of limited value. The third example uses a hexadecimal value which can be useful as most packet formats from parsing or sniffing are represented in hexadecimal format.</p> <p><strong>Creating IPv4 Interface Address Objects with <code>ipaddress</code></strong>:</p> <p>The <code>ipaddress.ip_interface()</code> factory function is used to create an <code>ip_interface</code> object, which automatically determines whether to create an IPv4 or IPv6 address based on the passed-in value (IPv6 addressing will be discussed at a latter point in this tutorial).</p> <p>As previously discussed, the ip_interface object represents the ip address found on a host or network interface where the CIDR (or mask) is required for proper handling of the packet.</p> <div class="codehilite"><pre><span></span><span class="c1"># An ip_interface object is used to represent IP addressing</span> <span class="c1"># for a host or router interface, including the CIDR:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_ip</span> <span class="o">=</span> <span class="n">ipaddress</span><span class="o">.</span><span class="n">ip_interface</span><span class="p">(</span><span class="s1">'192.168.100.10/24'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_ip</span> <span class="n">IPv4Interface</span><span class="p">(</span><span class="s1">'192.168.100.10/24'</span><span class="p">)</span> <span class="c1"># This method translates the CIDR into a mask as would normally</span> <span class="c1"># be used on a host or router interface</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_ip</span><span class="o">.</span><span class="n">netmask</span> <span class="n">IPv4Address</span><span class="p">(</span><span class="s1">'255.255.255.0'</span><span class="p">)</span> </pre></div> <p>One can use the same options in the creation of an <code>ip_interface</code> option as with an <code>ip_address</code> option (binary, decimal value, hexadecimal). However, the only way to effectively create an <code>ip_interface</code> with the proper CIDR notation or mask is with a dotted decimal IPv4 address string.</p> <p><strong>Creating IPv4 Network Address Objects with <code>ipadress</code></strong>:</p> <p>The <code>ipaddress.ip_network()</code> factory function is used to create an <code>ip_network</code> object, which automatically determines whether to create an IPv4 or IPv6 address based on the passed-in value (IPv6 addressing will be discussed at a latter point in this tutorial).</p> <p>An IP network is defined as a range of consecutive IP address that define a network or subnet. Example:</p> <ul> <li><code>192.168.100.0/24</code> is the <code>192.168.100.0</code> network where the <code>/24</code> specifies that the first three octets comprise the network identification.</li> <li>The 4th octet is used for assignment to individual hosts and router interfaces.</li> <li>The address range is <code>192.168.100.1</code> through to <code>.254</code>.</li> <li><code>192.168.100.0</code> is used to define the network/subnet and <code>192.168.100.255</code> is the broadcast address for this network. Neither can be used for assignment to a host or router interface.</li> </ul> <p>The creation of an <code>ip_network</code> object follows the same syntax as the creation of an <code>ip_interface</code> object:</p> <div class="codehilite"><pre><span></span><span class="c1"># Creates an ip_network object. The IPv4 address and CIDR must be</span> <span class="c1"># a valid network address, the first address in an address range:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ipaddress</span><span class="o">.</span><span class="n">ip_network</span><span class="p">(</span><span class="s1">'192.168.100.0/24'</span><span class="p">)</span> <span class="n">IPv4Network</span><span class="p">(</span><span class="s1">'192.168.100.0/24'</span><span class="p">)</span> </pre></div> <p>In the above example, the network address used must be a valid network address, which is the first address in the range of IPv4 addresses that constitute the network. If this is not the case, Python will throw an exception:</p> <div class="codehilite"><pre><span></span><span class="c1"># Python will throw an exception if the address used is not</span> <span class="c1"># a valid network address. In the following, ".10" is a host address</span> <span class="c1"># not a valid network address ident cation, which is ".0":</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ipaddress</span><span class="o">.</span><span class="n">ip_network</span><span class="p">(</span><span class="s1">'192.168.100.10/24'</span><span class="p">)</span> <span class="ne">ValueError</span><span class="p">:</span> <span class="s2">"192.168.100.10/24 has host bits set"</span> </pre></div> <p>When working with host or router interfaces, it is often necessary to determine the network address. This can be calculated, but takes several steps which can be accomplished in a single step using the <code>strict=False</code> option (<code>strict=True</code> is default).</p> <div class="codehilite"><pre><span></span><span class="c1"># If the network address needs to be calculated,</span> <span class="c1"># use the strict=False option. This will calculate and populate</span> <span class="c1"># the ip_network object with the network rather than the</span> <span class="c1"># interface address:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_ip</span> <span class="o">=</span> <span class="n">ipaddress</span><span class="o">.</span><span class="n">ip_interface</span><span class="p">(</span><span class="s1">'192.168.100.10/24'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_ip</span> <span class="n">IPv4Interface</span><span class="p">(</span><span class="s1">'192.168.100.10/24'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_ip_net</span> <span class="o">=</span> <span class="n">ipaddress</span><span class="o">.</span><span class="n">ip_network</span><span class="p">(</span><span class="n">my_ip</span><span class="p">,</span> <span class="n">strict</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_ip_net</span> <span class="n">IPv4Network</span><span class="p">(</span><span class="s1">'192.168.100.0/24'</span><span class="p">)</span> </pre></div> <p>In the above example, the <code>ip_interface</code> address is known (<code>192.168.100.10</code>) but not the <code>ip_network</code> the interface belongs to. Using the <code>strict=False</code> option, the <code>ip_network</code> address (<code>192.168.100.0/24</code>) is calculated and populated in the <code>ip_network</code> object.</p> </section><section><h2>Python’s <code>ipaddress</code> Module and IPv6 Addresses</h2> <p>As with IPv4, the ipaddress module uses the same three basic factory functions already described for IPv4. includes include:</p> <ol> <li>a “host” or an individual address object that does not include CIDR notation,</li> <li>an interface address object that includes CIDR notation, and</li> <li>and a network address object that refers to the range of IP addresses for the entire network.</li> </ol> <p>Since the detail is covered in the section on IPv4, a brief overview is only necessary.</p> <p><strong>Creating IPv6 Host Address Objects with <code>ipaddress</code></strong>:</p> <p>The <code>ipaddress.ip_address()</code> factory function is used to create an <code>ip_address</code> object. This automatically knows to use the IPv6 address format based on the passed-in value. Note that the CIDR notation is not used with the <code>ip_address</code> function.</p> <p>In the majority of cases, the value used to create an <code>ip_address</code> object for IPv6 will be a string in the IPv6 quartet/hextet format as per this example:</p> <div class="codehilite"><pre><span></span><span class="c1"># Create an IPv6 Address Object for a Global Address:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ipaddress</span><span class="o">.</span><span class="n">ip_address</span><span class="p">(</span><span class="s1">'2001:db8:abcd:100::1'</span><span class="p">)</span> <span class="n">IPv6Address</span><span class="p">(</span><span class="s1">'2001:db8:abcd:100::1'</span><span class="p">)</span> <span class="c1"># Create an IPv6 Address Object for a link-local address:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ipaddress</span><span class="o">.</span><span class="n">ip_address</span><span class="p">(</span><span class="s1">'fe80::1'</span><span class="p">)</span> <span class="n">IPv6Address</span><span class="p">(</span><span class="s1">'fe80::1'</span><span class="p">)</span> </pre></div> <p>As with IPv4, it is possible to create an IPv6 address object using the full binary, decimal, or hexadecimal value. This is unwieldy with 32 bits for an IPv4 address and is even more awkward for a 128 bit IPv6 address. As a practical matter, it is anticipated that the string representation of the eight quartets will be the norm.</p> <p><strong>Creating IPv6 Interface Address Objects with <code>ipaddress</code></strong>:</p> <p>The <code>ipaddress.ip_interface()</code> factory function is used to create an <code>ip_interface object</code>, which automatically create an IPv6 address based on the passed-in value. Note that the CIDR notation must be included in the function.</p> <div class="codehilite"><pre><span></span><span class="c1"># Creates an IP Interface Object for a Global Address:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ipaddress</span><span class="o">.</span><span class="n">ip_interface</span><span class="p">(</span><span class="s1">'2001:db8:abcd:100::1/64'</span><span class="p">)</span> <span class="n">IPv6Interface</span><span class="p">(</span><span class="s1">'2001:db8:abcd:100::1/64'</span><span class="p">)</span> <span class="c1"># Creates an IP Interface Object for a Link-local Address:</span> <span class="n">ipaddress</span><span class="o">.</span><span class="n">ip_interface</span><span class="p">(</span><span class="s1">'fe80::1/64'</span><span class="p">)</span> <span class="n">IPv6Interface</span><span class="p">(</span><span class="s1">'fe80::1/64'</span><span class="p">)</span> </pre></div> <p><strong>Creating IPv6 Network Address Objects with <code>ipaddress</code></strong>:</p> <p>The <code>ipaddress.ip_network()</code> factory function is used to create an <code>ip_network</code> object for IPv6 based on the passed-in value.</p> <p>As with IPv4, an IPv6 network is defined as a range of consecutive IP address that can be assigned to specific host or router interfaces.</p> <p>Using our previous example <code>2001:db8:abcd:100::/64</code>, the <code>/64</code> CIDR specifies that the four quartets make up the full network identification. Remember that the first three quartets are global ID assigned by the IPS and the fourth quartet identifies the internal subnet number. The balance of the 64 bits are used for host identification with a range from “<code>0000:0000:0000:0001</code>” though to “<code>ffff:ffff:ffff:fffe</code>”.</p> <p>As with IPv4 addressing, the first and last address in an IPv6 subnet cannot be used for host addressing. Given a <code>/64</code> CIDR, this means that there are 2 to the 64th power (minus 2) possible host addresses, which is means there are 18,446,744,073,709,551,614 mathematically possible host addresses per network/subnet.</p> <div class="codehilite"><pre><span></span><span class="c1"># Creates an IP Network Object for a Global Address:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">myIPv6net</span> <span class="o">=</span> <span class="n">ipaddress</span><span class="o">.</span><span class="n">ip_network</span><span class="p">(</span><span class="s1">'2001:db8:abcd:100::/64'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">myIPv6net</span> <span class="n">IPv6Network</span><span class="p">(</span><span class="s1">'2001:db8:abcd:100::/64'</span><span class="p">)</span> <span class="c1"># Creates an IP Network Object for a Link-local Address:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">myIPv6</span> <span class="o">=</span> <span class="n">ipaddress</span><span class="o">.</span><span class="n">ip_network</span><span class="p">(</span><span class="s1">'fe80::/64'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">myIPv6</span> <span class="n">IPv6Network</span><span class="p">(</span><span class="s1">'fe80::/64'</span><span class="p">)</span> </pre></div> <p>The above global address is broken down as follows:</p> <ul> <li>Global Identifier assigned by ISP: <code>2001:db8:abcd::/48</code></li> <li>Subnet identification: <code>2001:db8:abcd:100::/64</code></li> <li>First usable address in the subnet: <code>2001:db8:abcd:100::1/64</code></li> <li>Last usable address in the subnet: <code>2001:db8:abcd:100:ffff:ffff:ffff:fffeffff/64</code></li> </ul> </section><section><h2>Additional Resources</h2> <p>These are some additional resources where you can learn about the <code>ipaddress</code> module in Python:</p> <ul> <li><a href="/img/ipaddress-module-introduction.pdf">An extended PDF version of this article with additional information</a></li> <li><a href="https://docs.python.org/3/library/ipaddress.html" target="_blank"><code>ipaddress</code> Module Documentation</a></li> <li><a href="https://docs.python.org/3/howto/ipaddress.html" target="_blank">An Introduction to the <code>ipaddress</code> Module</a></li> <li><a href="https://en.wikipedia.org/wiki/IPv4" target="_blank">Wikipedia - IPv4</a></li> <li><a href="https://en.wikipedia.org/wiki/IPv6" target="_blank">Wikipedia - IPv6</a></li> </ul></section><footer></footer></article>https://dbader.org/blog/python-ipaddress-moduleThu, 01 Jun 2017 00:00:00 GMTHow to Prepare for a Python Coding Interviewhttps://dbader.org/blog/how-to-prepare-for-a-python-coding-interview<article><header><h1>How to Prepare for a Python Coding Interview</h1> <p>A “minimalist guide” on how to prepare for your upcoming Python interview in three steps.</p> </header><section><figure><img alt="" src="/blog/figures/python-coding-interview-prep.png" width="1280" height="720"></figure> <p>Participating in a “Silicon Valley style” coding interview can feel scary as heck.</p> <p>Unlike other professionals, it seems to be okay for software developers to expect to get humiliated at a job interview:</p> </section><section><h2 class="crosshead thin">“What, you can’t code up a recursive descent parser on a whiteboard in 7.5 minutes? How DARE YOU<br>even apply for this job!”</h2> <p>Yeah, it’s one of the things that sucks about our industry—</p> <p>Personally, I believe that 80-90% of the questions that you get asked during a typical coding interview have very little to do with your real performance on the job.</p> <p>But unfortunately these interviews aren’t going to go away over night.</p> <p>If you want a well-paid job as a software developer, you’re likely going to encounter some coding quiz as part of your interviewing experience.</p> <p>For the foreseeable future, interviewers are going to keep squeezing you through the same processes and will keep asking you those same questions…</p> <p>And if you’re like me, there’s a pretty slim chance you’ll pass an interview like that without some serious prep work—either to learn the right skills or to refresh your memory.</p> <p>Alright, that all sounds pretty glum, no?</p> <p>But here’s what you need to realize:</p> </section><section><h2 class="crosshead thin">Interviewing is a skill you can<br>learn like any other.</h2> <p>It’s something you get better at with practice.</p> <p>It’s true—just remember that all that prep work needs time. So be sure to plan ahead with ample of buffer to get enough study days in before your “big day.”</p> <p>If I had a coding interview coming up in 1-2 months, here’s a rough outline of what I’d do to prepare:</p> <div class="callout-box"> <h2>Step 1:</h2> <p>Buy the following two books:</p> <ul> <li><a href="http://amzn.to/2sq9gEj" target="_blank">“Elements of Programming Interviews (Python Ed.)”</a> by Aziz, Lee and Prakash; and</li> <li><a href="http://amzn.to/1oeOjn9" target="_blank">“Cracking the Coding Interview”</a> by Gayle Laakmann McDowell</li> </ul> </div> <div class="callout-box"> <h2>Step 2:</h2> <p>Buy a whiteboard and some markers. Put the whiteboard on an actual wall, and make sure you get a board with a decent size. This is where 90% of your prep work will happen over the next few weeks.</p> </div> <div class="callout-box"> <h2>Step 3:</h2> <p>Every day, stand in front of your whiteboard and work on at least one problem from the books listed in Step 1.</p> <p>Talk out loud about what you’re doing, and snap a photo of the board when you’re done. Set a 30 minute timer for each problem to put some pressure on yourself.</p> <p>If you can’t solve a problem, pick up the book and go through all the motions with the solution in front of you. Rinse and repeat.</p> </div> <p>The closer you get to your interview date the more you want to practice—ramp it up to around 5 problems per day in the last two weeks before your interview.</p> <p>Just repeat after me:</p> </section><section><h2 class="crosshead thin">“Interviewing is a learned skill.”</h2> <p>The more “reps” you can get on each problem, the better your chances of getting a job offer will be. I know this sounds tough—but with persistence and regular practice you can do it. Keep at it and you’ll eventually succeed. It’ll be worth it!</p></section><footer></footer></article>https://dbader.org/blog/how-to-prepare-for-a-python-coding-interviewTue, 30 May 2017 00:00:00 GMTIn Love, War, and Open-Source: Never Give Uphttps://dbader.org/blog/in-love-war-and-open-source-never-give-up<article><header><h1>In Love, War, and Open-Source: Never Give Up</h1> <p>I’ll never forget launching my first open-source project and sharing it publicly on Reddit…</p> </header><section><figure><img alt="" src="/blog/figures/love-war-open-source.png" width="1280" height="720"></figure> <p>I had spent a couple of days at my parents’ place over Christmas that year and decided to use some of my spare time to work on a Python library I christened <a href="https://github.com/dbader/schedule" target="_blank"><code>schedule</code></a>.</p> <p>The idea behind <code>schedule</code> was very simple and had a narrow focus (I find that that that’s always a good idea for libraries by the way):</p> <p>Developers would use it like a timer to periodically call a function inside their Python programs.</p> <p>The kicker was that <code>schedule</code> used a funky “natural sounding” syntax to specify the timer interval. For example, if you wanted to run a function every 10 minutes you’d do this:</p> <div class="codehilite"><pre><span></span><span class="n">schedule</span><span class="o">.</span><span class="n">every</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span><span class="o">.</span><span class="n">minutes</span><span class="o">.</span><span class="n">do</span><span class="p">(</span><span class="n">myfunc</span><span class="p">)</span> </pre></div> <p>Or, if you wanted to run a particular task every day at 10:30 in the morning, you’d do this:</p> <div class="codehilite"><pre><span></span><span class="n">schedule</span><span class="o">.</span><span class="n">every</span><span class="p">()</span><span class="o">.</span><span class="n">day</span><span class="o">.</span><span class="n">at</span><span class="p">(</span><span class="s1">'10:30'</span><span class="p">)</span><span class="o">.</span><span class="n">do</span><span class="p">(</span><span class="n">mytask</span><span class="p">)</span> </pre></div> <p>Because I was so frustrated with Cron’s syntax I thought this approach was really cool. And so I decided this would be the first Python module I’d release as open-source.</p> <p>I cleaned up the code and spent some time coming up with <a href="write-a-great-readme-for-your-github-project">a nice README file</a>—because that’s really the first thing that your potential users will see when they check out your library.</p> <p>Once I had my module available on PyPI and the source code on GitHub I decided to call some attention to the project. The same night I posted a link to the repository to Reddit and a couple of other sites.</p> </section><section><h2 class="crosshead thin">I still remember that I had shaky hands when I clicked the “submit” button…</h2> <p>It’s scary to put your work out there for the whole world to judge! Also, I didn’t know what to expect.</p> <p>Would people call me stupid for writing a “simple” library like that?</p> <p>Would they think my code wasn’t good enough?</p> <p>Would they find all kinds of bugs and publicly shame me for them? I felt almost a physical sense of dread about pushing the “submit” button on Reddit that night!</p> <p>The next morning I woke up and immediately checked my email. Were there any comments? Yes, about twenty or so!</p> <p>I started reading through all of them, faster and faster—</p> <p>And of course my still frightful mind immediately zoomed in on the negative ones, like</p> <blockquote> <p>“Cool idea, but not particularly useful”,</p> </blockquote> <p>and</p> <blockquote> <p>“The documentation is not enough”,</p> </blockquote> <p>or</p> <blockquote> <p>“Not a big fan of the pseudo-english syntax. Way too clever and gimmicky.”</p> </blockquote> <p>At this point I was starting to feel a <em>little</em> discouraged… I’d never really shared my code publicly before and to be honest I my skin receiving criticism on it was paper thin. After all, this was just something I wrote in a couple of hours and gave away for free.</p> <p>The comment that really made my stomach churn was one from a well known member of the Python community:</p> <blockquote> <p>“And another library with global state :-( … Such an API should not even exist. It sets a bad example.”</p> </blockquote> <p>Ouch, that stung. I really looked up to that person and had used some of their libraries in other projects… It was almost like my worst fears were now playing out in front of me!</p> </section><section><h2 class="crosshead thin">I’d never be able to get another job as a Python developer after this…</h2> <p>At the time I didn’t see the positive and supportive comments in that discussion thread. I didn’t see the almost 70 upvotes. I didn’t see the valuable lessons hidden in the seemingly rude comments. I dwelled on the negative and felt terrible and depressed that whole day.</p> <p>So how do you think this story ends?</p> <p>Did I delete the <code>schedule</code> repo, switched careers and never looked at Reddit again?</p> <p>Wrong!</p> <p><code>schedule</code> now has almost 3,000 stars on GitHub and is among the top 70 Python repositories (out of more than 215,000). When PyPI’s download statistics we’re still working I saw that it got several thousand downloads per month. I get emails every week from people asking questions about it or thanking me for writing it…</p> </section><section><h2 class="crosshead thin">Isn’t that crazy!? How’s that possible after all of these disheartening comments?</h2> <p>My answer is “I don’t know”—and I also don’t think that <code>schedule</code> is a particularly great library that deserves all this attention, by the way.</p> <p>But, it seems to solve a problem for some people. It also seems to have a polarizing effect on developers who see it—some love it, some hate it.</p> <p>Today I’m glad I shipped <code>schedule</code> that night.</p> <p>Glad because it was helpful to so many people over the years and glad because it helped me develop a thicker skin when it comes to sharing and launching things publicly.</p> <p>I’m partly writing this meandering post because not very long ago I found this comment buried in my Reddit message history:</p> <blockquote> <p>As someone who has posted a number of projects and blog posts in r/Python, just wanted to drop you a line and encourage that you don’t let the comments in your thread get you down. You see all those upvotes?</p> <p>Those are people that like your library, but don’t really have a comment to make in the thread proper. My biggest issue with /r/Python is that it tends towards cynicism and sometimes cruelty rather than encouragement and constructive criticism.</p> <p>Keep up the great work,</p> <p>Rob</p> </blockquote> <p>Wow! What a positive and encouraging comment!</p> <p>Back when I felt discouraged by all of these negative comments I must’ve missed it. But reading it a few years later made me re-live that whole situation and it showed me how much I’d grown as a developer and as a person in the meantime.</p> <p>If you find yourself in a similar situation, maybe feeling bogged down by the developer community who can be unfiltered and pretty rude sometimes, don’t get discouraged.</p> <p>Even if some people don’t like what you did there can be thousands who love your work.</p> <p>It’s a big pond, and sometimes the best ideas are polarizing.</p> <p>The only way to find out is to ship, ship, ship.</p></section><footer></footer></article>https://dbader.org/blog/in-love-war-and-open-source-never-give-upThu, 25 May 2017 00:00:00 GMTThe Meaning of Underscores in Pythonhttps://dbader.org/blog/meaning-of-underscores-in-python<article><header><h1>The Meaning of Underscores in Python</h1> <p>The various meanings and naming conventions around single and double underscores (“dunder”) in Python, how name mangling works and how it affects your own Python classes.</p> </header><section><figure><img alt="The Meaning of _ and __ Underscores in Python" src="/blog/figures/python-underscores.png" width="1280" height="720"></figure> <p>Single and double underscores have a meaning in Python variable and method names. Some of that meaning is merely by convention and intended as a hint to the programmer—and some of it is enforced by the Python interpreter.</p> <p>If you’re wondering <em>“What’s the meaning of single and double underscores in Python variable and method names?”</em> I’ll do my best to get you the answer here.</p> <p>In this article I’ll discuss the following five underscore patterns and naming conventions and how they affect the behavior of your Python programs:</p> <ul> <li>Single Leading Underscore: <code>_var</code></li> <li>Single Trailing Underscore: <code>var_</code></li> <li>Double Leading Underscore: <code>__var</code></li> <li>Double Leading and Trailing Underscore: <code>__var__</code></li> <li>Single Underscore: <code>_</code></li> </ul> <p>At the end of the article you’ll also find a brief “cheat sheet” summary of the five different underscore naming conventions and their meaning, as well as a short video tutorial that gives you a hands-on demo of their behavior.</p> <p>Let’s dive right in!</p> </section><section><h2>1. Single Leading Underscore: <code>_var</code></h2> <p>When it comes to variable and method names, the single underscore prefix has a meaning by convention only. It’s a hint to the programmer—and it means what the Python community agrees it should mean, but it does not affect the behavior of your programs.</p> <p>The underscore prefix is meant as a <em>hint</em> to another programmer that a variable or method starting with a single underscore is intended for internal use. This convention is <a href="http://pep8.org/#descriptive-naming-styles" target="_blank">defined in PEP 8</a>.</p> <p>This isn’t enforced by Python. Python does not have strong distinctions between “private” and “public” variables like Java does. It’s like someone put up a tiny underscore warning sign that says:</p> <blockquote> <p>“Hey, this isn’t really meant to be a part of the public interface of this class. Best to leave it alone.”</p> </blockquote> <p>Take a look at the following example:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Test</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">foo</span> <span class="o">=</span> <span class="mi">11</span> <span class="bp">self</span><span class="o">.</span><span class="n">_bar</span> <span class="o">=</span> <span class="mi">23</span> </pre></div> <p>What’s going to happen if you instantiate this class and try to access the <code>foo</code> and <code>_bar</code> attributes defined in its <code>__init__</code> constructor? Let’s find out:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">t</span> <span class="o">=</span> <span class="n">Test</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">foo</span> <span class="mi">11</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">_bar</span> <span class="mi">23</span> </pre></div> <p>You just saw that the leading single underscore in <code>_bar</code> did not prevent us from “reaching into” the class and accessing the value of that variable.</p> <p>That’s because the single underscore prefix in Python is merely an agreed upon convention—at least when it comes to variable and method names.</p> <p><strong>However, leading underscores do impact how names get imported from modules.</strong> Imagine you had the following code in a module called <code>my_module</code>:</p> <div class="codehilite"><pre><span></span><span class="c1"># This is my_module.py:</span> <span class="k">def</span> <span class="nf">external_func</span><span class="p">():</span> <span class="k">return</span> <span class="mi">23</span> <span class="k">def</span> <span class="nf">_internal_func</span><span class="p">():</span> <span class="k">return</span> <span class="mi">42</span> </pre></div> <p>Now if you use a <em>wildcard import</em> to import all names from the module, Python will <em>not</em> import names with a leading underscore (unless the module defines an <a href="https://docs.python.org/3/tutorial/modules.html#importing-from-a-package" target="_blank"><code>__all__</code> list</a> that overrides this behavior):</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">my_module</span> <span class="kn">import</span> <span class="o">*</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">external_func</span><span class="p">()</span> <span class="mi">23</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">_internal_func</span><span class="p">()</span> <span class="ne">NameError</span><span class="p">:</span> <span class="s2">"name '_internal_func' is not defined"</span> </pre></div> <p>By the way, <a href="http://pep8.org/#imports" target="_blank">wildcard imports should be avoided</a> as they make it unclear which names are present in the namespace. It’s better to stick to regular imports for the sake of clarity.</p> <p>Unlike wildcard imports, regular imports are not affected by the leading single underscore naming convention:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">my_module</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_module</span><span class="o">.</span><span class="n">external_func</span><span class="p">()</span> <span class="mi">23</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_module</span><span class="o">.</span><span class="n">_internal_func</span><span class="p">()</span> <span class="mi">42</span> </pre></div> <p>I know this might be a little confusing at this point. If you stick to the PEP 8 recommendation that wildcard imports should be avoided, then really all you need to remember is this:</p> <blockquote> <p><em>Single underscores are a Python naming convention indicating a name is meant for internal use. It is generally not enforced by the Python interpreter and meant as a hint to the programmer only.</em></p> </blockquote> </section><section><h2>2. Single Trailing Underscore: <code>var_</code></h2> <p>Sometimes the most fitting name for a variable is already taken by a keyword. Therefore names like <code>class</code> or <code>def</code> cannot be used as variable names in Python. In this case you can append a single underscore to break the naming conflict:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">make_object</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">class</span><span class="p">):</span> <span class="ne">SyntaxError</span><span class="p">:</span> <span class="s2">"invalid syntax"</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">make_object</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">class_</span><span class="p">):</span> <span class="o">...</span> <span class="k">pass</span> </pre></div> <p>In summary, a single trailing underscore (postfix) is used by convention to avoid naming conflicts with Python keywords. This convention is <a href="http://pep8.org/#descriptive-naming-styles" target="_blank">explained in PEP 8</a>.</p> </section><section><h2>3. Double Leading Underscore: <code>__var</code></h2> <p>The naming patterns we covered so far received their meaning from agreed upon conventions only. With Python class attributes (variables and methods) that start with double underscores, things are a little different.</p> <p>A double underscore prefix causes the Python interpreter to rewrite the attribute name in order to avoid naming conflicts in subclasses.</p> <p>This is also called <em>name mangling</em>—the interpreter changes the name of the variable in a way that makes it harder to create collisions when the class is extended later.</p> <p>I know this sounds rather abstract. This is why I put together this little code example we can use for experimentation:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Test</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">foo</span> <span class="o">=</span> <span class="mi">11</span> <span class="bp">self</span><span class="o">.</span><span class="n">_bar</span> <span class="o">=</span> <span class="mi">23</span> <span class="bp">self</span><span class="o">.</span><span class="n">__baz</span> <span class="o">=</span> <span class="mi">23</span> </pre></div> <p>Let’s take a look at the attributes on this object using the <a href="https://docs.python.org/3/library/functions.html#dir" target="_blank">built-in <code>dir()</code> function</a>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">t</span> <span class="o">=</span> <span class="n">Test</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">dir</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="p">[</span><span class="s1">'_Test__baz'</span><span class="p">,</span> <span class="s1">'__class__'</span><span class="p">,</span> <span class="s1">'__delattr__'</span><span class="p">,</span> <span class="s1">'__dict__'</span><span class="p">,</span> <span class="s1">'__dir__'</span><span class="p">,</span> <span class="s1">'__doc__'</span><span class="p">,</span> <span class="s1">'__eq__'</span><span class="p">,</span> <span class="s1">'__format__'</span><span class="p">,</span> <span class="s1">'__ge__'</span><span class="p">,</span> <span class="s1">'__getattribute__'</span><span class="p">,</span> <span class="s1">'__gt__'</span><span class="p">,</span> <span class="s1">'__hash__'</span><span class="p">,</span> <span class="s1">'__init__'</span><span class="p">,</span> <span class="s1">'__le__'</span><span class="p">,</span> <span class="s1">'__lt__'</span><span class="p">,</span> <span class="s1">'__module__'</span><span class="p">,</span> <span class="s1">'__ne__'</span><span class="p">,</span> <span class="s1">'__new__'</span><span class="p">,</span> <span class="s1">'__reduce__'</span><span class="p">,</span> <span class="s1">'__reduce_ex__'</span><span class="p">,</span> <span class="s1">'__repr__'</span><span class="p">,</span> <span class="s1">'__setattr__'</span><span class="p">,</span> <span class="s1">'__sizeof__'</span><span class="p">,</span> <span class="s1">'__str__'</span><span class="p">,</span> <span class="s1">'__subclasshook__'</span><span class="p">,</span> <span class="s1">'__weakref__'</span><span class="p">,</span> <span class="s1">'_bar'</span><span class="p">,</span> <span class="s1">'foo'</span><span class="p">]</span> </pre></div> <p>This gives us a list with the object’s attributes. Let’s take this list and look for our original variable names <code>foo</code>, <code>_bar</code>, and <code>__baz</code>—I promise you’ll notice some interesting changes.</p> <ul> <li>The <code>self.foo</code> variable appears unmodified as <code>foo</code> in the attribute list.</li> <li><code>self._bar</code> behaves the same way—it shows up on the class as <code>_bar</code>. Like I said before, the leading underscore is just a <em>convention</em> in this case. A hint for the programmer.</li> <li>However with <code>self.__baz</code>, things look a little different. When you search for <code>__baz</code> in that list you’ll see that there is no variable with that name.</li> </ul> <p>So what happened to <code>__baz</code>?</p> <p>If you look closely you’ll see there’s an attribute called <code>_Test__baz</code> on this object. This is the <em>name mangling</em> that the Python interpreter applies. It does this to protect the variable from getting overridden in subclasses.</p> <p>Let’s create another class that extends the <code>Test</code> class and attempts to override its existing attributes added in the constructor:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">ExtendedTest</span><span class="p">(</span><span class="n">Test</span><span class="p">):</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">foo</span> <span class="o">=</span> <span class="s1">'overridden'</span> <span class="bp">self</span><span class="o">.</span><span class="n">_bar</span> <span class="o">=</span> <span class="s1">'overridden'</span> <span class="bp">self</span><span class="o">.</span><span class="n">__baz</span> <span class="o">=</span> <span class="s1">'overridden'</span> </pre></div> <p>Now what do you think the values of <code>foo</code>, <code>_bar</code>, and <code>__baz</code> will be on instances of this <code>ExtendedTest</code> class? Let’s take a look:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">t2</span> <span class="o">=</span> <span class="n">ExtendedTest</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">t2</span><span class="o">.</span><span class="n">foo</span> <span class="s1">'overridden'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">t2</span><span class="o">.</span><span class="n">_bar</span> <span class="s1">'overridden'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">t2</span><span class="o">.</span><span class="n">__baz</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="s2">"'ExtendedTest' object has no attribute '__baz'"</span> </pre></div> <p>Wait, why did we get that <code>AttributeError</code> when we tried to inspect the value of <code>t2.__baz</code>? Name mangling strikes again! It turns out this object doesn’t even have a <code>__baz</code> attribute:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">dir</span><span class="p">(</span><span class="n">t2</span><span class="p">)</span> <span class="p">[</span><span class="s1">'_ExtendedTest__baz'</span><span class="p">,</span> <span class="s1">'_Test__baz'</span><span class="p">,</span> <span class="s1">'__class__'</span><span class="p">,</span> <span class="s1">'__delattr__'</span><span class="p">,</span> <span class="s1">'__dict__'</span><span class="p">,</span> <span class="s1">'__dir__'</span><span class="p">,</span> <span class="s1">'__doc__'</span><span class="p">,</span> <span class="s1">'__eq__'</span><span class="p">,</span> <span class="s1">'__format__'</span><span class="p">,</span> <span class="s1">'__ge__'</span><span class="p">,</span> <span class="s1">'__getattribute__'</span><span class="p">,</span> <span class="s1">'__gt__'</span><span class="p">,</span> <span class="s1">'__hash__'</span><span class="p">,</span> <span class="s1">'__init__'</span><span class="p">,</span> <span class="s1">'__le__'</span><span class="p">,</span> <span class="s1">'__lt__'</span><span class="p">,</span> <span class="s1">'__module__'</span><span class="p">,</span> <span class="s1">'__ne__'</span><span class="p">,</span> <span class="s1">'__new__'</span><span class="p">,</span> <span class="s1">'__reduce__'</span><span class="p">,</span> <span class="s1">'__reduce_ex__'</span><span class="p">,</span> <span class="s1">'__repr__'</span><span class="p">,</span> <span class="s1">'__setattr__'</span><span class="p">,</span> <span class="s1">'__sizeof__'</span><span class="p">,</span> <span class="s1">'__str__'</span><span class="p">,</span> <span class="s1">'__subclasshook__'</span><span class="p">,</span> <span class="s1">'__weakref__'</span><span class="p">,</span> <span class="s1">'_bar'</span><span class="p">,</span> <span class="s1">'foo'</span><span class="p">,</span> <span class="s1">'get_vars'</span><span class="p">]</span> </pre></div> <p>As you can see <code>__baz</code> got turned into <code>_ExtendedTest__baz</code> to prevent accidental modification:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">t2</span><span class="o">.</span><span class="n">_ExtendedTest__baz</span> <span class="s1">'overridden'</span> </pre></div> <p>But the original <code>_Test__baz</code> is also still around:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">t2</span><span class="o">.</span><span class="n">_Test__baz</span> <span class="mi">42</span> </pre></div> <p>Double underscore name mangling is fully transparent to the programmer. Take a look at the following example that will confirm this:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">ManglingTest</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">__mangled</span> <span class="o">=</span> <span class="s1">'hello'</span> <span class="k">def</span> <span class="nf">get_mangled</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">__mangled</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ManglingTest</span><span class="p">()</span><span class="o">.</span><span class="n">get_mangled</span><span class="p">()</span> <span class="s1">'hello'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ManglingTest</span><span class="p">()</span><span class="o">.</span><span class="n">__mangled</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="s2">"'ManglingTest' object has no attribute '__mangled'"</span> </pre></div> <p>Does name mangling also apply to method names? It sure does—name mangling affects <em>all</em> names that start with two underscore characters (“dunders”) in a class context:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">MangledMethod</span><span class="p">:</span> <span class="k">def</span> <span class="nf">__method</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="mi">42</span> <span class="k">def</span> <span class="nf">call_it</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">__method</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">MangledMethod</span><span class="p">()</span><span class="o">.</span><span class="n">__method</span><span class="p">()</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="s2">"'MangledMethod' object has no attribute '__method'"</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">MangledMethod</span><span class="p">()</span><span class="o">.</span><span class="n">call_it</span><span class="p">()</span> <span class="mi">42</span> </pre></div> <p>Here’s another, perhaps surprising, example of name mangling in action:</p> <div class="codehilite"><pre><span></span><span class="n">_MangledGlobal__mangled</span> <span class="o">=</span> <span class="mi">23</span> <span class="k">class</span> <span class="nc">MangledGlobal</span><span class="p">:</span> <span class="k">def</span> <span class="nf">test</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="n">__mangled</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">MangledGlobal</span><span class="p">()</span><span class="o">.</span><span class="n">test</span><span class="p">()</span> <span class="mi">23</span> </pre></div> <p>In this example I declared a global variable called <code>_MangledGlobal__mangled</code>. Then I accessed the variable inside the context of a class named <code>MangledGlobal</code>. Because of name mangling I was able to reference the <code>_MangledGlobal__mangled</code> global variable as just <code>__mangled</code> inside the <code>test()</code> method on the class.</p> <p>The Python interpreter automatically expanded the name <code>__mangled</code> to <code>_MangledGlobal__mangled</code> because it begins with two underscore characters. This demonstrated that name mangling isn’t tied to class attributes specifically. It applies to any name starting with two underscore characters used in a class context.</p> <p><strong>Now this was a lot of stuff to absorb.</strong></p> <p>To be honest with you I didn’t write these examples and explanations down off the top of my head. It took me some research and editing to do it. I’ve been using Python for years but rules and special cases like that aren’t constantly on my mind.</p> <p>Sometimes the most important skills for a programmer are “pattern recognition” and knowing where to look things up. If you feel a little overwhelmed at this point, don’t worry. Take your time and play with some of the examples in this article.</p> <p>Make these concepts sink in enough so that you’ll recognize the general idea of name mangling and some of the other behaviors I showed you. If you encounter them “in the wild” one day, you’ll know what to look for in the documentation.</p> <div class="update-box"> <h2>⏰ Sidebar: What’s a “dunder” in Python?</h2> <p>I’ve you’ve heard some experienced Pythonistas talk about Python or watched a few conference talks you may have heard the term <em>dunder</em>. If you’re wondering what that is, here’s your answer:</p> <p>Double underscores are often referred to as <a href="https://nedbatchelder.com/blog/200605/dunder.html" target="_blank">“dunders”</a> in the Python community. The reason is that double underscores appear quite often in Python code and to avoid fatiguing their jaw muscles Pythonistas often shorten “double underscore” to “dunder.”</p> <p>For example, you’d pronounce <code>__baz</code> as “dunder baz”. Likewise <code>__init__</code> would be pronounced as “dunder init”, even though one might think it should be “dunder init dunder.” But that’s just yet another quirk in the naming convention.</p> <p>It’s like a <em>secret handshake</em> for Python developers 🙂</p> </div> </section><section><h2>4. Double Leading and Trailing Underscore: <code>__var__</code></h2> <p>Perhaps surprisingly, name mangling is <em>not</em> applied if a name <em>starts and ends</em> with double underscores. Variables surrounded by a double underscore prefix and postfix are left unscathed by the Python interpeter:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">PrefixPostfixTest</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">__bam__</span> <span class="o">=</span> <span class="mi">42</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">PrefixPostfixTest</span><span class="p">()</span><span class="o">.</span><span class="n">__bam__</span> <span class="mi">42</span> </pre></div> <p>However, names that have both leading and trailing double underscores are reserved for special use in the language. This rule covers things like <code>__init__</code> for object constructors, or <code>__call__</code> to make an object callable.</p> <p>These <em>dunder methods</em> are often referred to as <em>magic methods</em>—but many people in the Python community, including myself, <a href="http://www.pixelmonkey.org/2013/04/11/python-double-under-double-wonder" target="_blank">don’t like that</a>.</p> <p>It’s best to stay away from using names that start and end with double underscores (“dunders”) in your own programs to avoid collisions with future changes to the Python language.</p> </section><section><h2>5. Single Underscore: <code>_</code></h2> <p>Per convention, a single standalone underscore is sometimes used as a name to indicate that a variable is temporary or insignificant.</p> <p>For example, in the following loop we don’t need access to the running index and we can use “<code>_</code>” to indicate that it is just a temporary value:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">32</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s1">'Hello, World.'</span><span class="p">)</span> </pre></div> <p>You can also use single underscores in unpacking expressions as a “don’t care” variable to ignore particular values. Again, this meaning is “per convention” only and there’s no special behavior triggered in the Python interpreter. The single underscore is simply a valid variable name that’s sometimes used for this purpose.</p> <p>In the following code example I’m unpacking a <code>car</code> tuple into separate variables but I’m only interested in the values for <code>color</code> and <code>mileage</code>. However, in order for the unpacking expression to succeed I need to assign all values contained in the tuple to variables. That’s where “<code>_</code>” is useful as a placeholder variable:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">car</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'red'</span><span class="p">,</span> <span class="s1">'auto'</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mf">3812.4</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">color</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">mileage</span> <span class="o">=</span> <span class="n">car</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">color</span> <span class="s1">'red'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">mileage</span> <span class="mf">3812.4</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">_</span> <span class="mi">12</span> </pre></div> <p>Besides its use as a temporary variable, “<code>_</code>” is a special variable in most Python REPLs that represents the result of the last expression evaluated by the interpreter.</p> <p>This is handy if you’re working in an interpreter session and you’d like to access the result of a previous calculation. Or if you’re constructing objects on the fly and want to interact with them without assigning them a name first:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="mi">20</span> <span class="o">+</span> <span class="mi">3</span> <span class="mi">23</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">_</span> <span class="mi">23</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="n">_</span><span class="p">)</span> <span class="mi">23</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">list</span><span class="p">()</span> <span class="p">[]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">_</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">_</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">_</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">_</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> </pre></div> </section><section><h2>📓 Python Underscore Naming Patterns – Summary</h2> <p>Here’s a quick summary or “cheat sheet” of what the five underscore patterns I covered in this article mean in Python:</p> <table> <thead> <tr> <th>Pattern</th> <th>Example</th> <th>Meaning</th> </tr> </thead> <tbody> <tr> <td><strong>Single Leading Underscore</strong></td> <td><code>_var</code></td> <td>Naming convention indicating a name is meant for internal use. Generally not enforced by the Python interpreter (except in wildcard imports) and meant as a hint to the programmer only.</td> </tr> <tr> <td><strong>Single Trailing Underscore</strong></td> <td><code>var_</code></td> <td>Used by convention to avoid naming conflicts with Python keywords.</td> </tr> <tr> <td><strong>Double Leading Underscore</strong></td> <td><code>__var</code></td> <td>Triggers name mangling when used in a class context. Enforced by the Python interpreter.</td> </tr> <tr> <td><strong>Double Leading and Trailing Underscore</strong></td> <td><code>__var__</code></td> <td>Indicates special methods defined by the Python language. Avoid this naming scheme for your own attributes.</td> </tr> <tr> <td><strong>Single Underscore</strong></td> <td><code>_</code></td> <td>Sometimes used as a name for temporary or insignificant variables (“don’t care”). Also: The result of the last expression in a Python REPL.</td> </tr> </tbody> </table> </section><section><h2>📺 Underscore Patterns – Video Tutorial</h2> <p>Watch a short video tutorial to see first-hand how things like double underscore name mangling work in Python and how they affect your own classes and modules:</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/ALZmCy2u0jQ?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p>Did I miss anything in this explanation? Want to add your own thoughts on the matter? Leave a comment below, I’d appreciate it.</p></section><footer></footer></article>https://dbader.org/blog/meaning-of-underscores-in-pythonTue, 23 May 2017 00:00:00 GMTWhen to Use Pythonhttps://dbader.org/blog/when-to-use-python<article><header><h1>When to Use Python</h1> <p>What is the Python programming language used for in the real world, and when is using Python the right choice?</p> </header><section><figure><img alt="When To Use Python?" src="/blog/figures/when-to-use-python.png" width="1280" height="720"></figure> <p>When I grew up in Germany as a kid there was this craze about “desks that can grow with you.” The idea was you’d buy your kid an adjustable desk and then they’d be able to use it throughout their whole education career.</p> <p>As your kid grows taller, so does his or her desk. Just turn the little crank handle every few months… And voila, you’re right on track for raising the next Albert Einstein or Marie Curie.</p> </section><section><h2 class="crosshead thin">Python is a great<br>“adjustable desk” language.</h2> <p>With the small but important difference that Python is also a <em>much prettier</em> desk. One that you wouldn’t be embarrassed of using past elementary school. And one you’d be okay with showing to your girlfriend/boyfriend. (Okay, time to stop with that desk analogy.)</p> <p>My point is this:</p> <p>What I love about Python is how it <em>scales so well</em> (no pun intended)—from writing simple prototypes to validate an idea, all the way to building “production grade” systems.</p> <p>Sure, sometimes it would be nice to have a compiler and static type checks to lean on—but often I realized that I would’ve never come this far in so little time with Java or C++. And with <a href="https://www.python.org/dev/peps/pep-0484/" target="_blank">optional type hints</a> in Python 3 and <a href="https://www.youtube.com/watch?v=2xWhaALHTvU" target="_blank">type checking tools like <code>mypy</code></a> this gap is starting to close.</p> <p>But not only does Python scale and grow with the project at hand, it also scales and grows <em>with your skills</em> as a developer.</p> <p>It’s relatively easy to get started with Python—but it’s not going to prevent you from growing as a developer and getting impressive real-world work done with it. My friend and fellow Python wrangler <a href="https://talkpython.fm/" target="_blank">Michael Kennedy</a> refers to it as a “full spectrum” language. And I really like that as an analogy.</p> <p>Python spans the gamut from <code>print('hello, world')</code> all the way to running the back-end infrastructure for massive applications like <a href="why-learn-python">Reddit, Instagram, or YouTube</a>.</p> </section><section><h2 class="crosshead thin">Now, is using Python<br><u>always</u> the right choice?</h2> <p>No.</p> <p>No single programming language is.</p> <p>For example, it’s unlikely you’re going to write a real-time operating system kernel in Python. Neither will id Software use it to implement their next-generation rendering engine…</p> <p>But millions of developers around the world are using Python to build web applications, write data-crunching pipelines, generate reports, automate tests, conduct research, and do all kinds of other amazing work in a multitude of domains.</p> </section><section><h2 class="crosshead thin">By learning Python you’re not limiting yourself to a specific niche.</h2> <p>And that’s what I love about this adorable, “adjustable desk” of a language.</p> <p>Happy Pythoning!</p></section><footer></footer></article>https://dbader.org/blog/when-to-use-pythonThu, 18 May 2017 00:00:00 GMTStacks in Pythonhttps://dbader.org/blog/stacks-in-python<article><header><h1>Stacks in Python</h1> <p>How to implement a stack data structure (LIFO) in Python using built-in types and classes from the standard library.</p> </header><section><figure><img alt="" src="/blog/figures/stacks-in-python.png" width="1280" height="720"></figure> <p>A stack is a collection of objects that supports fast <em>last-in, first-out (LIFO)</em> semantics for inserts and deletes. Unlike lists or arrays, stacks typically don’t allow for random access to the objects they contain. The insert and delete operations are also often called <em>push</em> and <em>pop</em>.</p> <p>A useful real-world analogy for a stack data structure is a <em>stack of plates</em>:</p> <blockquote> <p>New plates are added to the top of the stack. And because the plates are precious and heavy, only the topmost plate can be moved (last-in, first-out). To reach the plates lower down in the stack the topmost plates must be removed one by one.</p> </blockquote> <p>Stacks and queues are similar. They’re both linear collections of items and the difference lies in the order that items are accessed in:</p> <p>With a <strong>queue</strong> you remove the item <strong>least recently added</strong> (<em>first-in, first-out</em> or <em>FIFO</em>); and with a <strong>stack</strong> you remove the item <strong>most recently added</strong> (<em>last-in, first-out</em> or <em>LIFO</em>).</p> <p>Performance-wise, a proper stack implementation is expected to take <em>O(1)</em> time for insert and delete operations.</p> <p>Stacks have a wide range of uses in algorithms, for example in language parsing and <a href="https://en.wikipedia.org/wiki/Stack-based_memory_allocation" target="_blank">runtime memory management (“call stack”)</a>. A short and beautiful algorithm using a stack is <a href="https://en.wikipedia.org/wiki/Depth-first_search" target="_blank">depth-first search (DFS)</a> on a tree or graph data structure.</p> <p>Python ships with several stack implementations that each have slightly different characteristics. Let’s take a look at them:</p> </section><section><h2>✅ The <a href="https://docs.python.org/3/tutorial/datastructures.html#using-lists-as-stacks" target="_blank">list</a> Built-in</h2> <p>Python’s built-in <code>list</code> type makes a decent stack data structure as it supports push and pop operations in amortized <em>O(1)</em> time.</p> <p>Python’s lists are implemented as dynamic arrays internally which means they occasional need to resize the storage space for elements stored in them when elements are added or removed. The <a href="http://www.laurentluce.com/posts/python-list-implementation/" target="_blank">list over-allocates its backing storage</a> so that not every push or pop requires resizing and you get an amortized <em>O(1)</em> time complexity for these operations.</p> <p>The downside is that this makes their performance less consistent than the stable <em>O(1)</em> inserts and deletes provided by a linked list based implementation (like <code>collections.deque</code>, see below). On the other hand lists do provide fast <em>O(1)</em> time random access to elements on the stack which can be an added benefit.</p> <p>Here’s an <strong>important performance caveat</strong> when using lists as stacks:</p> <p>To get the amortized <em>O(1)</em> performance for inserts and deletes new items must be added to the end of the list with the <code>append()</code> method and removed again from the end using <code>pop()</code>. Stacks based on Python lists grow to the right and shrink to the left.</p> <p>Adding and removing from the front is much slower and takes <em>O(n)</em> time, as the existing elements must be shifted around to make room for the new element.</p> <div class="codehilite"><pre><span></span><span class="c1"># How to use a Python list as a stack (LIFO):</span> <span class="n">s</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">s</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'eat'</span><span class="p">)</span> <span class="n">s</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'sleep'</span><span class="p">)</span> <span class="n">s</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'code'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span> <span class="p">[</span><span class="s1">'eat'</span><span class="p">,</span> <span class="s1">'sleep'</span><span class="p">,</span> <span class="s1">'code'</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="s1">'code'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="s1">'sleep'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="s1">'eat'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="ne">IndexError</span><span class="p">:</span> <span class="s2">"pop from empty list"</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/collections.html#collections.deque" target="_blank">collections.deque</a> Class</h2> <p>The <code>deque</code> class implements a double-ended queue that supports adding and removing elements from either end in <em>O(1)</em> time (non-amortized).</p> <p>Because deques support adding and removing elements from either end equally well, they can serve both as queues and as stacks.</p> <p>Python’s <a href="https://github.com/python/cpython/blob/947629916a5ecb1f6f6792e9b9234e084c5bf274/Modules/_collectionsmodule.c#L24-L26" target="_blank">deque objects are implemented as doubly-linked lists</a> which gives them excellent and consistent performance for inserting and deleting elements, but poor <em>O(n)</em> performance for randomly accessing elements in the middle of the stack.</p> <p><code>collections.deque</code> is a great choice if you’re looking for a stack data structure in Python’s standard library with the performance characteristics of a linked-list implementation.</p> <div class="codehilite"><pre><span></span><span class="c1"># How to use collections.deque as a stack (LIFO):</span> <span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">deque</span> <span class="n">q</span> <span class="o">=</span> <span class="n">deque</span><span class="p">()</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'eat'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'sleep'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'code'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span> <span class="n">deque</span><span class="p">([</span><span class="s1">'eat'</span><span class="p">,</span> <span class="s1">'sleep'</span><span class="p">,</span> <span class="s1">'code'</span><span class="p">])</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="s1">'code'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="s1">'sleep'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="s1">'eat'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="ne">IndexError</span><span class="p">:</span> <span class="s2">"pop from an empty deque"</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/queue.html" target="_blank">queue.LifoQueue</a> Class</h2> <p>This stack implementation in the Python standard library is synchronized and provides locking semantics to support multiple concurrent producers and consumers.</p> <p>The <a href="https://docs.python.org/3/library/queue.html" target="_blank"><code>queue</code> module</a> contains several other classes implementing multi-producer, multi-consumer queues that are useful for parallel computing.</p> <p>Depending on your use case the locking semantics might be helpful, or just incur unneeded overhead. In this case you’d be better off with using a <code>list</code> or a <code>deque</code> as a general purpose stack.</p> <div class="codehilite"><pre><span></span><span class="c1"># How to use queue.LifoQueue as a stack:</span> <span class="kn">from</span> <span class="nn">queue</span> <span class="kn">import</span> <span class="n">LifoQueue</span> <span class="n">s</span> <span class="o">=</span> <span class="n">LifoQueue</span><span class="p">()</span> <span class="n">s</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'eat'</span><span class="p">)</span> <span class="n">s</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'sleep'</span><span class="p">)</span> <span class="n">s</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'code'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span> <span class="o">&lt;</span><span class="n">queue</span><span class="o">.</span><span class="n">LifoQueue</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x108298dd8</span><span class="o">&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'code'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'sleep'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'eat'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">get_nowait</span><span class="p">()</span> <span class="n">queue</span><span class="o">.</span><span class="n">Empty</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="c1"># Blocks / waits forever...</span> </pre></div> </section><section><h2>A good default choice: <code>collections.deque</code></h2> <p>If you’re not looking for parallel processing support (or don’t want to handle locking and unlocking manually) your choice comes down to the built-in <code>list</code> type or <code>collections.deque</code>.</p> <p>The difference lies in the data structure used behind the scenes and ease of use.</p> <ul> <li> <p><code>list</code> is backed by a dynamic array which makes it great for fast random access but requires occasional resizing when elements are added or removed. The list over-allocates its backing storage so that not every push or pop requires resizing and you get an amortized <em>O(1)</em> time complexity for these operations. But you do need to be careful to only insert and remove items from the right-hand side (<code>append</code> and <code>pop</code>) or otherwise performance slows down to <em>O(n)</em>.</p> </li> <li> <p><code>collections.deque</code> is backed by a doubly-linked list which optimizes appends and deletes at both ends and provides consistent <em>O(1)</em> performance for these operations. Not only is its performance more stable, the <code>deque</code> class is also easier to use because you don’t have to worry about adding or removing items from “the wrong end.”</p> </li> </ul> <p>For these reasons, <code>collections.deque</code> makes an excellent choice for implementing a stack (LIFO queue) data structure in Python.</p> <p><em><a href="fundamental-data-structures-in-python">Read the full “Fundamental Data Structures in Python” article series here</a>. This article is missing something or you found an error? Help a brother out and leave a comment below.</em></p></section><footer></footer></article>https://dbader.org/blog/stacks-in-pythonTue, 16 May 2017 00:00:00 GMTLet’s Program with Python: Reacting to User Input (Part 4)https://dbader.org/blog/python-intro-reacting-to-user-input<article><header><h1>Let’s Program with Python: Reacting to User Input (Part 4)</h1> <p>In the fourth (and final) class in this series you’ll learn how to make your Python programs interactive by letting them react to user input.</p> </header><section><figure><img alt="" src="/blog/figures/pywelcome-part4.png" width="1280" height="720"></figure> <p><em>In this guest post series by <a href="#author">Doug Farrell</a> you’ll learn the basics of programming with Python from scratch. If you’ve never programmed before or need a fun little class to work through with your kids, you’re welcome to follow along.</em></p> <p><strong>Looking for the rest of the “Let’s Program with Python” series?</strong> Here you go:</p> <ul> <li><a href="python-intro-statements-variables-and-loops">Part 1: Statements, Variables, and Loops</a></li> <li><a href="python-intro-functions-and-lists">Part 2: Functions and Lists</a></li> <li><a href="python-intro-conditionals-and-if-statements">Part 3: Conditionals and “if” Statements</a></li> <li>Part 4: Reacting to User Input (This article)</li> </ul> </section><section><h2>Table of Contents – Part 4</h2> <ol> <li><a href="#lets-write-a-program">Let’s Write a Program Together</a></li> <li><a href="#getting-info-from-the-player">Getting Information From the Player</a></li> <li><a href="#converting-a-string-to-a-number">Converting a String to a Number</a></li> <li><a href="#another-kind-of-loop">Another Kind of Loop</a></li> <li><a href="#more-things-we-can-do-with-lists">More Things We Can Do With Lists</a></li> <li><a href="#how-many-items-are-in-a-list">How Many Items Are in a List?</a></li> <li><a href="#how-to-pick-random-things-from-a-list">How to Pick Random Things From a List?</a></li> <li><a href="#our-completed-guess-my-number-program">Our Completed “Guess My Number” Program</a></li> <li><a href="#congratulations">Congratulations!</a></li> <li><a href="#appendix">Appendix – Python Info That Doesn’t Fit in Class</a></li> </ol> </section><section><h2><a class="anchor" name="lets-write-a-program"></a> Let’s Write a Program Together</h2> <p>For this class we’re going to write a “Guess My Number” game program. In this game the program will pick a random number from 1 to 10 and the player will try to guess what the number is. The program will respond in different ways depending on whether the player guessed correctly or incorrectly. The player can also end the game whenever they want by telling the program to “quit”.</p> <p>The interesting part of this program is you’re going to tell me how to write it instead of the other way around. But before we get started, we need to learn a few more things about Python to help us build our game.</p> </section><section><h2><a class="anchor" name="getting-info-from-the-player"></a> Getting Information From the Player</h2> <p>In order to play our game the player has to interact with it. We need a way to get guesses from the player so the game can compare its secret number to the players guess. To do this we use the <code>input()</code> function.</p> <p>The <code>input()</code> function let’s us ask the user for some information, and then wait for them to enter something using the keyboard. In the Python interactive mode it looks like this:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">guess</span> <span class="o">=</span> <span class="nb">input</span><span class="p">(</span><span class="s2">"Please enter a number: "</span><span class="p">)</span> <span class="n">Please</span> <span class="n">enter</span> <span class="n">a</span> <span class="n">number</span><span class="p">:</span> </pre></div> <p>At the point where the <code>input()</code> function runs, the cursor is at the end of the <code>"Please enter a number: "</code> string, waiting for you to type something.</p> <p>You can type anything you want, when you hit the <code>&lt;ENTER&gt;</code> key whatever you typed will be assigned to the <code>guess</code> variable as a string. This is a very simple way to get input from the user using the keyboard.</p> </section><section><h2><a class="anchor" name="converting-a-string-to-a-number"></a> Converting a String to a Number</h2> <p>We haven’t talked about this yet, but there is a difference between a string like <code>"10"</code> and the number <code>10</code>. Try this in the interactive mode:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="mi">10</span> <span class="o">==</span> <span class="mi">10</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="s2">"10"</span> <span class="o">==</span> <span class="mi">10</span> <span class="bp">False</span> </pre></div> <p>On the first line we are comparing the two number 10’s to each other to see if they are equal. Python knows they are, so it responds by printing <code>True</code> to the screen.</p> <p>But the next comparison, <code>"10" == 10</code>, why does Python respond with <code>False</code>? The simple answer is Python doesn’t think they’re equal.</p> <p>But why aren’t they equal? This can be confusing, <code>"10"</code> looks like the number ten. And <code>10</code> definitely looks like the number ten as well. For Python however, this isn’t true.</p> <p>The number <code>10</code> is exactly that, the numerical value 10. The string <code>"10"</code> is just a string, it has no numerical value, even though <code>"10"</code> looks like ten to us.</p> <p>The difference is the representation. The <code>"10"</code> represents a string to Python, it doesn’t know that string represents ten to us. The <code>10</code> however does mean numerical ten to Python, ten things, ten cars, ten whatever.</p> <p>What does this have to do with our game? A lot actually. When the game starts the program will randomly pick a <em>number</em> from 1 to 10, not a string, a number. However when the player types something into our <code>guess = input("Please enter a number: ")</code> prompt, <code>guess</code> is a string variable.</p> <p>Even if the player enters a “1” and then a “0” and then hits enter, the <code>guess</code> variable will be a string. This is where a problem comes in. Let’s say we call the game’s variable for its number <code>secret_number</code>. If we write some Python code that compares them, like this:</p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="n">secret_number</span> <span class="o">==</span> <span class="n">guess</span><span class="p">:</span> </pre></div> <p>This code will fail because comparing a string to a number will always be <code>False</code>. We need to make Python compare two of the same kinds of things. For our game, both things need to be numbers. We need to convert the player’s <code>guess</code> variable to a number. Python can do this using the <code>int()</code> function. It looks like this:</p> <div class="codehilite"><pre><span></span><span class="n">guess_number</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">guess</span><span class="p">)</span> </pre></div> <p>With this code we’re taking the player’s input, <code>guess</code>, which could be something like “8”, and converting it to the numerical value 8 and assigning it to the new variable <code>guess_number</code>. Now when we compare <code>guess_number</code> with <code>secret_number</code>, they are the same kind of thing (numbers) and will compare correctly when we write Python code like this:</p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="n">guess_number</span> <span class="o">==</span> <span class="n">secret_number</span><span class="p">:</span> </pre></div> </section><section><h2><a class="anchor" name="another-kind-of-loop"></a> Another Kind of Loop</h2> <p>We’ve only used the <code>for</code> loop so far because it’s handy when you know ahead of time how many times you want to loop. For our game program we won’t know ahead of time how many guesses it will take our player to guess the <code>secret_number</code>. We also don’t know how many times they’ll want to play the game.</p> <p>This is a perfect use for the other loop Python supports, the <code>while</code> loop. The <code>while</code> loop is called a <em>conditional loop</em> because it will continue looping until some condition it is testing is True. Here’s an example of a <code>while</code> loop:</p> <div class="codehilite"><pre><span></span><span class="n">game_running</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">while</span> <span class="n">game_running</span><span class="p">:</span> <span class="c1"># Run some Python statements</span> </pre></div> <p>What these program lines mean is that while the variable <code>game_running</code> is <code>True</code>, the while loop will keep looping. This also means something in the <code>while</code> loop will have to change the value of <code>game_running</code> in order for the program to exit the loop.</p> <p>Forgetting to provide a way for the <code>while</code> loop to end creates what’s called an <em>infinite loop</em>. This is usually a bad thing and means in order to exit the program it has to be crashed or stopped in some other way.</p> </section><section><h2><a class="anchor" name="more-things-we-can-do-with-lists"></a> More Things We Can Do With Lists</h2> <p>We’ve used Python lists before to hold things we want to deal with as one thing, like lists of turtles. We’ve created lists and appended things to lists. So far we’ve used the things in the list one at a time using the <code>for</code> loop. But how do we get to the individual things inside a list? For example, suppose I have this list in Python:</p> <div class="codehilite"><pre><span></span><span class="n">names</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"Andy"</span><span class="p">,</span> <span class="s2">"George"</span><span class="p">,</span> <span class="s2">"Sally"</span><span class="p">,</span> <span class="s2">"Sharon"</span><span class="p">,</span> <span class="s2">"Sam"</span><span class="p">,</span> <span class="s2">"Chris"</span><span class="p">]</span> </pre></div> <p>How can I get just the <code>"Sally"</code> name from the <code>names</code> list variable? We use something called <em>list indexing</em> to do that. Everything in a list has a position in the list, and all lists in Python start at position 0. The position is called an index, so to get <code>"Sally"</code> from the list, remembering all lists start at index 0, we do this:</p> <div class="codehilite"><pre><span></span><span class="n">name</span> <span class="o">=</span> <span class="n">names</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> </pre></div> <p>When we do this the variable <code>name</code> will be equal to <code>"Sally"</code> from our list. The <code>[2]</code> above is called the index into the list. We’ve told Python we want the thing inside the <code>names</code> list at index 2.</p> </section><section><h2><a class="anchor" name="how-many-items-are-in-a-list"></a> How Many Items Are in a List?</h2> <p>It’s often useful to be able to find out how many things are in a list. For instance, our <code>names</code> list above has six strings in it. But how could we find this out using Python? We use the <code>len()</code> function. It looks like this:</p> <div class="codehilite"><pre><span></span><span class="n">number_of_names_in_list</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">names</span><span class="p">)</span> </pre></div> <p>This will set the variable <code>number_of_names_in_list</code> equal to six. Notice something about the number of items in the <code>names</code> list and the largest index, the name “Chris”. To get the name “Chris” from our <code>names</code> list we would do this:</p> <div class="codehilite"><pre><span></span><span class="n">name</span> <span class="o">=</span> <span class="n">names</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span> </pre></div> <p>The last thing in the list is at index 5, but the number of things in the list is 6. This is because all lists start with index 0, which is included in the number of things in the list. So for the names list we have indexes 0, 1, 2, 3, 4 and 5, totaling 6 things.</p> </section><section><h2><a class="anchor" name="how-to-pick-random-things-from-a-list"></a> How to Pick Random Things From a List?</h2> <p>Now we know how to pick individual things from a list, how to determine how long a list is and what the maximum index value in a list is. Can we use this information to choose a random thing from a list? For a minute let’s think about our turtle programs, we had a list something like this:</p> <div class="codehilite"><pre><span></span><span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"black"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">,</span> <span class="s2">"organge"</span><span class="p">,</span> <span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"green"</span><span class="p">,</span> <span class="s2">"blue"</span><span class="p">]</span> </pre></div> <p>How could we pick a random color from this list to use when we were creating a turtle? We know the smallest index is 0, which would be the color “black”. We also know by looking at the list that our largest index is 5, the color blue. This is one less than the number of colors in the list. So we could do something like this:</p> <div class="codehilite"><pre><span></span><span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"black"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">,</span> <span class="s2">"organge"</span><span class="p">,</span> <span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"green"</span><span class="p">,</span> <span class="s2">"blue"</span><span class="p">]</span> <span class="n">turtle_color</span> <span class="o">=</span> <span class="n">colors</span><span class="p">[</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">5</span><span class="p">)]</span> </pre></div> <p>This Python statement would set the <code>turtle_color</code> variable to a random color from our <code>colors</code> list. But what if we added more colors to our list? Something like this:</p> <div class="codehilite"><pre><span></span><span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"black"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">,</span> <span class="s2">"organge"</span><span class="p">,</span> <span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"green"</span><span class="p">,</span> <span class="s2">"blue"</span><span class="p">,</span> <span class="s2">"violet"</span><span class="p">,</span> <span class="s2">"pink"</span><span class="p">]</span> <span class="n">turtle_color</span> <span class="o">=</span> <span class="n">colors</span><span class="p">[</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">5</span><span class="p">)]</span> </pre></div> <p>Unless we change the <code>5</code> in the <code>random.randint(5)</code> function we’ll still be picking from the first six colors and ignoring the new ones we added. What if we’re picking random colors all over our program, we’d have to change all the lines that pick a color every time we changed the number of colors in our <code>colors</code> list. Can we get Python to handle this for us? Sure we can, we can use the <code>len()</code> function to help us out. We can change our code to look like this:</p> <div class="codehilite"><pre><span></span><span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"black"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">,</span> <span class="s2">"organge"</span><span class="p">,</span> <span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"green"</span><span class="p">,</span> <span class="s2">"blue"</span><span class="p">,</span> <span class="s2">"violet"</span><span class="p">,</span> <span class="s2">"pink"</span><span class="p">]</span> <span class="n">turtle_color</span> <span class="o">=</span> <span class="n">colors</span><span class="p">[</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">colors</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)]</span> </pre></div> <p>What’s going on here? We still have our <code>colors</code> list variable, but now we’re using the <code>len()</code> function inside our <code>random.randint()</code> function. This is okay, the <code>len()</code> function returns a number and <code>random.randint()</code> expects a number as its second parameter.</p> <p>But now we’re telling <code>random.randint()</code> the upper index limit of the numbers we want to choose from is one less than the number of things in the <code>colors</code> list variable. And as we’ve seen, one less than the number of things in a list will always be the highest index in the list. By using the code above we can add or subtract as many items from the <code>colors</code> list as we want and our random selection will still work, using all the things in the list.</p> </section><section><h2><a class="anchor" name="our-completed-guess-my-number-program"></a> Our Completed “Guess My Number” Program</h2> <p>Here’s our Guess My Number program, complete with comments:</p> <div class="codehilite"><pre><span></span><span class="c1">#</span> <span class="c1"># Guess My Number</span> <span class="c1">#</span> <span class="kn">import</span> <span class="nn">random</span> <span class="c1"># Set our game ending flag to False</span> <span class="n">game_running</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">while</span> <span class="n">game_running</span><span class="p">:</span> <span class="c1"># Greet the user to our game</span> <span class="k">print</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="s2">"I'm thinking of a number between 1 and 10, can you guess it?"</span><span class="p">)</span> <span class="c1"># Have the program pick a random number between 1 and 10</span> <span class="n">secret_number</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="c1"># Set the player's guess number to something outside the range</span> <span class="n">guess_number</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span> <span class="c1"># Loop until the player guesses our number</span> <span class="k">while</span> <span class="n">guess_number</span> <span class="o">!=</span> <span class="n">secret_number</span><span class="p">:</span> <span class="c1"># Get the player's guess from the player</span> <span class="k">print</span><span class="p">()</span> <span class="n">guess</span> <span class="o">=</span> <span class="nb">input</span><span class="p">(</span><span class="s2">"Please enter a number: "</span><span class="p">)</span> <span class="c1"># Does the user want to quit playing?</span> <span class="k">if</span> <span class="n">guess</span> <span class="o">==</span> <span class="s2">"quit"</span><span class="p">:</span> <span class="n">game_running</span> <span class="o">=</span> <span class="bp">False</span> <span class="k">break</span> <span class="c1"># Otherwise, nope, player wants to keep going</span> <span class="k">else</span><span class="p">:</span> <span class="c1"># Convert the players guess from a string to an integer</span> <span class="n">guess_number</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">guess</span><span class="p">)</span> <span class="c1"># Did the player guess the program's number?</span> <span class="k">if</span> <span class="n">guess_number</span> <span class="o">==</span> <span class="n">secret_number</span><span class="p">:</span> <span class="k">print</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="s2">"Congratulations, you guessed my number!"</span><span class="p">)</span> <span class="c1"># Otherwise, whoops, nope, go around again</span> <span class="k">else</span><span class="p">:</span> <span class="k">print</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="s2">"Oh, to bad, that's not my number..."</span><span class="p">)</span> <span class="c1"># Say goodbye to the player</span> <span class="k">print</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="s2">"Thanks for playing!"</span><span class="p">)</span> </pre></div> </section><section><h2><a class="anchor" name="congratulations"></a> Congratulations!</h2> <p>We’ve completed our course and I hope you’ve had as much fun as I had! We’ve written some pretty amazing programs together and learned quite a bit about programming and Python along the way. My wish is this interested you enough to keep learning about programming and to continue on to discover new things you can do with Python.</p> <hr> </section><section><h2><a class="anchor" name="appendix"></a>Appendix – Python Info That Doesn’t Fit in Class</h2> </section><section><h2>Differences Between Python and Other Languages</h2> <p>There are many programming languages out in the wild you can use to program a computer. Some have been around for a long time, like Fortran and C, and some are quite new, like Dart or Go. Python falls in the middle ground of being fairly new, but quite mature.</p> <p>Why would a programmer choose one language to learn over another? That’s a somewhat complicated question as most languages will allow you to do anything you want. However it can be difficult to express what you want to do with a particular language instead of something else.</p> <p>For instance, Fortran excels at computation and in fact it’s name comes from Fromula Translation (ForTran). However it’s not known as a great language if you need to do a lot of string/text manipulation. The C programming language is a great language if your goal is to maximize the performance of your program. If you program it well you can create extremely fast programs. Notice I said “if you program it well”, if you don’t you can completely crash not only your program, but perhaps even your computer. The C language doesn’t hold your hand to prevent you from doing things that could be bad for your program.</p> <p>In addition to how well a language fits the problem you’re trying to solve, it might not be able to be used with the tools you like, or might not provide the tools you need, a particular language just may not appeal to you visually and appear ugly to you.</p> <p>My choice of teaching Python fits a “sweet spot” for me. It’s fast enough to create the kinds of programs I want to create. It’s visually very appealing to me, and the grammar and syntax of the language fit the way I want to express the problems I’m trying to solve.</p> </section><section><h2>Python Vocabulary</h2> <p>Let’s talk about some of the vocabulary used in the class and what it means. Programming languages have their own “jargon”, or words, meaning specific things to programmers and that language. Here are some terms we’ve used in relation to Python.</p> <p><strong>IDLE – command prompt</strong>: IDLE is the programming environment that comes with Python. It’s what’s called an IDE, or Integrated Development Environment, and pulls together some useful things to help write Python programs. When you start IDLE it opens up a window that has the Python interactive prompt <code>&gt;&gt;&gt;</code> in it.</p> <p>This is a window running the Python interpreter in interactive mode. This is where you can play around with some simple Python program statements. It’s kind of a sandbox where you can try things out. However, there is no way to save or edit your work; once the Python interpreter runs your statements, they’re gone.</p> <p><strong>IDLE – editor window</strong>: The file window (File → New Window) opens up a simple text editor. This is like Notepad in Windows, except it knows about Python code, how to format it and colorizes the text. This is where you can write, edit and save your work and run it again later. When you run this code, behind the scenes IDLE is running the program in the Python interpreter, just like it is in the first IDLE window.</p> <p><strong>Syntax Highlighting</strong>: When we edit code in the file window of IDLE it knows about Python code. One of the things this means is the editor can “colorize”, or syntax highlight, various parts of the Python code you’re entering. It sets the keywords of Python, like for and if, to certain colors. Strings to other colors and comments to another. This is just the file window being helpful and providing syntax highlighting to make it easier for the programmer to read and understand what’s going on in the program.</p> <p><strong>Python Command Line</strong>: In Windows if you open up a command line window, what used to be called a DOS box, and run python, the system will respond with the Python command prompt <code>&gt;&gt;&gt;</code>. At this point you’re running Python in it’s interactive mode, just like when you’re inside of IDLE. In fact they are the same thing, IDLE is running it’s own Python command line inside the window, they are functionally identical.</p> <p>You might think “what use is that?”, and I agree, I’d rather work in IDLE if I’m going to using the interactive mode and play with the sandbox mode and the <code>&gt;&gt;&gt;</code> command prompt. The real use of the Python command line is when you enter something like this at the system command prompt:</p> <div class="codehilite"><pre><span></span>python myprogram.py </pre></div> <p>If I’ve written a program called <code>myprogram.py</code> and entered the line above, instead of going into interactive mode, Python will read <code>myprogram.py</code> and run the code. This is very useful if you’re written a program you want to use and not run inside of IDLE. As a programmer I run programs in this manner all day long, and in many cases these programs run essentially forever as servers.</p> <p><strong>Attribute and Property</strong>: We’ve thrown around the terms “attribute” and “property” kind of randomly, and this can lead to some confusion. The reason it’s confusing is these things mean essentially the same thing. When talking about programming there is always the goal to use specific words and terms to eliminate confusion about what you’re talking about.</p> <p>For instance let’s talk about you. You have many qualities that different people want to express. Your friends want to know your name and phone number. Your school wants to know that as well, and your age, the grade you’re in and our attendance record. In programming terms we can think of these as attributes or properties about you.</p> <p>The attributes and properties of a thing (you for example) help get more specific information about the thing. And the specific information wanted depends on the audience asking. For example when meeting someone new they are more likely to be interested in your name property. Whereas your school might be more interested in your attendance property.</p> <p>In Python we’ve been working with turtles, and those turtles have attributes and properties. For example a turtle as a property called forward. This property happens to be a function that moves the turtle forward, but it’s still a property of the turtle. In fact all the properties and attributes associated with a turtle are expressed as functions. These functions either make the turtle do something, or tell us something about the turtle.</p> <p>Attributes and properties lead into a concept of Object Oriented Programming (OOP) that adds the concept of “things” to programs rather than just data and statements. Object Oriented Programming is beyond the scope of this book, but is very interesting and useful.</p> </section><section><h2>Interpreter vs Compiler</h2> <p>In class you’ve heard me talk about the Python interpreter, what does this mean. As we’ve talked about, computer languages are a way for people to tell a computer what to do. But the truth is a computer only understands 0’s and 1’s, so how does a computer understand a language like Python? That’s where a translation layer comes into play, and that translation layer is the interpreter for Python (and other interpreted languages) and a compiler for compiled languages. Let’s talk about compilers first.</p> <p><strong>Compiler</strong>: A compiler is a translator that converts a computer language into machine code, the 0’s and 1’s a computer understands. A compiler usually produces an executable file, on Windows machines this is a file that ends in .exe. This file contains machine code information the computer can run directly. Languages like C, C++ and Fortran are compiled languages and have to be processed by a compiler before the program can run. One thing this means is you can’t run a compiled language directly, you have to compile it first. It also means there is nothing like the interactive mode (the <code>&gt;&gt;&gt;</code> prompt in Python) in a compiled language. The entire program has to be compiled, it can’t compile and run single statements.</p> <p><strong>Interpreter</strong>: Here’s where things get a little more confusing. Most interpreted languages also have a compiled step, but the output of that step isn’t machine code, no 0’s and 1’s. Instead the compilation step produces what is called ByteCode. The ByteCode is kind of an intermediate step between the near English computer language and the machine code understood by the computer.</p> <p>The ByteCode can’t be run directly, it is run by a thing called a virtual machine. When the program is run, the virtual machine reads the ByteCode and it generates the computer specific machine code that actually is run by the computer. When you run the program the virtual machine is constantly “interpreting” the ByteCode and generating computer specific machine code. Unlike a compiled language, languages like Python with virtual machines can provide an interactive mode (the <code>&gt;&gt;&gt;</code> prompt) as the interpreter and virtual machine can translate and run program statements on the fly.</p> <p><strong>Advantages And Disadvantages</strong>: So why would a programmer pick a compiled language over an interpreted language, and vice versa? Well, what we said before still applies, expressiveness of the language, style, etc, those are important things to think about when choosing a language for a project. But there are some differences beyond that. In general compiled languages produce programs that run faster than programs produced by an interpreter. Remember, compiled languages produce programs containing machine code that can be run directly, whereas interpreted languages usually have a virtual machine between the ByteCode and the machine code, so there’s a speed penalty there. However, also keep in mind modern computers are so fast this difference is less important. In addition, interpreted languages are being constantly improved so their performance gets better and better, so the performance difference between the two is shrinking.</p> <p>Most interpreted languages also offer safety features to prevent the programmer from crashing the program. Interpreted languages make it hard to corrupt memory. They make it difficult to get direct access to the hardware. They don’t force the programmer to manage memory explicitly. Compiled programs like C offer none of this, and therefore it’s easy to do all of those things, which can put your program at risk, unless you’re a skilled programmer. The safety features can be added to a C program, but this has to be done manually by the programmer and isn’t handled by the language natively.</p> </section><section><h2>Python Reference Materials</h2> <p>Included below is a list of reference materials to help you go further in your study of Python.</p> <ul> <li><a href="https://www.python.org/" target="_blank">Python Website</a> – Main Python website</li> <li><a href="https://docs.python.org/3/" target="_blank">Python Documentation</a> – Official Python 3 Documentation</li> <li><a href="https://docs.python.org/3/library/turtle.html" target="_blank">Python Turtle Documentation</a> – Official Python Documentation for Turtle</li> <li><a href="/python-basics">Python Tutorials for Beginners on dbader.org</a></li> <li><a href="http://www.learnpython.org/" target="_blank">Learn Python</a> – An interesting tutorial to help learn Python</li> <li><a href="http://interactivepython.org/runestone/static/thinkcspy/toc.html" target="_blank">How To Think Like A Computer Scientist</a> – Interesting and interactive way to learn Python</li> <li><a href="http://www.pygame.org/news.html" target="_blank">PyGame</a> – An add on module for writing games with Python</li> </ul></section><footer></footer></article>https://dbader.org/blog/python-intro-reacting-to-user-inputThu, 11 May 2017 00:00:00 GMTPythonistaCafe: A Peer-to-Peer Learning Community for Python Developershttps://dbader.org/blog/introducing-pythonistacafe<article><header><h1>PythonistaCafe: A Peer-to-Peer Learning Community for Python Developers</h1> <p>Introducing PythonistaCafe—an invite-only, online community of Python and software development enthusiasts helping each other succeed and grow.</p> </header><section><figure><img alt="Introducing Pythonista Cafe" src="/blog/figures/pythonistacafe-announcement.png" width="670" height="335"></figure> <p>Most programmers I know say they sometimes feel “stuck” in their learning progress. Whether you’re a beginner, an intermediate developer, or an experienced senior dev—you’ll eventually hit a point where you feel like you’re no longer making progress:</p> <ul> <li>Where you think you’re no longer learning new things.</li> <li>Where it feels like you’re making <em>backwards</em> progress, and your skills seemingly atrophy and get <em>worse</em> over time; or</li> <li>Where you recognize you’re good at what you do—but you don’t know what to learn next, or how.</li> </ul> <p>It happens to all of us. For example, I found that I’m the happiest when I can learn new things and then apply them in practice or teach them to others. Every time I hit a plateau like that this feeling of being “stuck” strikes at the core of my identity. If it lasts for too long I get uneasy. It’s almost like I’m losing my sense of purpose.</p> <p>Not a nice feeling at all.</p> <p>But I’ve also been around the block enough that I know I can overcome it, that I can shake it off eventually. What usually helps me get my bearings straight again is talking to my friends who also work in the tech industry. I found it’s important to talk with people who are in the same boat:</p> <p>The folks that I went to university with or former colleagues who can relate to my “geek angst.”</p> <p>I get it that many of the things us programmers are struggling with seem funny or weird to “outsiders”—</p> <p>I’m not expecting my friend who works as an insurance broker to understand the strange headspace I can get in when I think that “my learning progress is stuck” (after having worked in the industry for a decade.)</p> <p>Let’s be honest here, this sounds ridiculous to a “normal” person… 🙂</p> <p>But it’s a real feeling—and one that’s quite common in the programming community.</p> <p>The quickest way I found to overcome it is to talk to people that you feel comfortable pouring your geeky heart out to. Whether you’re learning Python on your own, working by yourself as a freelancer, or if you’re the only Pythonista at your company:</p> <p>Experiencing this sense of community and exchanging your thoughts with other techies will have a huge benefit on your quality of life. I know it did on mine. And I want every developer I know to be able to experience the same:</p> <p>That’s why I started <a href="https://www.pythonistacafe.com?utm_source=dbaderorg&amp;utm_medium=web&amp;utm_campaign=pc-announcement" target="_blank">PythonistaCafe, a peer-to-peer learning community for Python developers</a>.</p> <p>It’s an idea I’ve been kicking around and refining since July 2016. And over the last few months it finally became a reality.</p> <p>A good way to think of PythonistaCafe is to see it as a <em>club of mutual improvement for Python enthusiasts</em>.</p> <p>We have members located all over the world, and with a wide range of proficiency levels. I’m impressed by their diverse skill set and the depth and quality of the conversations we had. Every day we discuss a broad range of programming questions, career advice, and other topics:</p> <figure><img alt="PythonistaCafe Screenshot" src="/blog/figures/pythonistacafe-screenshot.jpg" width="1122" height="366"></figure> <p>We now even have some open-source projects and Kaggle data science competitions we’re collaborating on to help people build up their portfolio and gain experience. It’s been a ton of fun—and a great “Python support group.”</p> <p>You can learn more about PythonistaCafe, our community values, and what we’re all about at <a href="https://www.pythonistacafe.com?utm_source=dbaderorg&amp;utm_medium=web&amp;utm_campaign=pc-announcement" target="_blank"><strong>www.pythonistacafe.com</strong></a>.</p> <p>In part, I also started PythonistaCafe to “scratch my own itch”—it is my new home when it comes to Python. Each week I receive a ton of emails asking me for programming or career advice. And to be honest, it’s hard to keep up with them all.</p> <p>If you need access to me to help solve a Python problem or get advice in what direction to go, the <a href="https://www.pythonistacafe.com?utm_source=dbaderorg&amp;utm_medium=web&amp;utm_campaign=pc-announcement" target="_blank">PythonistaCafe forums</a> are where you can find me. I’m checking the forums and replying to topics and questions every single day. If you’re looking for an alternative to my <a href="https://dbader.org/products/mentorship/">1:1 Python coaching program</a> a membership in <a href="https://www.pythonistacafe.com?utm_source=dbaderorg&amp;utm_medium=web&amp;utm_campaign=pc-announcement" target="_blank">PythonistaCafe</a> might be a great fit for you.</p></section><footer></footer></article>https://dbader.org/blog/introducing-pythonistacafeTue, 09 May 2017 00:00:00 GMTLet’s Program with Python: Conditionals and “if” Statements (Part 3)https://dbader.org/blog/python-intro-conditionals-and-if-statements<article><header><h1>Let’s Program with Python: Conditionals and “if” Statements (Part 3)</h1> <p>In part three of this four-part Python introduction you’ll see how to teach your program how to make decisions with conditionals and if-statements.</p> </header><section><figure><img alt="" src="/blog/figures/pywelcome-part3.png" width="1280" height="720"></figure> <p><em>In this guest post series by <a href="#author">Doug Farrell</a> you’ll learn the basics of programming with Python from scratch. If you’ve never programmed before or need a fun little class to work through with your kids, you’re welcome to follow along.</em></p> <p><strong>Looking for the rest of the “Let’s Program with Python” series?</strong> Here you go:</p> <ul> <li><a href="python-intro-statements-variables-and-loops">Part 1: Statements, Variables, and Loops</a></li> <li><a href="python-intro-functions-and-lists">Part 2: Functions and Lists</a></li> <li>Part 3: Conditionals and “if” Statements (This article)</li> <li><a href="python-intro-reacting-to-user-input">Part 4: Reacting to User Input</a></li> </ul> </section><section><h2>Table of Contents – Part 3</h2> <ul> <li><a href="#lets-get-those-turtles-thinking">Let’s Get Those Turtles Thinking</a></li> <li><a href="#new-turtle-drawing-functions">New Turtle Drawing Functions</a></li> <li><a href="#new-modules-and-functions">New Modules and Functions</a></li> <li><a href="#lets-get-our-program-going">Let’s Get Our Program Going</a></li> <li><a href="#conditionals-and-if-statements">Conditionals and “if” Statements</a></li> <li><a href="#conclusion">Conclusion</a></li> </ul> </section><section><h2><a class="anchor" name="lets-get-those-turtles-thinking"></a> Let’s Get Those Turtles Thinking</h2> <p>In our last class we used a Python list to help us get multiple turtles drawing on the screen. We could keep adding turtles to our hearts content and the program would faithfully make each turtle draw our flower. This worked great for drawing the well controlled structure of the flower.</p> <p>But what if we want to draw something that’s randomly generated, something where the turtles draw something and we don’t know ahead of time what that will be? How can we use what we know already to help us do that?</p> <p>Let’s teach our program how to make decisions and do things on its own. Here is an image of one possible graphical outcome for our class:</p> <figure><img alt="image" src="/blog/figures/pywelcome-random-turtles.png" width="1280" height="1102"></figure> </section><section><h2><a class="anchor" name="new-turtle-drawing-functions"></a> New Turtle Drawing Functions</h2> <p>We’re going to create a new program where our turtles use some new drawing functions and new modules to create a randomly drawn image. Let’s learn the new turtle drawing functions first.</p> <p>Let’s start out by starting <strong>Idle</strong>, opening a new program editor window and creating a new Python program. In this new program let’s start as we’ve done before by entering this Python statement:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> </pre></div> <p>Save this program to a new file name, somewhere you can remember where to find it.</p> <p><strong>Get the Turtle Screen: turtle.Screen()</strong></p> <p>The first new turtle drawing function we’re going to learn isn’t really about the turtles at all, but about the screen they draw on. Up until now we haven’t cared to much about the screen the turtles are drawing on, we’ve just let the turtles create it as needed and away we go.</p> <p>But now we want to modify something about the screen. In order to do that we have to first get the screen in a manner that we can change it. As with everything we do with Python programming, any time we want to get something so we can modify it, we save it to a variable. To get the screen we enter the following into our new program:</p> <div class="codehilite"><pre><span></span><span class="n">screen</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Screen</span><span class="p">()</span> </pre></div> <p>This calls another function of our <code>turtle</code> module, <code>Screen()</code>, which gets the screen the module will use to draw turtles on, and saves it in the newly created variable <code>screen</code>.</p> <p>Notice how the <code>Screen()</code> function of the turtle module has it’s first letter capitalized, like when we create a turtle with <code>Turtle()</code>.</p> <p><strong>Set the Screen Size: turtle.setup()</strong></p> <p>Until now we’ve let the turtle module create our window to be whatever size it wants. We can control this using the <code>setup()</code> function of a turtle. I’m not sure why this is a turtle function instead of a screen function, but sometimes programming is like that. This function looks like this:</p> <div class="codehilite"><pre><span></span><span class="n">turtle</span><span class="o">.</span><span class="n">setup</span><span class="p">(</span><span class="mi">1024</span><span class="p">,</span> <span class="mi">768</span><span class="p">)</span> </pre></div> <p>This Python statement sets our turtle drawing window to be 1024 pixels wide by 768 pixels tall.</p> <p><strong>Set the Background Color of the Screen: screen.bgcolor()</strong></p> <p>Now that we have a variable that represents the screen, we can modify a feature of it. We’re going to change the background color from white to some other color. We do this using this Python statement:</p> <div class="codehilite"><pre><span></span><span class="n">screen</span><span class="o">.</span><span class="n">bgcolor</span><span class="p">(</span><span class="s2">"#FFFFE0"</span><span class="p">)</span> </pre></div> <p>This statement shows how to use the <code>screen</code> variable and call one of its functions, <code>bgcolor()</code> (short for <em>background color</em>) to set the background color on the screen.</p> <p>If we save and run this you’ll see an empty turtle window that has a light yellow color instead of white. The light yellow color is the <code>"#FFFFE0"</code> we passed as a parameter to the <code>bgcolor()</code> function.</p> <p>So what does <code>"#FFFFE0"</code> mean? We could have just passed <code>"yellow"</code> to the <code>bgcolor()</code> function, like we’ve done with our turtles, but that yellow is pretty intense and I wanted something lighter for a background color.</p> <p>So we’ve used a different way to define a color, this way comes right out of <strong>HTML</strong> (web page) coding. The <code>"#FFFFE0"</code> value represents setting the RGB (Red / Green / Blue) color value, each two character portion of the string <code>FFFFE0</code> represents a value from 0 - 255 in <a href="https://en.wikipedia.org/wiki/Hexadecimal" target="_blank">hexadecimal</a> (base 16, common in programming). This breaks down like this:</p> <div class="codehilite"><pre><span></span>FF FF E0 | | | | | +--- 224 Blue | +------ 255 Green +--------- 255 Red </pre></div> <p>This somewhat complex color code let’s us pick a color much more precisely than the limited pre-defined set of named colors (like <code>"red"</code> or <code>"yellow"</code>) that are inside the turtle module.</p> <p><strong>Turtles Are Rubber Stamps!</strong></p> <p>We can also use our turtles as rubber stamps! By this I mean we can tell the turtle to leave a permanent image of itself at any point the turtle exists on the screen. We do this by using the turtle <code>stamp()</code> function, which looks like this:</p> <div class="codehilite"><pre><span></span><span class="n">turtle</span><span class="o">.</span><span class="n">stamp</span><span class="p">()</span> </pre></div> <p>Running this Python statement makes a “stamp” of our turtle on the screen. When next we move the turtle you’ll see the stamp it left behind, kind of like bread crumbs of where its been. Let’s see how this works by entering the following into our program to make it look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="n">screen</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Screen</span><span class="p">()</span> <span class="n">turtle</span><span class="o">.</span><span class="n">setup</span><span class="p">(</span><span class="mi">1024</span><span class="p">,</span> <span class="mi">768</span><span class="p">)</span> <span class="n">screen</span><span class="o">.</span><span class="n">bgcolor</span><span class="p">(</span><span class="s2">"#FFFFE0"</span><span class="p">)</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"red"</span><span class="p">)</span> <span class="k">for</span> <span class="n">side</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">4</span><span class="p">):</span> <span class="n">t1</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">stamp</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> </pre></div> <p>When we save and run this program we should end up with a box outlined in red and a turtle “stamp” at each corner. The screen should look like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-stamping-1.png" width="1024" height="790"></figure> </section><section><h2><a class="anchor" name="new-modules-and-functions"></a> New Modules and Functions</h2> <p>In order to make our new program have random behavior we need to import a new module, logically enough called “random”. The <code>random</code> module, like the <code>turtle</code> module, brings additional functionality into our program so we can use it.</p> <p>Add this line at the top of our program right under the <code>import turtle</code> statement:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">random</span> </pre></div> <p>Just like the turtle module this doesn’t do anything immediately, but now our program has access to the functions in the <code>random</code> module.</p> <p><strong>Pick a Number, Any Number: random.randint()</strong></p> <p>The module <code>random</code>, as the name suggests, creates randomness. We’ll use the functions in the random module to make our turtle drawing less predictable, and maybe more interesting.</p> <p>One of those functions on the module is called <code>randint()</code>, and it generates random integers. If we jump over to our Idle interactive window we can try out the function.</p> <p>Enter this into our <strong>Idle</strong> interactive window to try the <code>randint()</code> function out:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="mi">4</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="mi">10</span> </pre></div> <p>You can see that just like the functions in the turtle module we have to use the module name random and a dot (<code>.</code>) character before the function we want.</p> <p>In the above lines we’ve used the <code>randint()</code> function twice and it returned a different number each time. This is what <code>randint()</code> does, it returns randomly generated integers. This also means that the numbers you’ll see in your Idle window when you run this example will (likely) be different.</p> <p>The two numbers we passed to it (0 and 10) are parameters telling <code>randint()</code> the beginning and ending limits of numbers we want it to generate. In our case we want integer numbers ranging from 0 to 10, including both 0 and 10. Random number generators are used a lot in game programming to create unexpected behavior and challenges for the player.</p> </section><section><h2><a class="anchor" name="lets-get-our-program-going"></a> Let’s Get Our Program Going</h2> <p>Let’s get our random turtle program going so we can add things to it. Make your program look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="kn">import</span> <span class="nn">random</span> <span class="n">screen</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Screen</span><span class="p">()</span> <span class="n">screen</span><span class="o">.</span><span class="n">bgcolor</span><span class="p">(</span><span class="s2">"#FFFFE0"</span><span class="p">)</span> <span class="k">for</span> <span class="n">move</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="k">for</span> <span class="n">a_turtle</span> <span class="ow">in</span> <span class="n">turtles</span><span class="p">:</span> <span class="n">move_turtle</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">)</span> </pre></div> <p>If we save and try to run the above program we’ll get errors. Why is this?</p> <p>Well for a couple of reasons, we have no variable named <code>turtles</code> and the function <code>move_turtle()</code> isn’t defined. Let’s fix that. Like our flower program we want to create a list of turtles, and we’ll need to define our <code>move_turtle()</code> function.</p> <p>So make your program look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="kn">import</span> <span class="nn">random</span> <span class="n">screen</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Screen</span><span class="p">()</span> <span class="n">screen</span><span class="o">.</span><span class="n">bgcolor</span><span class="p">(</span><span class="s2">"#FFFFE0"</span><span class="p">)</span> <span class="k">def</span> <span class="nf">move_turtle</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="k">pass</span> <span class="n">turtles</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">move</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="k">for</span> <span class="n">a_turtle</span> <span class="ow">in</span> <span class="n">turtles</span><span class="p">:</span> <span class="n">move_turtle</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">)</span> </pre></div> <p>Now when we save and run our program it doesn’t crash with an error, but it doesn’t do anything other than open a light yellow window.</p> <p>Why is that? Again, a couple of reasons. We haven’t defined any turtles in our <code>turtles</code> list variable. We’ve also defined our <code>move_turtle()</code> function, but it doesn’t do anything. The <code>pass</code> statement is just a placeholder that makes program work, but doesn’t provide any functionality. First things first, let’s create our turtles.</p> <p><strong>Getting a Variable From a Function</strong></p> <p>In our flower program when we wanted to create our turtles we did so by copying the turtle creation and setup code for every turtle we wanted. Then we put all those turtles into a list we called <code>turtles</code>.</p> <p>This works fine, but let’s do something clever and create a function to create our turtle for us. And let’s define it so we can set the color of the turtle by passing the color as a parameter to the function. Here’s a function that will do just that:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">create_turtle</span><span class="p">(</span><span class="n">color</span><span class="p">):</span> <span class="n">t</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="n">color</span><span class="p">)</span> <span class="k">return</span> <span class="n">t</span> </pre></div> <p>Notice at the end of the <code>create_turtle(color)</code> definition, the <code>return t</code> statement. What does this do?</p> <p>This is how to return the turtle we just created for use in the rest of the program. We’ve seen this before when we used the <code>t1 = turtle.Turtle()</code> statement. The <code>turtle.Turtle()</code> function <em>returns</em> a turtle, and that returned turtle is assigned to the variable <code>t1</code>. In our case we’re returning the turtle we created, what we called <code>t</code>, so it can be saved someplace in our program and used later.</p> <p>Now we have a function that will create a turtle for us that will draw with the color we’ve asked for. But we need to create multiple turtles to put into our <code>turtles</code> list variable.</p> <p>The <code>create_turtle()</code> function only creates one turtle, how can we create multiple turtles with it? An easy way to do this is to create another function using <code>create_turtles()</code> inside a loop to create our list of turtles. Here’s a function that does that:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">create_turtles</span><span class="p">(</span><span class="n">colors</span><span class="p">):</span> <span class="n">turtles</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">color</span> <span class="ow">in</span> <span class="n">colors</span><span class="p">:</span> <span class="n">t</span> <span class="o">=</span> <span class="n">create_turtle</span><span class="p">(</span><span class="n">color</span><span class="p">)</span> <span class="n">turtles</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">return</span> <span class="n">turtles</span> </pre></div> <p>Here we’ve created a function <code>create_turtles(colors)</code> (notice the plural on both the name of the function and the parameter, this just helps us be clear what our intent is) that creates a list of turtles. We use this function like this:</p> <div class="codehilite"><pre><span></span><span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"black"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">,</span> <span class="s2">"orange"</span><span class="p">,</span> <span class="s2">"green"</span><span class="p">]</span> <span class="n">turtles</span> <span class="o">=</span> <span class="n">create_turtles</span><span class="p">(</span><span class="n">colors</span><span class="p">)</span> </pre></div> <p>In the above code we created a variable <code>colors</code> containing a list of four valid turtle colors. We then passed the list to our <code>create_turtles()</code> function. Inside that function we create an empty turtles list with the <code>turtles = []</code> statement.</p> <p>Then we start a <code>for</code> loop taking one color at a time from the <code>colors</code> list parameter, passes that to our <code>create_turtle()</code> function, which creates a turtle that draws in that color.</p> <p>We then use the <code>turtles.append(t)</code> statement to add the turtle to our <code>turtles</code> variable. The <code>append()</code> function is part of the functionality associated with lists, and lets us add elements to the end of the list programmatically. At the end of the loop we return our <code>turtles</code> list variable so it can be used later.</p> <p>If we save and run this program it works, but doesn’t draw anything but the last green turtle on the screen. Remember turtles are all created in the center of the screen, so all four are there, just stacked on top of each other.</p> <p>Let’s put some code in our <code>move_turtle(t)</code> function to get those turtles moving.</p> <p><strong>Moving Turtles Randomly</strong></p> <p>We want our turtles to draw randomly around the screen, so inside the <strong>draw_turtle(t)</strong> function is where we’re going to use our <strong>random.randint()</strong> function we learned about earlier. We also want to <em>stamp</em> a turtle on the screen with every move, which is where we’ll use our <strong>stamp()</strong> function. Here’s a function that will turn a turtle a random angle and move it a random distance:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">move_turtle</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">stamp</span><span class="p">()</span> <span class="n">angle</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="o">-</span><span class="mi">90</span><span class="p">,</span> <span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="n">angle</span><span class="p">)</span> <span class="n">distance</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">50</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">distance</span><span class="p">)</span> </pre></div> <p>This function does a couple of things. First, it expects a turtle as a parameter variable, in the example above that parameter variable is <code>t</code>. The first thing the function does is use our turtle <code>t</code> to <code>stamp()</code> a turtle image on the screen.</p> <p>It then uses the <code>random.randint()</code> function to create an <code>angle</code> variable set to between -90 and 90 degrees. This allows our turtle to turn left or right some random amount. We pass this random <code>angle</code> variable to our <code>t.turn(angle)</code> function to turn our <code>t</code> turtle.</p> <p>We then do a similar thing to create a random <code>distanace</code> varible set to between 50 and 100. We use this variable in our <code>t.forward(distance)</code> function call to move our <code>t</code> turtle forward some random distance.</p> <p><strong>Our Program So Far</strong></p> <p>Let’s see what we’ve got for our program so far:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="kn">import</span> <span class="nn">random</span> <span class="n">screen</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Screen</span><span class="p">()</span> <span class="n">turtle</span><span class="o">.</span><span class="n">setup</span><span class="p">(</span><span class="mi">1024</span><span class="p">,</span> <span class="mi">768</span><span class="p">)</span> <span class="n">screen</span><span class="o">.</span><span class="n">bgcolor</span><span class="p">(</span><span class="s2">"#FFFFE0"</span><span class="p">)</span> <span class="c1"># The number of turtles to create and what color to create them with</span> <span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"black"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">,</span> <span class="s2">"orange"</span><span class="p">,</span> <span class="s2">"green"</span><span class="p">]</span> <span class="c1"># Create a new turtle with a certain color</span> <span class="k">def</span> <span class="nf">create_turtle</span><span class="p">(</span><span class="n">color</span><span class="p">):</span> <span class="n">t</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="n">color</span><span class="p">)</span> <span class="k">return</span> <span class="n">t</span> <span class="c1"># Create a list of turtles from a list of colors</span> <span class="k">def</span> <span class="nf">create_turtles</span><span class="p">(</span><span class="n">colors</span><span class="p">):</span> <span class="n">turtles</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">color</span> <span class="ow">in</span> <span class="n">colors</span><span class="p">:</span> <span class="n">t</span> <span class="o">=</span> <span class="n">create_turtle</span><span class="p">(</span><span class="n">color</span><span class="p">)</span> <span class="n">turtles</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">return</span> <span class="n">turtles</span> <span class="k">def</span> <span class="nf">move_turtle</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">stamp</span><span class="p">()</span> <span class="n">angle</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="o">-</span><span class="mi">90</span><span class="p">,</span> <span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="n">angle</span><span class="p">)</span> <span class="n">distance</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">50</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">distance</span><span class="p">)</span> <span class="n">turtles</span> <span class="o">=</span> <span class="n">create_turtles</span><span class="p">(</span><span class="n">colors</span><span class="p">)</span> <span class="k">for</span> <span class="n">move</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="k">for</span> <span class="n">a_turtle</span> <span class="ow">in</span> <span class="n">turtles</span><span class="p">:</span> <span class="n">move_turtle</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">)</span> </pre></div> <p>If you save and run our program it will generate a screen that looks something like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-stamping-2.png" width="1026" height="790"></figure> <p>You probably noticed that your turtles might have wandered off the screen, sometimes never to return. How can we keep our turtles on the screen so we can see what they’re drawing?</p> <p>We have them make decisions so they know how to turn around if they go off the screen. This is where we use something called <em>conditionals</em> in programming, a way of making a decision based on a condition that is happening in our program.</p> </section><section><h2><a class="anchor" name="conditionals-and-if-statements"></a> Conditionals and “if” Statements</h2> <p>As we briefly talked about in our first class, the way to make programs act smarter is to have them make decisions. To do this we use something called <em>conditionals</em>.</p> <p>Conditionals are just a way for a program to look at something (a condition) and make a decision to do something or something else. For instance, here’s some possible Python conditional program statements:</p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="n">x</span> <span class="o">&lt;</span> <span class="o">-</span><span class="mi">250</span> <span class="ow">or</span> <span class="n">x</span> <span class="o">&gt;</span> <span class="mi">250</span><span class="p">:</span> <span class="n">outside_box</span> <span class="o">=</span> <span class="bp">True</span> </pre></div> <p>Here’s what’s happening in these Python statements:</p> <ol> <li>Use the <code>if</code> statement to test whether the variable <code>x</code> is less than negative 250, or greater than positive 250</li> <li>If <code>x</code> is outside those two values, set the variable <code>outside_box</code> to <em>Boolean</em> <code>True</code></li> </ol> <p>How can we use conditionals to keep our turtles inside a viewable area? First off let’s make our viewable area a box that’s inside our screen so we can see what our turtles do when they go outside that box.</p> <p>In our program we’ll create a variable <code>box_size</code> equal to the size of the box we want to make our viewable area, let’s say 500. We’ll also use one of our turtles to draw this viewable box on the screen so we can see the box edges.</p> <p>Let’s make our program look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="kn">import</span> <span class="nn">random</span> <span class="n">screen</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Screen</span><span class="p">()</span> <span class="n">turtle</span><span class="o">.</span><span class="n">setup</span><span class="p">(</span><span class="mi">1024</span><span class="p">,</span> <span class="mi">768</span><span class="p">)</span> <span class="n">screen</span><span class="o">.</span><span class="n">bgcolor</span><span class="p">(</span><span class="s2">"#FFFFE0"</span><span class="p">)</span> <span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"black"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">,</span> <span class="s2">"orange"</span><span class="p">,</span> <span class="s2">"green"</span><span class="p">]</span> <span class="n">box_size</span> <span class="o">=</span> <span class="mi">500</span> <span class="k">def</span> <span class="nf">create_turtle</span><span class="p">(</span><span class="n">color</span><span class="p">):</span> <span class="n">t</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="n">color</span><span class="p">)</span> <span class="k">return</span> <span class="n">t</span> <span class="k">def</span> <span class="nf">create_turtles</span><span class="p">(</span><span class="n">colors</span><span class="p">):</span> <span class="n">turtles</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">color</span> <span class="ow">in</span> <span class="n">colors</span><span class="p">:</span> <span class="n">t</span> <span class="o">=</span> <span class="n">create_turtle</span><span class="p">(</span><span class="n">color</span><span class="p">)</span> <span class="n">turtles</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">return</span> <span class="n">turtles</span> <span class="k">def</span> <span class="nf">move_turtle</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">stamp</span><span class="p">()</span> <span class="n">angle</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="o">-</span><span class="mi">90</span><span class="p">,</span> <span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="n">angle</span><span class="p">)</span> <span class="n">distance</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">50</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">distance</span><span class="p">)</span> <span class="n">turtles</span> <span class="o">=</span> <span class="n">create_turtles</span><span class="p">(</span><span class="n">colors</span><span class="p">)</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">turtles</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="n">box_size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">,</span> <span class="n">box_size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="k">for</span> <span class="n">side</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">4</span><span class="p">):</span> <span class="n">t1</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">box_size</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="k">for</span> <span class="n">move</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="k">for</span> <span class="n">a_turtle</span> <span class="ow">in</span> <span class="n">turtles</span><span class="p">:</span> <span class="n">move_turtle</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">)</span> </pre></div> <p>Right under where we create our <code>colors</code> list we’ve created the <code>box_size</code> variable and set it equal to 500. Further down under where we created our <code>turtles</code> list variable, we’ve used the first turtle from the list, <code>t1 = turtles[0]</code>, to draw our viewable boundary box. After we’re done drawing the box the turtle is moved back to it’s starting position.</p> <p>So how do we use a conditional to keep our turtles inside the box we’ve just drawn? First things first, we need to know where the turtle is in order to figure out if it’s outside the boundary box. To do this we need another turtle function.</p> <p><strong>Where’s My Turtle: xcor() and ycor()</strong></p> <p>A turtle has two functions telling us where it is in relation to the home position, (0, 0). Those functions are called <code>xcor()</code> and <code>ycor()</code>, which are short for x coordinate and y coordinate. They are used like this:</p> <div class="codehilite"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="n">t</span><span class="o">.</span><span class="n">xcor</span><span class="p">()</span> <span class="n">y</span> <span class="o">=</span> <span class="n">t</span><span class="o">.</span><span class="n">ycor</span><span class="p">()</span> </pre></div> <p>As you might have guessed, the <code>t.xcor()</code> function returns the current x coordinate of the turtle <code>t</code>, and <code>t.ycor()</code> returns the current y coordinate of the turtle.</p> <p>Now we have enough information to decide if a turtle is inside or outside our boundary box. We know where the edges of the boundary box are in relation to where we started drawing it, plus and minus 250 pixels in relation to the starting position of the turtles, (0, 0). We also can figure out where our turtles are any time we want, which we can compare to the boundary box edges.</p> <p>Let’s create a function that returns <code>True</code> if the turtle is outside the box and False otherwise. The function will need the turtle to test and information about the box. That function looks like this:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">is_turtle_outside_box</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="n">size</span><span class="p">):</span> <span class="n">outside_box</span> <span class="o">=</span> <span class="bp">False</span> <span class="n">x</span> <span class="o">=</span> <span class="n">t</span><span class="o">.</span><span class="n">xcor</span><span class="p">()</span> <span class="n">y</span> <span class="o">=</span> <span class="n">t</span><span class="o">.</span><span class="n">ycor</span><span class="p">()</span> <span class="k">if</span> <span class="n">x</span> <span class="o">&lt;</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">)</span> <span class="ow">or</span> <span class="n">x</span> <span class="o">&gt;</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">):</span> <span class="n">outside_box</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">if</span> <span class="n">y</span> <span class="o">&lt;</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="o">-</span><span class="mi">2</span><span class="p">)</span> <span class="ow">or</span> <span class="n">y</span> <span class="o">&gt;</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">):</span> <span class="n">outside_box</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">return</span> <span class="n">outside_box</span> </pre></div> <p>This function expects a turtle to be passed as the first parameter and a number for the size of the boundary box as the second parameter. It then sets the return variable <code>outside_box</code> initially to False. It then creates the <code>x</code> and <code>y</code> variables, setting them to the x and y coordinates of the passed in turtle <code>t</code> respectively. Then using an <code>if</code> statement it compares the <code>x</code> and <code>y</code> variables to the <code>size</code> divided by 2.</p> <p>Why is the <code>size</code> divided by 2? Because my intention is to pass the <code>box_size</code> variable to this function, and the boundary box is centered on the screen, with half (250 pixels) on each side of that.</p> <p>Now that we have this function, how can we use it? Inside our inner most loop we move our turtle, at which point it might be outside the boundary box, so this seems like a good place to use our <code>is_turtle_outside_box()</code> function. Here’s just the looping portion of our current program showing the inclusion of the new function:</p> <div class="codehilite"><pre><span></span><span class="k">for</span> <span class="n">move</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="k">for</span> <span class="n">a_turtle</span> <span class="ow">in</span> <span class="n">turtles</span><span class="p">:</span> <span class="n">move_turtle</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">)</span> <span class="k">if</span> <span class="n">is_turtle_outside_box</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">,</span> <span class="n">box_size</span><span class="p">)</span> <span class="o">==</span> <span class="bp">True</span><span class="p">:</span> <span class="n">a_turtle</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">180</span><span class="p">)</span> <span class="n">a_turtle</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> </pre></div> <p>What we’ve done is after our <code>move_turtle()</code> function call, we added an <code>if</code> statement using our <code>is_turtle_outside_box()</code> function to figure out if our turtle <code>t</code> is outside the boundary box. If the return value of <code>is_turtle_outside_box()</code> is True, we turn our turtle <code>t</code> around 180 degrees from where it’s currently facing and move it 100 pixels back inside the boundary box. Then the loop moves onto the next turtle and the next move for all turtles.</p> <p>Here’s our completed program with comments:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="kn">import</span> <span class="nn">random</span> <span class="c1"># Change the color of the background</span> <span class="n">screen</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Screen</span><span class="p">()</span> <span class="n">screen</span><span class="o">.</span><span class="n">bgcolor</span><span class="p">(</span><span class="s2">"#FFFFE0"</span><span class="p">)</span> <span class="c1"># The number of turtles to create and what color to create them with</span> <span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"black"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">,</span> <span class="s2">"orange"</span><span class="p">,</span> <span class="s2">"green"</span><span class="p">]</span> <span class="c1"># Size of our box</span> <span class="n">box_size</span> <span class="o">=</span> <span class="mi">500</span> <span class="c1"># Create a new turtle with a certain color</span> <span class="k">def</span> <span class="nf">create_turtle</span><span class="p">(</span><span class="n">color</span><span class="p">):</span> <span class="n">t</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="n">color</span><span class="p">)</span> <span class="k">return</span> <span class="n">t</span> <span class="c1"># Create a list of turtles from a list of colors</span> <span class="k">def</span> <span class="nf">create_turtles</span><span class="p">(</span><span class="n">colors</span><span class="p">):</span> <span class="n">turtles</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">color</span> <span class="ow">in</span> <span class="n">colors</span><span class="p">:</span> <span class="n">t</span> <span class="o">=</span> <span class="n">create_turtle</span><span class="p">(</span><span class="n">color</span><span class="p">)</span> <span class="n">turtles</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">return</span> <span class="n">turtles</span> <span class="c1"># Stamp and move the turtle</span> <span class="k">def</span> <span class="nf">move_turtle</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">stamp</span><span class="p">()</span> <span class="n">angle</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="o">-</span><span class="mi">90</span><span class="p">,</span> <span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="n">angle</span><span class="p">)</span> <span class="n">distance</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">50</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">distance</span><span class="p">)</span> <span class="c1"># Is the turtle outside the box?</span> <span class="k">def</span> <span class="nf">is_turtle_outside_box</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="n">size</span><span class="p">):</span> <span class="n">outside_box</span> <span class="o">=</span> <span class="bp">False</span> <span class="n">x</span> <span class="o">=</span> <span class="n">t</span><span class="o">.</span><span class="n">xcor</span><span class="p">()</span> <span class="n">y</span> <span class="o">=</span> <span class="n">t</span><span class="o">.</span><span class="n">ycor</span><span class="p">()</span> <span class="k">if</span> <span class="n">x</span> <span class="o">&lt;</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="o">-</span><span class="mi">2</span><span class="p">)</span> <span class="ow">or</span> <span class="n">x</span> <span class="o">&gt;</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">):</span> <span class="n">outside_box</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">if</span> <span class="n">y</span> <span class="o">&lt;</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="o">-</span><span class="mi">2</span><span class="p">)</span> <span class="ow">or</span> <span class="n">y</span> <span class="o">&gt;</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">):</span> <span class="n">outside_box</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">return</span> <span class="n">outside_box</span> <span class="c1"># Create our list of turtles</span> <span class="n">turtles</span> <span class="o">=</span> <span class="n">create_turtles</span><span class="p">(</span><span class="n">colors</span><span class="p">)</span> <span class="c1"># Use the first turtle to draw our boundary box</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">turtles</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="n">box_size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">,</span> <span class="n">box_size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="k">for</span> <span class="n">side</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">4</span><span class="p">):</span> <span class="n">t1</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">box_size</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="c1"># Move all the turtles a hundred times</span> <span class="k">for</span> <span class="n">move</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="c1"># Move a particular turtle from our list of turtles</span> <span class="k">for</span> <span class="n">a_turtle</span> <span class="ow">in</span> <span class="n">turtles</span><span class="p">:</span> <span class="n">move_turtle</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">)</span> <span class="c1"># Is the turtle outside the boundary box?</span> <span class="k">if</span> <span class="n">is_turtle_outside_box</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">,</span> <span class="n">box_size</span><span class="p">)</span> <span class="o">==</span> <span class="bp">True</span><span class="p">:</span> <span class="c1"># Turn the turtle around and move it back</span> <span class="n">a_turtle</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">180</span><span class="p">)</span> <span class="n">a_turtle</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> </pre></div> <p>When we run our program the screen should look something like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-random-turtles.png" width="1280" height="1102"></figure> </section><section><h2><a class="anchor" name="conclusion"></a> Conclusion</h2> <p>You’re all getting to be real Python programmers now! You’ve created a program that draws with turtles, and makes decisions based on where those turtles are, very, very cool!</p> <p><a href="python-intro-reacting-to-user-input">In the fourth (and final) class in this series</a> you’ll learn how to make your Python programs interactive by letting them react to user input:</p> <ul> <li><a href="python-intro-reacting-to-user-input">Let’s Program with Python: Reacting to User Input (Part 4)</a></li> </ul></section><footer></footer></article>https://dbader.org/blog/python-intro-conditionals-and-if-statementsThu, 04 May 2017 00:00:00 GMTQueues in Pythonhttps://dbader.org/blog/queues-in-python<article><header><h1>Queues in Python</h1> <p>How to implement a FIFO queue data structure in Python using only built-in data types and classes from the standard library.</p> </header><section><figure><img alt="" src="/blog/figures/queues-in-python.png" width="1280" height="720"></figure> <p>A queue is a collection of objects that supports fast <em>first-in, first-out (FIFO)</em> semantics for inserts and deletes. The insert and delete operations sometimes called <em>enqueue</em> and <em>dequeue</em>. Unlike lists or arrays, queues typically don’t allow for random access to the objects they contain.</p> <p>Here’s a real-world analogy for a first-in, first-out queue:</p> <blockquote> <p>Imagine a line of Pythonistas waiting to pick up their conference badges on day one of PyCon registration. New additions to the line are made to the back of the queue as new people enter the conference venue and “queue up” to receive their badges. Removal (serving) happens in the front of the queue, as developers receive their badges and conference swag bags and leave the queue.</p> </blockquote> <p>Another way to memorize the characteristics of a queue data structure is to think of it as a <em>pipe</em>:</p> <blockquote> <p>New items (water molecules, ping-pong balls, …) are put in at one end and travel to the other where you or someone else removes them again. While the items are in the queue, a solid metal pipe, you can’t get at them. The only way to interact with the items in the queue is to add new items at the back (<em>enqueue</em>) or to remove items at the front (<em>dequeue</em>) of the pipe.</p> </blockquote> <p>Queues are similar to stacks and the difference between them is in removing items:</p> <p>With a <strong>queue</strong> you remove the item <strong>least recently added</strong> (<em>first-in, first-out</em> or <em>FIFO</em>); and with a <strong>stack</strong> you remove the item <strong>most recently added</strong> (<em>last-in, first-out</em> or <em>LIFO</em>).</p> <p>Performance-wise, a proper queue implementation is expected to take <em>O(1)</em> time for insert and delete operations. These are the two main operations performed on a queue and they should be fast in a correct implementation.</p> <p>Queues have a wide range of applications in algorithms and to solve scheduling, as well as parallel programming problems. A short and beautiful algorithm using a queue is <a href="https://en.wikipedia.org/wiki/Breadth-first_search" target="_blank">breadth-first search (BFS)</a> on a tree or graph data structure.</p> <p>Scheduling algorithms often use <a href="priority-queues-in-python">priority queues</a> internally. These are specialized queues: instead of retrieving the next element by insertion time, a priority queue retrieves the highest-priority element. The priority of individual elements is decided by the queue based on the ordering applied to their keys.</p> <p>A regular queue, however, won’t re-order the items it carries. You get what you put in, and in exactly that order (remember the pipe example?)</p> <p>Python ships with several queue implementations that each have slightly different characteristics. Let’s take a look at them:</p> </section><section><h2>⛔ The <a href="https://docs.python.org/3/tutorial/datastructures.html#using-lists-as-queues" target="_blank">list</a> Built-in</h2> <p>It’s possible to use a regular <code>list</code> as a queue but this is <strong>not ideal from a performance perspective</strong>. Lists are quite slow for this purpose because inserting or deleting an element at the beginning requires shifting all of the other elements by one, requiring <em>O(n)</em> time.</p> <p>Therefore I would <strong>not recommend</strong> you use a <code>list</code> as a makeshift queue in Python (unless you’re dealing with a small number of elements only).</p> <div class="codehilite"><pre><span></span><span class="c1"># How to use Python's list as a FIFO queue:</span> <span class="n">q</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'eat'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'sleep'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'code'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span> <span class="p">[</span><span class="s1">'eat'</span><span class="p">,</span> <span class="s1">'sleep'</span><span class="p">,</span> <span class="s1">'code'</span><span class="p">]</span> <span class="c1"># Careful: This is slow!</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="s1">'eat'</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/collections.html#collections.deque" target="_blank">collections.deque</a> Class</h2> <p>The <code>deque</code> class implements a double-ended queue that supports adding and removing elements from either end in <em>O(1)</em> time.</p> <p>Python’s <a href="https://github.com/python/cpython/blob/947629916a5ecb1f6f6792e9b9234e084c5bf274/Modules/_collectionsmodule.c#L24-L26" target="_blank">deque objects are implemented as doubly-linked lists</a> which gives them excellent performance for enqueuing and dequeuing elements, but poor <em>O(n)</em> performance for randomly accessing elements in the middle of the queue.</p> <p>Because deques support adding and removing elements from either end equally well, they can serve both as queues and as stacks.</p> <p><strong><code>collections.deque</code> is a great default choice if you’re looking for a queue data structure in Python’s standard library.</strong></p> <div class="codehilite"><pre><span></span><span class="c1"># How to use collections.deque as a FIFO queue:</span> <span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">deque</span> <span class="n">q</span> <span class="o">=</span> <span class="n">deque</span><span class="p">()</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'eat'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'sleep'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'code'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span> <span class="n">deque</span><span class="p">([</span><span class="s1">'eat'</span><span class="p">,</span> <span class="s1">'sleep'</span><span class="p">,</span> <span class="s1">'code'</span><span class="p">])</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">popleft</span><span class="p">()</span> <span class="s1">'eat'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">popleft</span><span class="p">()</span> <span class="s1">'sleep'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">popleft</span><span class="p">()</span> <span class="s1">'code'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">popleft</span><span class="p">()</span> <span class="ne">IndexError</span><span class="p">:</span> <span class="s2">"pop from an empty deque"</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/queue.html" target="_blank">queue.Queue</a> Class</h2> <p>This queue implementation in the Python standard library is synchronized and provides locking semantics to support multiple concurrent producers and consumers.</p> <p>The <a href="https://docs.python.org/3/library/queue.html" target="_blank"><code>queue</code> module</a> contains several other classes implementing multi-producer, multi-consumer queues that are useful for parallel computing.</p> <p>Depending on your use case the locking semantics might be helpful, or just incur unneeded overhead. In this case you’d be better off with using <code>collections.deque</code> as a general purpose queue.</p> <div class="codehilite"><pre><span></span><span class="c1"># How to use queue.Queue as a FIFO queue:</span> <span class="kn">from</span> <span class="nn">queue</span> <span class="kn">import</span> <span class="n">Queue</span> <span class="n">q</span> <span class="o">=</span> <span class="n">Queue</span><span class="p">()</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'eat'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'sleep'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'code'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span> <span class="o">&lt;</span><span class="n">queue</span><span class="o">.</span><span class="n">Queue</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x1070f5b38</span><span class="o">&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'eat'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'sleep'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'code'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get_nowait</span><span class="p">()</span> <span class="n">queue</span><span class="o">.</span><span class="n">Empty</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="c1"># Blocks / waits forever...</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue" target="_blank">multiprocessing.Queue</a> Class</h2> <p>This is a shared job queue implementation that allows queued items to be processed in parallel by multiple concurrent workers. Process-based parallelization is popular in Python due to the <a href="https://wiki.python.org/moin/GlobalInterpreterLock" target="_blank">global interpreter lock (GIL)</a>.</p> <p><code>multiprocessing.Queue</code> is meant for sharing data between processes and can store any pickle-able object.</p> <div class="codehilite"><pre><span></span><span class="c1"># How to use multiprocessing.Queue as a FIFO queue:</span> <span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="kn">import</span> <span class="n">Queue</span> <span class="n">q</span> <span class="o">=</span> <span class="n">Queue</span><span class="p">()</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'eat'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'sleep'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'code'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span> <span class="o">&lt;</span><span class="n">multiprocessing</span><span class="o">.</span><span class="n">queues</span><span class="o">.</span><span class="n">Queue</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x1081c12b0</span><span class="o">&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'eat'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'sleep'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'code'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="c1"># Blocks / waits forever...</span> </pre></div> </section><section><h2>A good default choice: <code>collections.deque</code></h2> <p>If you’re not looking for parallel processing support the implementation offered by <code>collections.deque</code> is an excellent default choice for implementing a FIFO queue data structure in Python.</p> <p>I’d provides the performance characteristics you’d expect from a good queue implementation and can also be used as a stack (LIFO Queue).</p> <p><em><a href="fundamental-data-structures-in-python">Read the full “Fundamental Data Structures in Python” article series here.</a> This article is missing something or you found an error? Help a brother out and leave a comment below.</em></p></section><footer></footer></article>https://dbader.org/blog/queues-in-pythonTue, 02 May 2017 00:00:00 GMTLet’s Program with Python: Functions and Lists (Part 2)https://dbader.org/blog/python-intro-functions-and-lists<article><header><h1>Let’s Program with Python: Functions and Lists (Part 2)</h1> <p>In part two of this four-part Python introduction you’ll see how to write reusable “code building blocks” in your Python programs with functions.</p> </header><section><figure><img alt="" src="/blog/figures/pywelcome-part2.png" width="1280" height="720"></figure> <p><em>In this guest post series by <a href="#author">Doug Farrell</a> you’ll learn the basics of programming with Python from scratch. If you’ve never programmed before or need a fun little class to work through with your kids, you’re welcome to follow along.</em></p> <ul> <li><a href="python-intro-statements-variables-and-loops">Part 1: Statements, Variables, and Loops</a></li> <li>Part 2: Functions and Lists (This article)</li> <li><a href="python-intro-conditionals-and-if-statements">Part 3: Conditionals and “if” Statements</a></li> <li><a href="python-intro-reacting-to-user-input">Part 4: Reacting to User Input</a></li> </ul> </section><section><h2>Table of Contents – Part 2</h2> <ul> <li><a href="#programmers-are-lazy">Programmers Are Lazy</a></li> <li><a href="#introduction-to-functions">Introduction to Functions</a></li> <li><a href="#new-turtle-drawing-functions">New Turtle Drawing Functions</a></li> <li><a href="#drawing-with-multiple-turtles">Drawing With Multiple Turtles</a></li> <li><a href="#grouping-things-with-lists">Grouping Things With Lists</a></li> <li><a href="#conclusion">Conclusion</a></li> </ul> </section><section><h2><a class="anchor" name="programmers-are-lazy"></a>Programmers Are Lazy</h2> <p>We mentioned this in the last class, but if you’re going to be a programmer, you have to embrace basic laziness. Programmers don’t like to repeat themselves and always look for ways to write less code rather than more to get the same things done.</p> <p><a href="python-intro-statements-variables-and-loops">In our last class</a> we saw how using a for loop could reduce the amount of code we had to write to draw a flower. We used a loop to repeat drawing the “petals” of our flower so we didn’t have to write code for every one.</p> <p>Let’s learn about another tool we can put in our programmers toolbelt called functions.</p> </section><section><h2><a class="anchor" name="introduction-to-functions"></a>Introduction to Functions</h2> <p>Functions allow us to use the same set of Python statements over and over again, and even change what the Python code does without having to change the code. We’ve already used functions in the previous session in our turtle program. We used the <code>range()</code> function as part of a <code>for</code> loop.</p> <p>The <code>range()</code> function is built into Python, but what does it do?</p> <p>It generates a range of numbers we can use inside a <code>for</code> loop, as simple as that. Let’s start <strong>Idle</strong>, get into interactive mode and enter this at the Python command prompt:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> </pre></div> <p>The <code>range(10)</code> function created something that will generate a count from <code>0</code> to <code>9</code> (that’s 10 numbers in total). Notice we told the <code>range()</code> function how big the range we wanted was by passing <code>10</code> as the parameter of the function.</p> <p>Using this in a <code>for</code> loop shows the values generated by <code>range(10)</code>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="mi">0</span> <span class="mi">1</span> <span class="mi">2</span> <span class="mi">3</span> <span class="mi">4</span> <span class="mi">5</span> <span class="mi">6</span> <span class="mi">7</span> <span class="mi">8</span> <span class="mi">9</span> </pre></div> <p>What we’ve done is:</p> <ul> <li>Create a for loop that’s going to assign the range of values generated one at a time to the variable <code>x</code>.</li> <li>Then inside the loop we’re just printing the latest value of <code>x</code>.</li> </ul> <p>You’ll notice that value of <code>x</code> goes from <code>0</code> to <code>9</code>, not <code>10</code> as you might expect. There are still ten values, but because Python is zero based (starts things at zero, unless told otherwise), the <code>range(10)</code> function goes from <code>0</code> → <code>9</code>.</p> <p>In our flower drawing turtle program we called <code>range()</code> like this:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">)</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">36</span><span class="p">)</span> </pre></div> <p>This generated a range of 36 values, from <code>0</code> to <code>35</code>. These two examples demonstrate we are changing what the <code>range()</code> function does based on the value we give to it.</p> <p>The value we give to the <code>range()</code> function is called a <strong>parameter</strong>, and the value of that parameter is used to change what the <code>range()</code> function does. In the examples above the parameter tells the <code>range()</code> function how many numbers to generate and gives back to our program a way to use them.</p> <p>We’ve also used functions when we were working with our turtle. For example when I changed the color of my turtle <code>t</code>, with the <code>color()</code> function, like this:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> </pre></div> <p>I was calling the <code>color()</code> function of the turtle variable <code>t</code>, and passed it two parameters, <code>"yellow"</code> and <code>"red"</code>:</p> <ul> <li>The <code>"yellow"</code> parameter changed the color of the <code>t</code> turtle and the color it draws with.</li> <li>The <code>"red"</code> parameter changed the color the turtle used when filling a shape.</li> </ul> <p><strong>Flower Drawing Using Functions</strong></p> <p>Okay, so it’s great Python provides a bunch of functions we can use to do different things, how do functions help me be lazy?</p> <p>Well, Python also lets us create our own functions and use them just like we would any built in function.</p> <p>In <strong>Idle</strong> let’s open our turtle program code from last class and try something out. Modify your program to look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="k">def</span> <span class="nf">draw_box</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="k">for</span> <span class="n">petal</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">):</span> <span class="n">draw_box</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>Save and run our program and it should create our flower exactly as it did before. You’re probably thinking “what’s the big deal, it did exactly the same thing”, and you’d be right!</p> <p>Notice I renamed our turtle variable from <code>t</code> to <code>t1</code>. Why did I do this?</p> <p>I’m getting ready to draw with two turtles at the same time (coming soon to a lesson near you!). Notice also the function I’ve defined, <code>draw_box</code>, has a <code>t</code> in between the parenthesis. Even though my turtle variable is defined as <code>t1</code>, I’m using a variable called <code>t</code> inside the <code>draw_box</code> function.</p> <p>The <code>draw_box</code> function is <strong>defined</strong> by beginning the program line with the Python keyword <code>def</code>, followed by any word we’d like, parenthesis and finally a colon character ‘<code>:</code>’.</p> <p>Just like the <code>range(36)</code> function, where I pass it a value of 36 so it generates 36 numbers, here I’m passing a parameter I’m calling <code>t</code>, and it’s using it to draw with.</p> <p>Inside my <code>for</code> loop notice I’m calling <code>draw_box</code> with my newly renamed <code>t1</code> variable. This is because the variable name passed to a function as a parameter has nothing to do with the variable name inside the function when it’s defined.</p> <p>Notice also that all the drawing code in the <code>draw_box</code> function is indented. Just like the <code>for</code> loop this indicates these Python statements are part of the function definition for <code>draw_box()</code>.</p> <p>When our program runs the <code>for</code> loop calls our <code>draw_box</code> function 36 times, and each time it turns our turtle (<code>t1</code>) 10 degrees to the right.</p> </section><section><h2><a class="anchor" name="new-turtle-drawing-functions"></a> New Turtle Drawing Functions</h2> <p>We’re getting ready to draw multiple flowers with multiple turtles. To do that and have them look good on the screen we’ll learn some more turtle drawing functions.</p> <p><strong>Turtle Pen Up: penup()</strong></p> <p>We can move our turtle without drawing a line by lifting our pen up. In this way we can move the turtle and no line will be drawn. To do this we use the turtle <code>penup()</code> function. It looks like this:</p> <div class="codehilite"><pre><span></span><span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> </pre></div> <p><strong>Turtle Pen Down: pendown()</strong></p> <p>Once we’ve moved our turtle where we want it to be without drawing a line, we need to put the pen down again, and the turtle system provides this. We use the <code>pendown()</code> function. It looks like this:</p> <div class="codehilite"><pre><span></span><span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> </pre></div> <p><strong>Turtle Goto: goto()</strong></p> <p>We can move our turtle to a specific position on the screen using the <code>goto()</code> funciton. We pass x and y coordinates to the <code>goto()</code> function to position our turtle. One thing to be aware of is the 0, 0 coordinates are where our turtle is created (center of the screen) when we did this <code>t1 = turtle.Turtle()</code>.</p> <p>So the coordinates we pass to <code>goto()</code> are relative to that starting position. The <code>goto()</code> function looks like this to move our turtle up and to the right:</p> <div class="codehilite"><pre><span></span><span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="mi">150</span><span class="p">,</span> <span class="mi">150</span><span class="p">)</span> </pre></div> <p>Let’s update our program and move our <code>t1</code> turtle up and to the right a bit just to see how these new drawing functions work. Make your flower program look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="mi">150</span><span class="p">,</span> <span class="mi">150</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="k">def</span> <span class="nf">draw_box</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="k">for</span> <span class="n">petal</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">):</span> <span class="n">draw_box</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>Save and run your program and you should see your flower, but its offset up and to the right side of the screen by 150 pixels. Those are the offsets we passed as the first and second parameter to the <code>t1.goto(150, 150)</code> function call.</p> </section><section><h2><a class="anchor" name="drawing-with-multiple-turtles"></a>Drawing With Multiple Turtles</h2> <p>We want to draw with multiple turtles, and our goal for this class is to create this image:</p> <figure><img alt="image" src="/blog/figures/pywelcome-three-turtles.png" width="988" height="952"></figure> <p>So far our flower drawing program is working pretty well, but can we change it even more to draw two, or perhaps more, flowers at once?</p> <p>Sure we can, we’re programmers! In order to use two turtles we’ll have to create a second turtle. I’m going to call the second turtle <code>t2</code> just to stay consistent. Add this to your program right below where we created our first turtle <code>t1</code>:</p> <div class="codehilite"><pre><span></span><span class="n">t2</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"blue"</span><span class="p">,</span> <span class="s2">"orange"</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> </pre></div> <p>This creates a second turtle with a different variable name, drawing color and fill color. When we create a turtle it’s starting position is right in the center of the screen, so our second turtle starts out right in the middle of the screen.</p> <p>Let’s move it left and down so <code>t1</code> and <code>t2</code> don’t draw on top of each other. Add these lines for turtle <code>t2</code> under the same lines for <code>t1</code>:</p> <div class="codehilite"><pre><span></span><span class="n">t2</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">penup</span><span class="p">(</span><span class="o">-</span><span class="mi">150</span><span class="p">,</span> <span class="o">-</span><span class="mi">150</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> </pre></div> <p><strong>Houston We Have a Problem</strong></p> <p>At this point our program should look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> <span class="n">t2</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"blue"</span><span class="p">,</span> <span class="s2">"orange"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="mi">150</span><span class="p">,</span> <span class="mi">150</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="o">-</span><span class="mi">150</span><span class="p">,</span> <span class="o">-</span><span class="mi">150</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="k">def</span> <span class="nf">draw_box</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="k">for</span> <span class="n">petal</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">):</span> <span class="n">draw_box</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>If save our program and run it our turtle screen looks like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-two-turtles-1.png" width="988" height="952"></figure> <p><strong>Where’s The Second Flower?</strong></p> <p>When you get your program running you’ll notice the second turtle didn’t draw a flower. Why not? Well, we didn’t tell it to draw anything, so it just waited around while the first turtle drew a flower.</p> <p>How do we get it to draw it’s own flower? We add it to the <code>for</code> loop. Our updated program now looks like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> <span class="n">t2</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"blue"</span><span class="p">,</span> <span class="s2">"orange"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="mi">150</span><span class="p">,</span> <span class="mi">150</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="o">-</span><span class="mi">150</span><span class="p">,</span> <span class="o">-</span><span class="mi">150</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="k">def</span> <span class="nf">draw_box</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="k">for</span> <span class="n">petal</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">):</span> <span class="n">draw_box</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="n">draw_box</span><span class="p">(</span><span class="n">t2</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>Just by adding two lines we got our second turtle <code>t2</code> to draw its own complete flower. This is a definite win for laziness. All we had to do was add a couple Python statements to draw a complete second flower!</p> <p>By setting things up and using a function we are able to build more and more interesting programs. In fact we could keep going and add more and more turtles to fill the screen with flowers and all we’d have to do is create more turtles and add statements to our <code>for</code> loop.</p> <p>But this is starting to look like when we were adding flower petals to start with. Can we be even lazier and organize things differently to handle multiple turtles better? Yes of course, we can use something Python calls <em>lists</em>.</p> </section><section><h2><a class="anchor" name="grouping-things-with-lists"></a> Grouping Things With Lists</h2> <p>Lists are a way of grouping things together so we can work with them all at once. They’re a handy way of keeping things grouped together and giving that group a name. There’s nothing magical about this, we can create lists easily with Python. If we enter these statements in the interactive window:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">my_list</span> <span class="o">=</span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">0</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="n">my_list</span><span class="p">)</span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">0</span><span class="p">]</span> </pre></div> <p>We created a variable we called <code>my_list</code> containing the list <code>[4, 2, 3, 0]</code>.</p> <p>You can see the things in the list don’t have to be in order. Lists are created by surrounding a set of things separated by commas with the <code>[</code> and <code>]</code> characters at either end.</p> <p>We can use a list to organize our turtles. Let’s create a list of turtles like this in our program:</p> <div class="codehilite"><pre><span></span><span class="n">turtles</span> <span class="o">=</span> <span class="p">[</span><span class="n">t1</span><span class="p">,</span> <span class="n">t2</span><span class="p">]</span> </pre></div> <p>This creates a variable called <code>turtles</code> that is a list containing our two turtles. Now we can create a new <code>for</code> loop that gets a turtle from our <code>turtles</code> list one at a time and draws with it. We do this with these Python statements:</p> <div class="codehilite"><pre><span></span><span class="k">for</span> <span class="n">a_turtle</span> <span class="ow">in</span> <span class="n">turtles</span><span class="p">:</span> <span class="n">draw_box</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">)</span> <span class="n">a_turtle</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>We’re using a <code>for</code> loop to get each turtle one at a time from our <code>turtles</code> list, assigning it to the variable <code>a_turtle</code> and calling <code>draw_box(a_turtle)</code> and <code>a_turtle.right(10)</code> with that variable.</p> <p>If we put this inside our main <code>for</code> loop, it will be called for each petal the main <code>for</code> loop wants to draw.</p> <p>We can now add a third turtle easily by creating a new turtle and adding it to the <code>turtles</code> list.</p> <p>Let’s do that in our updated, three turtle program. I’ve added comments to describe what’s going on:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="c1"># Create our t1 turtle</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> <span class="c1"># Create our t2 turtle</span> <span class="n">t2</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"blue"</span><span class="p">,</span> <span class="s2">"orange"</span><span class="p">)</span> <span class="c1"># Create our t3 turtle</span> <span class="n">t3</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t3</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t3</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t3</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t3</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"red"</span><span class="p">,</span> <span class="s2">"blue"</span><span class="p">)</span> <span class="c1"># Move t1 to its starting position</span> <span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="mi">150</span><span class="p">,</span> <span class="mi">150</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="c1"># Move t2 to its starting position</span> <span class="n">t2</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="o">-</span><span class="mi">150</span><span class="p">,</span> <span class="o">-</span><span class="mi">150</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="c1"># Move t3 to its starting position</span> <span class="n">t3</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t3</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="o">-</span><span class="mi">150</span><span class="p">,</span> <span class="mi">150</span><span class="p">)</span> <span class="n">t3</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="c1"># Define our draw_box function</span> <span class="k">def</span> <span class="nf">draw_box</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="c1"># Create our list of turtles</span> <span class="n">turtles</span> <span class="o">=</span> <span class="p">[</span><span class="n">t1</span><span class="p">,</span> <span class="n">t2</span><span class="p">,</span> <span class="n">t3</span><span class="p">]</span> <span class="c1"># Create our for loop for 36 petals of the flower</span> <span class="k">for</span> <span class="n">petal</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">):</span> <span class="c1"># Create our for loop to draw a flower petal with</span> <span class="c1"># each turtle in the turtles list</span> <span class="k">for</span> <span class="n">a_turtle</span> <span class="ow">in</span> <span class="n">turtles</span><span class="p">:</span> <span class="c1"># Draw and rotate each turtle</span> <span class="n">draw_box</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">)</span> <span class="n">a_turtle</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>I created a third turtle called <code>t3</code> and just added <code>t3</code> to the <code>turtles</code> list. Notice that our main <code>for</code> loop didn’t change, as far as it’s concerned, it’s just looping 36 times.</p> <p>The inner <code>for</code> loop is responsible for calling the <code>draw_box()</code> function with each turtle variable, and then turning that turtle right 10 degrees. Here’s what the output of the program looks like:</p> <figure><img alt="image" src="/blog/figures/pywelcome-three-turtles.png" width="988" height="952"></figure> </section><section><h2><a class="anchor" name="conclusion"></a> Conclusion</h2> <p>Congratulations, you’re a multi-turtle genius now! You saw how to use Python lists to help us get multiple turtles drawing on the screen. We could keep adding turtles to our hearts content and the program would faithfully make each turtle draw our flower. This worked very well drawing the well controlled structure of the flower.</p> <p>But what if we want to draw something that’s randomly generated, something where the turtles draw something and we don’t know ahead of time what that will be? How can we use what we know already to help us do that?</p> <p><a href="python-intro-conditionals-and-if-statements">In the next class in this series</a> you’ll teach our program how to make decisions and do things on its own:</p> <ul> <li><a href="python-intro-conditionals-and-if-statements">Let’s Program with Python: Conditionals and “if” Statements (Part 3)</a></li> </ul></section><footer></footer></article>https://dbader.org/blog/python-intro-functions-and-listsThu, 27 Apr 2017 00:00:00 GMTSets and Multisets in Pythonhttps://dbader.org/blog/sets-and-multiset-in-python<article><header><h1>Sets and Multisets in Python</h1> <p>How to implement mutable and immutable set and multiset (bag) data structures in Python using built-in data types and classes from the standard library.</p> </header><section><figure><img alt="" src="/blog/figures/python-sets-and-multisets.png" width="1280" height="720"></figure> <p>A <em>set</em> is an unordered collection of objects that does not allow duplicate elements. Typically sets are used to quickly test a value for membership in the set, to insert or delete new values from a set, and to compute the union or intersection of two sets.</p> <p>In a “proper” set implementation, membership tests are expected to run in <em>O(1)</em> time. Union, intersection, difference, and subset operations should take <em>O(n)</em> time on average. The set implementations included in Python’s standard library <a href="https://wiki.python.org/moin/TimeComplexity" target="_blank">follow these performance characteristics</a>.</p> <p>Just like dictionaries, sets get special treatment in Python and have some syntactic sugar that makes it easier to create sets. For example, the curly-braces set expression syntax and set comprehensions allow you to conveniently define new set instances:</p> <div class="codehilite"><pre><span></span><span class="n">vowels</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'a'</span><span class="p">,</span> <span class="s1">'e'</span><span class="p">,</span> <span class="s1">'i'</span><span class="p">,</span> <span class="s1">'o'</span><span class="p">,</span> <span class="s1">'u'</span><span class="p">}</span> <span class="n">squares</span> <span class="o">=</span> <span class="p">{</span><span class="n">x</span> <span class="o">*</span> <span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)}</span> </pre></div> <p>Careful: To create <strong>an empty set</strong> you’ll need to call the <code>set()</code> constructor, as using empty curly-braces (<code>{}</code>) is ambiguous and will create a dictionary instead.</p> <p>Python and its standard library provide the following set implementations:</p> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/stdtypes.html#set-types-set-frozenset" target="_blank">set</a> Built-in</h2> <p>The built-in set implementation in Python. The <code>set</code> type in Python is mutable and allows the dynamic insertion and deletion of elements. Python’s sets are backed by the <code>dict</code> data type and share the same performance characteristics. Any <a href="https://docs.python.org/3/glossary.html#term-hashable" target="_blank">hashable</a> object can be stored in a set.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">vowels</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'a'</span><span class="p">,</span> <span class="s1">'e'</span><span class="p">,</span> <span class="s1">'i'</span><span class="p">,</span> <span class="s1">'o'</span><span class="p">,</span> <span class="s1">'u'</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="s1">'e'</span> <span class="ow">in</span> <span class="n">vowels</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">letters</span> <span class="o">=</span> <span class="nb">set</span><span class="p">(</span><span class="s1">'alice'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">letters</span><span class="o">.</span><span class="n">intersection</span><span class="p">(</span><span class="n">vowels</span><span class="p">)</span> <span class="p">{</span><span class="s1">'a'</span><span class="p">,</span> <span class="s1">'e'</span><span class="p">,</span> <span class="s1">'i'</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">vowels</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s1">'x'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">vowels</span> <span class="p">{</span><span class="s1">'i'</span><span class="p">,</span> <span class="s1">'a'</span><span class="p">,</span> <span class="s1">'u'</span><span class="p">,</span> <span class="s1">'o'</span><span class="p">,</span> <span class="s1">'x'</span><span class="p">,</span> <span class="s1">'e'</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">vowels</span><span class="p">)</span> <span class="mi">6</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/stdtypes.html#set-types-set-frozenset" target="_blank">frozenset</a> Built-in</h2> <p>An <em>immutable</em> version of <code>set</code> that cannot be changed after it was constructed. Frozensets are static and only allow query operations on their elements (no inserts or deletions.) Because frozensets are static and <a href="https://docs.python.org/3/glossary.html#term-hashable" target="_blank">hashable</a> they can be used as dictionary keys or as elements of another set.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">vowels</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">({</span><span class="s1">'a'</span><span class="p">,</span> <span class="s1">'e'</span><span class="p">,</span> <span class="s1">'i'</span><span class="p">,</span> <span class="s1">'o'</span><span class="p">,</span> <span class="s1">'u'</span><span class="p">})</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">vowels</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s1">'p'</span><span class="p">)</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="s2">"'frozenset' object has no attribute 'add'"</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/collections.html#collections.Counter" target="_blank">collections.Counter</a> Class</h2> <p>The <code>collections.Counter</code> class in the Python standard library implements a <a href="https://en.wikipedia.org/wiki/Multiset" target="_blank">multiset (or bag)</a> type that allows elements in the set to have more than one occurrence.</p> <p>This is useful if you need to keep track not only <em>if</em> an element is part of a set but also <em>how many times</em> it is included in the set.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">Counter</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">inventory</span> <span class="o">=</span> <span class="n">Counter</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">loot</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'sword'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">'bread'</span><span class="p">:</span> <span class="mi">3</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">inventory</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">loot</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">inventory</span> <span class="n">Counter</span><span class="p">({</span><span class="s1">'bread'</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span> <span class="s1">'sword'</span><span class="p">:</span> <span class="mi">1</span><span class="p">})</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">more_loot</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'sword'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">'apple'</span><span class="p">:</span> <span class="mi">1</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">inventory</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">more_loot</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">inventory</span> <span class="n">Counter</span><span class="p">({</span><span class="s1">'bread'</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span> <span class="s1">'sword'</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span> <span class="s1">'apple'</span><span class="p">:</span> <span class="mi">1</span><span class="p">})</span> </pre></div> <p>Careful with counting the number of elements in a <code>Counter</code> object. Calling <code>len()</code> returns the number of <em>unique</em> elements in the multiset, whereas the total number of elements must be retrieved slightly differently:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">inventory</span><span class="p">)</span> <span class="mi">3</span> <span class="c1"># Unique elements</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">sum</span><span class="p">(</span><span class="n">inventory</span><span class="o">.</span><span class="n">values</span><span class="p">())</span> <span class="mi">6</span> <span class="c1"># Total no. of elements</span> </pre></div> </section><section><h2>📺🐍 Learn More With This Video Tutorial</h2> <p>I recorded a step-by-step video tutorial to go along with the article. See how sets work in general and how to use them in Python. Watch the video embedded below or <a href="https://www.youtube.com/watch?v=b-K1ujf8u_k" target="_blank">on my YouTube channel</a>:</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/b-K1ujf8u_k?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p><em><a href="fundamental-data-structures-in-python">Read the full “Fundamental Data Structures in Python” article series here.</a> This article is missing something or you found an error? Help a brother out and leave a comment below.</em></p></section><footer></footer></article>https://dbader.org/blog/sets-and-multiset-in-pythonTue, 25 Apr 2017 00:00:00 GMTLet’s Program with Python: Statements, Variables, and Loops (Part 1)https://dbader.org/blog/python-intro-statements-variables-and-loops<article><header><h1>Let’s Program with Python: Statements, Variables, and Loops (Part 1)</h1> <p>In this four-part introduction for new programmers you’ll learn the basics of programming with Python using step-by-step descriptions and graphical examples.</p> </header><section><figure><img alt="" src="/blog/figures/pywelcome-part1.png" width="1280" height="720"></figure> <p><em>In this guest post series by <a href="#author">Doug Farrell</a> you’ll learn the basics of programming with Python from scratch. If you’ve never programmed before or need a fun little class to work through with your kids, you’re welcome to follow along.</em></p> <p><strong>Looking for the rest of the “Let’s Program with Python” series?</strong> Here you go:</p> <ul> <li>Part 1: Statements, Variables, and Loops (This article)</li> <li><a href="python-intro-functions-and-lists">Part 2: Functions and Lists</a></li> <li><a href="python-intro-conditionals-and-if-statements">Part 3: Conditionals and “if” Statements</a></li> <li><a href="python-intro-reacting-to-user-input">Part 4: Reacting to User Input</a></li> </ul> </section><section><h2>Table of Contents – Part 1</h2> <ul> <li><a href="#what-is-python">What is Python?</a></li> <li><a href="#natural-vs-formal-language">Natural Language vs. Formal Language</a></li> <li><a href="#elements-of-programming">Elements of Programming</a></li> <li><a href="#lets-write-some-python">Enough of That, Let’s Write Some Python!</a></li> <li><a href="#statements-in-python">Statements in Python</a></li> <li><a href="#creating-python-program-files">Creating Python Program Files</a></li> <li><a href="#saving-and-running-a-python-program">Saving and Running a Python Program</a></li> <li><a href="#variables-in-python">Variables in Python</a></li> <li><a href="#lets-get-back-to-drawing">Let’s Get Back to Drawing!</a></li> <li><a href="#loops-in-python">Loops in Python</a></li> <li><a href="#conclusion">Conclusion</a></li> </ul> </section><section><h2><a class="anchor" name="what-is-python"></a>What is Python?</h2> <p>Since you’re reading this I’m hoping you’re interested in learning how to program in <a href="https://www.python.org/" target="_blank">Python</a>.</p> <p>Python is a programming language, which means it’a a language both people and computers can understand. A computer language is a formal subset of an natural language, like English. A computer language lets people express what they want a computer to do, and tells a computer how to do it.</p> <p>A computer program is a set of instructions written in a particular computer language. There are lots of different computer languages in the world, most were created to solve certain kinds of problems in different ways, and most over lap in the kinds of things they can do.</p> <p>Python was developed by a Dutch software engineer named <a href="http://en.wikipedia.org/wiki/Guido_van_Rossum" target="_blank">Guido van Rossum</a>, who created the language to solve some problems he saw in computer languages of the time.</p> <p>Python draws from a lot of good ideas in other languages and pulls them together in one place. Python is a pretty easy computer language to learn, and yet is very powerful. The name Python comes from Guido’s favorite comedy group, <a href="http://www.montypython.com/" target="_blank">Monty Python’s Flying Circus</a>.</p> <p>This course uses Python 3.6.1, but the examples should work with any version of Python 3 and greater.</p> </section><section><h2><a class="anchor" name="natural-vs-formal-language"></a>Natural Language vs. Formal Language</h2> <p>English is a natural language that’s evolved over time to help us talk with each other. It has a big vocabulary, lots of multiple meanings and depends a lot on how it’s used to make the meaning clear.</p> <p>Natural languages work well for people because we fill in the gaps where needed. This kind of language fails completely for computers because they need exact instructions in order to run. Formal languages (all programming languages) have limited vocabularies and almost no multiple meanings.</p> <p>Let’s take an English example that’s something like a “program” for a person, how to make scrambled eggs:</p> <div class="codehilite"><pre><span></span>1. Place a frying pan on the stove burner 2. Turn the burner to medium 3. Melt butter in the pan 4. Crack two eggs into pan 5. Stir the eggs to cook and scramble them 6. When finished, serve the eggs on a plate </pre></div> <p>If the steps above are followed in order, someone should be able to make scrambled eggs. This simple set of steps describe how to perform a task. A computer program is very much the same, a set of steps telling a computer how to perform a task.</p> </section><section><h2><a class="anchor" name="elements-of-programming"></a>Elements of Programming</h2> <p>As you learn to program you’ll find you need to do certain things to make the program do what you want: how to make the computer do something, remember things, doing things over and over and make decisions. Almost all programming languages provide ways to do these four basic things, and they’re known as:</p> <ul> <li><strong>Statements</strong>: the things a program can do, like performing calculations, drawing on the screen, etc.</li> <li><strong>Variables</strong>: these are the “things” (information) you want your program to work on and remember</li> <li><strong>Loops</strong>: doing things over and over again very quickly</li> <li><strong>Conditionals</strong>: these are choices a program can make about what to do, this is what makes programs “appear” smart.</li> </ul> <p>We’ll make use of these four things as we go along.</p> </section><section><h2><a class="anchor" name="lets-write-some-python"></a>Enough of That, Let’s Write Some Python!</h2> <p>Our goal is to create a Python program that will draw an image on our computer screen. The image we’re going to create looks something like a flower, and we’re going to learn how to use Python to create it. The end results will look like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-flower.png" width="1074" height="1036"></figure> <p>So how do we create a Python program? There are two ways to work with Python; working with it directly, and creating Python program files.</p> <p>This is where we can use the tool called <strong>Idle</strong>. <strong>Idle</strong> is a program that lets you both work with Python directly and create Python program files.</p> <p>So let’s start <strong>Idle</strong>. When you installed Python you should also have gotten the <strong>Idle</strong> program installed, if so, let’s start it up!</p> <p>Starting <strong>Idle</strong> should give you a window that looks something like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-idle-window.png" width="730" height="459"></figure> <p>This window provides a Python command prompt (hit return a couple of times if you don’t see it) that allows you to run Python statements line by line.</p> <p>This is called interactive mode as it allows us to ‘interact’ with Python. The command prompt in interactive mode looks like this:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> </pre></div> <p>Its at this prompt where you enter Python statements to try things out.</p> </section><section><h2><a class="anchor" name="statements-in-python"></a> Statements in Python</h2> <p>Statements are the program commands that make the computer do something in a Python program. Statements can be as simple or as complicated as we’d like to make them.</p> <p>Here are some examples:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="s2">"Hello there"</span><span class="p">)</span> <span class="n">Hello</span> <span class="n">there</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="mi">12</span><span class="p">)</span> <span class="mi">12</span> <span class="o">&gt;&gt;&gt;</span> <span class="mi">12</span> <span class="o">*</span> <span class="mi">3</span> <span class="mi">36</span> <span class="o">&gt;&gt;&gt;</span> <span class="mi">12</span> <span class="o">/</span> <span class="mi">3</span> <span class="mf">4.0</span> </pre></div> <p>The above statements print out a welcome string, perform some basic math and Python is responding. What we’ve done above is enter some Python statements at our command prompt, and Python ran them.</p> </section><section><h2><a class="anchor" name="creating-python-program-files"></a> Creating Python Program Files</h2> <p>Working with interactive mode, is great for trying things out with Python. However, we want to create a Python program we can run over and over again without having to re-type it every time.</p> <p>This is where creating a Python program and saving it as a file is very handy. Python program files are just like any other text file, but usually have the extension “.py”.</p> <p>We can create a Python program file in <strong>Idle</strong> by clicking the <strong>File → New Window</strong> menu item. This opens up a new, empty window that is a simple text editor.</p> <p>You’ll notice there is no <code>&gt;&gt;&gt;</code> Python command prompt in the window. This is because in the file window we’re not interacting with Python directly, we’re creating a Python program file.</p> <p>Let’s use our new Python program file to create our first Python program.</p> <p><strong>The “Turtle” Graphics Module</strong></p> <p>Python comes with a large library of modules that let us do some interesting things, and one of those modules is called <code>turtle</code>.</p> <p>The turtle module is a nice tool for drawing graphics on the screen. You can find the turtle graphics documentation <a href="https://docs.python.org/3/library/turtle.html#module-turtle" target="_blank">here</a>.</p> <p>The turtle module is based on the idea of a “turtle” on the screen that draws a line as it moves around, as if it had a marker taped to it’s shell.</p> <p>In order to use the turtle module we have to “import” it into our Python program. Importing a module adds the features and capabilities of that module to our Python program.</p> <p>To import the turtle module add this line to our Python program:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> </pre></div> <p><strong>Drawing With Turtle</strong></p> <p>Once the turtle module is available to us we can use it to draw things with a turtle. Enter the following lines into our program:</p> <div class="codehilite"><pre><span></span><span class="n">t</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> </pre></div> </section><section><h2><a class="anchor" name="saving-and-running-a-python-program"></a> Saving and Running a Python Program</h2> <p>Once you’ve got this entered, let’s run the program. To do that we have to save the file first, which we can do from the <strong>File → Save</strong> menu selection.</p> <p>Give our program a name and save it to a directory on the hard disk where you can find it again.</p> <p>To run the program select <strong>Run → Run Module</strong>. If your program runs without any errors (which usually means you have a typo in your program), a window will open up with a turtle shape at the end of a short line.</p> <p>That window should look something like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-turtle-1.png" width="988" height="952"></figure> <p>This is what our program told Python to do, use the <code>turtle</code> module to create a turtle we’re calling <code>t</code>, change it’s shape to look like a ‘turtle’ and move it forward 100 pixels.</p> <p>Our turtle, <code>t</code>, is the first <em>variable</em> we’ve created with Python in our program.</p> </section><section><h2><a class="anchor" name="variables-in-python"></a> Variables in Python</h2> <p>In Python things like our turtle <code>t</code> are represented by variables. Variables let us give a name to something so you and the program can remember it and use it later.</p> <p>For instance, here’s a variable assignment:</p> <div class="codehilite"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="mi">10</span> </pre></div> <p>This looks a lot like math, and that’s actually where the idea of assigning variables came from.</p> <p>This simple Python statement assigns the number 10 to a variable called <code>x</code>. The equal sign (<code>=</code>) in the line above creates the variable <code>x</code> and assigns it a value.</p> <p>In our program we’ve done this by using the turtle module to create a turtle (the Python statement <code>turtle.Turtle()</code>) and assigned the results, a turtle object, to a variable we called <code>t</code>.</p> </section><section><h2><a class="anchor" name="lets-get-back-to-drawing"></a> Let’s Get Back to Drawing!</h2> <p>Let’s add some more statements to our program to make it draw some more. Let’s make our Python program look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="n">t</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"red"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> </pre></div> <p>When you save and run our program the screen your turtle is drawing on should look like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-turtle-2.png" width="1074" height="1036"></figure> <p>So what’s going on here? What we’ve done is given our turtle a set of commands (Python program statements), and it has run them. Here’s what the statements we entered were doing:</p> <ul> <li><strong>Line 1</strong>: import the turtle module so our program can use it</li> <li><strong>Line 3</strong>: use the turtle module to create our turtle, <code>t</code></li> <li><strong>Line 4</strong>: change the shape of our turtle to look like a turtle</li> <li><strong>Line 5</strong>: from where the turtle is, move forward 100 pixels</li> <li><strong>Line 6</strong>: from where the turtle is, turn right 90 degrees, a right angle</li> <li><strong>Line 7</strong>: from where the turtle is, move forward 100 pixels</li> <li><strong>Line 8</strong>: from where the turtle is, turn right 90 degrees, a right angle</li> <li><strong>Line 9</strong>: from where the turtle is, move forward 100 pixels</li> <li><strong>Line 10</strong>: from where the turtle is, turn right 90 degrees, a right angle</li> <li><strong>Line 11</strong>: change the color used by the turtle to red</li> <li><strong>Line 12</strong>: from where the turtle is, move forward 100 pixels</li> <li><strong>Line 13</strong>: from where the turtle is, turn right 90 degrees, a right angle. This brings our turtle back to its original starting position.</li> </ul> <p>These statements made the turtle draw a box with the last side of the box drawn in red. You can see something interesting about drawing with our turtle; what it draws is based on where it is on the screen and which way it’s headed.</p> <p>Let’s learn some more Python statements to draw with our turtles.</p> <p><strong>Turtle Speed</strong>: To make our turtle draw faster we use the turtle <code>speed()</code> method. To use this we’ll add this statement to our program:</p> <div class="codehilite"><pre><span></span><span class="n">t</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> </pre></div> <p>The number <code>0</code> between the parenthesises is called a parameter, which is being given to the turtle’s <code>speed()</code> method, making our turtle draw as fast as it can.</p> <p><strong>Turtle Line Width</strong>: We can make our turtle draw with a thicker line, making it easier to see on screen. We do this with the turtle <code>width()</code> method. We can pass a parameter to the width method, expressing a value in pixels. So for example adding this line to our program makes our turtle draw with a line 3 pixels wide:</p> <div class="codehilite"><pre><span></span><span class="n">t</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> </pre></div> <p><strong>Filling-in Shapes</strong>: We can also fill a shape (like our box) with color using two other turtle methods, <code>begin_fill()</code> and <code>end_fill()</code>, and by modifying our <code>t.color()</code> method. If we use these Python statements:</p> <div class="codehilite"><pre><span></span><span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="c1"># Draw shape</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> </pre></div> <p>We have told our turtle to draw with “yellow” and fill in any shapes with “red”.</p> <p>Then we use <code>begin_fill()</code> at the start of drawing a closed shape, draw our shape and then used <code>end_fill()</code> to fill that shape with “red”.</p> <p>The following line is a Python <em>comment</em>:</p> <div class="codehilite"><pre><span></span><span class="c1"># Draw shape</span> </pre></div> <p>I’m using the comment here to indicate where our shape drawing code should go. Comments in Python are used to tell us, the programmers, what’s going on, but are ignored by Python.</p> <p><strong>Putting It All Together</strong></p> <p>Taking the things we’ve learned from our box drawing, let’s draw something a little more complicated by drawing a sort of flower.</p> <p>We’ll do this by doing two things; draw multiple boxes, filling the box with color and turning the turtle slightly between each one using the new turtle methods we just learned about.</p> <p>To draw our flower we’re going to draw multiple boxes, turning each box slightly every time. One way to do that would be to just repeat our box code over and over like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="n">t</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="c1"># Draw our first filled in box</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="c1"># Draw our second filled in box</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="c1"># Keep going till you've drawn your flower</span> </pre></div> <p>This would work fine, but we would have to repeat these statements for as many petals as we want to give our flower.</p> <p>One thing you should know about being a programmer is we’re very lazy and don’t like to repeat ourselves if we don’t have to.</p> <p>Can Python help us not repeat ourselves? Yes, it can by letting us use a loop to repeat drawing the box multiple times.</p> </section><section><h2><a class="anchor" name="loops-in-python"></a> Loops in Python</h2> <p>One of the elements of programming mentioned earlier is being able to create loops. Loops are statements in a programming language that allows us to repeat a set of program statements over and over in a controlled way. In our program we’d like to repeat the statements:</p> <div class="codehilite"><pre><span></span><span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>This set of statements creates our outlined and filled in box. We’d like to repeat these statements, with a slight turn of 10 degrees each time, in order to create a flower. Creating a loop lets us do this. One of the looping statements in Python is called a “for loop”, and it’s used to create a loop that repeats a fixed number of times.</p> <p>Let’s do a little math to figure out how many times we need to repeat this if we want to make a full circle when our turtle is turning right by 10 degrees after every filled in box.</p> <p>In case you didn’t know already, there are 360 degrees in a full circle. That means dividing a circle of 360 degrees by 10 degrees gives us a value of 36. This means we want to repeat our box 36 times in order to create our flower.</p> <p>How can we do this with Python?</p> <p><strong>The “for” loop</strong>: We want to repeat our filled in box 36 times, so we know beforehand how many times we want to loop. This is where the Python <code>for</code> loop comes in handy.</p> <p>It is made for repeating things a known number of times. It looks like this as a Python definition statement:</p> <div class="codehilite"><pre><span></span><span class="k">for</span> &lt;thing&gt; in &lt;list of things&gt;: <span class="c1"># Everything we want to do on the loop</span> </pre></div> <p>What does that mean? That’s a kind of formal presentation of what a <code>for</code> loop should look like in Python. What it means is this:</p> <ul> <li>Take one <em>&lt;thing&gt;</em> from the <em>&lt;list of things&gt;</em></li> <li>End the for statement with a <code>:</code> character</li> <li>All the statements indented under the <code>for</code> loop should be run every time through the loop. Indentation is very important to Python</li> <li>Go back to the start of the <code>for</code> loop, get another <em>&lt;thing&gt;</em> from <em>&lt;list_of_things&gt;</em></li> <li>Keep doing this till there are no more things in &lt;list_of_things&gt;</li> </ul> <p>This sounds a lot harder than it actually is. For our program we’ll use a <code>for</code> loop that looks like this:</p> <div class="codehilite"><pre><span></span><span class="k">for</span> <span class="n">petal</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>Our box code is in there, under the <code>for</code> loop, but what’s that strange looking <code>range(36)</code> thing at a the end of our <code>for</code> loop where the &lt;list of things&gt; should go?</p> <p>The <code>range(36)</code> thing is what provides the <em>&lt;list_of_things&gt;</em> from our formal definition of a <code>for</code> loop. Let’s talk about that.</p> <p><strong>The “range” function</strong>: Let’s jump back over to our <strong>Idle</strong> interactive window for a minute and enter this statement:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">)</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">36</span><span class="p">)</span> </pre></div> <p>Python responds by running this code and prints out what looks like it just told us about itself. What does this mean?</p> <p>To Python the <code>range(36)</code> function is going to provide 36 things when used inside a for loop. Each time through the <code>for</code> loop Python will take a value from the range defined (0 to 35) and assign it to a variable, in our case that variable is called <code>petal</code>.</p> <p>It will continue this until there are no values left in the range. The way we’ve set things up it will loop 36 times, which is what we want. In this case we’re not using the <code>petal</code> variable, but it’s required by Python to create a correct <code>for</code> loop.</p> <p>To create our flower using a <code>for</code> loop make our program look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="n">t</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="k">for</span> <span class="n">petal</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>Let’s go through this line by line.</p> <ul> <li><strong>Line 1</strong>: import our turtle module</li> <li><strong>Line 3</strong>: create our turtle object and use our variable <code>t</code> to keep track of it</li> <li><strong>Line 4</strong>: set our turtle drawing speed to fast</li> <li><strong>Line 5</strong>: tell our turtle to draw in “yellow” and fill in shapes with “red”</li> <li><strong>Line 6</strong>: set our turtle drawing width to 3 pixels</li> <li><strong>Line 8</strong>: begin our <code>for</code> loop and tell it to loop 36 times</li> <li><strong>Line 9-19</strong>: draw our box and then turn slightly right 10 degrees.</li> </ul> <p>Notice how lines 9 through 19 are indented under the <code>for</code> loop. This is important as it tells Python all these lines are part of the <code>for</code> loop. In Python the indentation is required to tell the program a set of statements are part of a block like this.</p> <p>Once you’ve got this entered, save our program and then run it. If your program runs without any syntax errors (which usually means you have a typo in your program), you should get a window like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-flower.png" width="1074" height="1036"></figure> </section><section><h2><a class="anchor" name="conclusion"></a> Conclusion</h2> <p>Congratulations, you’ve written your first colorful, interesting Python program! In the <a href="python-intro-functions-and-lists">next class in this series</a> you’ll learn how to write reusable “code building blocks” with functions:</p> <ul> <li><a href="python-intro-functions-and-lists">Let’s Program with Python: Functions and Lists (Part 2)</a></li> </ul></section><footer></footer></article>https://dbader.org/blog/python-intro-statements-variables-and-loopsThu, 20 Apr 2017 00:00:00 GMTDictionaries, Maps, and Hash Tables in Pythonhttps://dbader.org/blog/python-dictionaries-maps-and-hashtables<article><header><h1>Dictionaries, Maps, and Hash Tables in Python</h1> <p>Need a dictionary, map, or hash table to implement an algorithm in your Python program? Read on to see how the Python standard library can help you.</p> </header><section><figure><img alt="" src="/blog/figures/dictionaries-maps-and-hashtables-in-python.png" width="1280" height="720"></figure> <p>In Python, dictionaries (or “dicts”, for short) are a central data structure:</p> <p>Dicts store an arbitrary number of objects, each identified by a unique dictionary <em>key</em>. Dictionaries are often also called <em>maps</em>, <em>hashmaps</em>, <em>lookup tables</em>, or <em>associative arrays</em>. They allow the efficient lookup, insertion, and deletion of any object associated with a given key.</p> <p>To give a more practical explanation—<em>phone books</em> are a decent real-world analog for dictionaries:</p> <blockquote> <p>Phone books allow you to quickly retrieve the information (phone number) associated with a given key (a person’s name). Instead of having to read a phonebook front to back in order to find someone’s number you can jump more or less directly to a name and look up the associated number.</p> </blockquote> <p>This analogy breaks down somewhat when it comes to how the information is organized to allow for fast lookups. But the fundamental performance characteristics hold:</p> <p>Dictionaries allow you to quickly find the information associated with a given key.</p> </section><section><h2>Python Dictionaries, Hashmaps, and Hash Tables</h2> <p>The dictionary abstract data type is one of the most frequently used and most important data structures in computer science. Because of this importance Python features a robust dictionary implementation as one of its built-in data types (<a href="https://docs.python.org/3/library/stdtypes.html#mapping-types-dict" target="_blank"><code>dict</code></a>).</p> <p>Python even provides some useful syntactic sugar for working with dictionaries in your programs. For example, the curly-braces dictionary expression syntax (<code>{}</code>) and dictionary comprehensions allow you to conveniently define new dictionaries:</p> <div class="codehilite"><pre><span></span><span class="n">phonebook</span> <span class="o">=</span> <span class="p">{</span> <span class="s1">'bob'</span><span class="p">:</span> <span class="mi">7387</span><span class="p">,</span> <span class="s1">'alice'</span><span class="p">:</span> <span class="mi">3719</span><span class="p">,</span> <span class="s1">'jack'</span><span class="p">:</span> <span class="mi">7052</span><span class="p">,</span> <span class="p">}</span> <span class="n">squares</span> <span class="o">=</span> <span class="p">{</span><span class="n">x</span><span class="p">:</span> <span class="n">x</span> <span class="o">*</span> <span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)}</span> </pre></div> <p>Python’s dictionaries are indexed by keys that can be of any <a href="https://docs.python.org/3/glossary.html#term-hashable" target="_blank">hashable</a> type. A hashable object has a hash value which never changes during its lifetime (see <code>__hash__</code>), and it can be compared to other objects (see <code>__eq__</code>).</p> <p>In addition, hashable objects which compare equal must have the same hash value. Immutable types like strings and numbers work well as dictionary keys. You can also use tuples as dictionary keys as long as they contain only hashable types themselves.</p> </section><section><h2>✅ Built-in <a href="https://docs.python.org/3/library/stdtypes.html#mapping-types-dict" target="_blank">dict</a> type</h2> <p>For most use cases you’ll face Python’s built-in dictionary implementation will do everything you need. Dictionaries are highly optimized and underlie many parts of the language, for example class attributes and variables in a stack frame are both stored internally in dictionaries.</p> <p>Python dictionaries are based on a well-tested and finely tuned hash table implementation that provides the performance characteristics you’d expect: <em>O(1)</em> time complexity for lookup, insert, update, and delete operations in the average case.</p> <p>There’s little reason to not use the standard <code>dict</code> implementation included with Python. However, specialized third-party dictionary data structures exist, for example skip lists or B-tree based dictionary implementations.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">phonebook</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'bob'</span><span class="p">:</span> <span class="mi">7387</span><span class="p">,</span> <span class="s1">'alice'</span><span class="p">:</span> <span class="mi">3719</span><span class="p">,</span> <span class="s1">'jack'</span><span class="p">:</span> <span class="mi">7052</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">phonebook</span><span class="p">[</span><span class="s1">'alice'</span><span class="p">]</span> <span class="mi">3719</span> </pre></div> <p>Interestingly, <strong>Python ships with a number of specialized dictionary implementations in its standard library</strong>. These specialized dictionaries are all based on the built-in dictionary implementation (and share its performance characteristics) but add some convenience features:</p> </section><section><h2>✅ <a href="https://docs.python.org/3/library/collections.html#collections.OrderedDict" target="_blank">collections.OrderedDict</a> – Remember the insertion order of keys</h2> <p>A dictionary subclass that remembers the insertion order of keys added to the collection.</p> <p>While standard <code>dict</code> instances <a href="https://mail.python.org/pipermail/python-dev/2016-September/146327.html" target="_blank">preserve the insertion order of keys in CPython 3.6+</a> this is just a side effect of the CPython implementation and not defined in the language spec. If key order is important for your algorithm to work it’s best to communicate this clearly by using the <code>OrderDict</code> class.</p> <p><code>OrderedDict</code> is not a built-in part of the core language and must be imported from the <code>collections</code> module in the standard library.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">collections</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">d</span> <span class="o">=</span> <span class="n">collections</span><span class="o">.</span><span class="n">OrderedDict</span><span class="p">(</span><span class="n">one</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">two</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">three</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">d</span> <span class="n">OrderedDict</span><span class="p">([(</span><span class="s1">'one'</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="s1">'two'</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="p">(</span><span class="s1">'three'</span><span class="p">,</span> <span class="mi">3</span><span class="p">)])</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">d</span><span class="p">[</span><span class="s1">'four'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">4</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">d</span> <span class="n">OrderedDict</span><span class="p">([(</span><span class="s1">'one'</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="s1">'two'</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="p">(</span><span class="s1">'three'</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span> <span class="p">(</span><span class="s1">'four'</span><span class="p">,</span> <span class="mi">4</span><span class="p">)])</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">d</span><span class="o">.</span><span class="n">keys</span><span class="p">()</span> <span class="n">odict_keys</span><span class="p">([</span><span class="s1">'one'</span><span class="p">,</span> <span class="s1">'two'</span><span class="p">,</span> <span class="s1">'three'</span><span class="p">,</span> <span class="s1">'four'</span><span class="p">])</span> </pre></div> </section><section><h2>✅ <a href="https://docs.python.org/3/library/collections.html#collections.defaultdict" target="_blank">collections.defaultdict</a> – Return default values for missing keys</h2> <p>Another dictionary subclass that accepts a default value in its constructor that will be returned if a requested key cannot be found in a <code>defaultdict</code> instance. This can save some typing and make the programmer’s intention more clear compared to using the <code>get()</code> methods or catching a <code>KeyError</code> exception in regular dictionaries.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">defaultdict</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dd</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">list</span><span class="p">)</span> <span class="c1"># Accessing a missing key creates it and initializes it</span> <span class="c1"># using the default factory, i.e. list() in this example:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dd</span><span class="p">[</span><span class="s1">'dogs'</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'Rufus'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dd</span><span class="p">[</span><span class="s1">'dogs'</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'Kathrin'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dd</span><span class="p">[</span><span class="s1">'dogs'</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'Mr Sniffles'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dd</span><span class="p">[</span><span class="s1">'dogs'</span><span class="p">]</span> <span class="p">[</span><span class="s1">'Rufus'</span><span class="p">,</span> <span class="s1">'Kathrin'</span><span class="p">,</span> <span class="s1">'Mr Sniffles'</span><span class="p">]</span> </pre></div> </section><section><h2>✅ <a href="https://docs.python.org/3/library/collections.html#collections.ChainMap" target="_blank">collections.ChainMap</a> – Search multiple dictionaries as a single mapping</h2> <p>This data structure groups multiple dictionaries into a single mapping. Lookups search the underlying mappings one by one until a key is found. Insertions, updates, and deletions only affect the first mapping added to the chain.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">ChainMap</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dict1</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'one'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">'two'</span><span class="p">:</span> <span class="mi">2</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dict2</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'three'</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span> <span class="s1">'four'</span><span class="p">:</span> <span class="mi">4</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">chain</span> <span class="o">=</span> <span class="n">ChainMap</span><span class="p">(</span><span class="n">dict1</span><span class="p">,</span> <span class="n">dict2</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">chain</span> <span class="n">ChainMap</span><span class="p">({</span><span class="s1">'one'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">'two'</span><span class="p">:</span> <span class="mi">2</span><span class="p">},</span> <span class="p">{</span><span class="s1">'three'</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span> <span class="s1">'four'</span><span class="p">:</span> <span class="mi">4</span><span class="p">})</span> <span class="c1"># ChainMap searches each collection in the chain</span> <span class="c1"># from left to right until it finds the key (or fails):</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">chain</span><span class="p">[</span><span class="s1">'three'</span><span class="p">]</span> <span class="mi">3</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">chain</span><span class="p">[</span><span class="s1">'one'</span><span class="p">]</span> <span class="mi">1</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">chain</span><span class="p">[</span><span class="s1">'missing'</span><span class="p">]</span> <span class="ne">KeyError</span><span class="p">:</span> <span class="s1">'missing'</span> </pre></div> </section><section><h2>✅ <a href="https://docs.python.org/3/library/types.html#types.MappingProxyType" target="_blank">types.MappingProxyType</a> – A wrapper for making read-only dictionaries</h2> <p>A wrapper around a standard dictionary that provides a read-only view into the wrapped dictionary’s data. This class was added in Python 3.3 and it can be used to create immutable proxy versions of dictionaries.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">types</span> <span class="kn">import</span> <span class="n">MappingProxyType</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">read_only</span> <span class="o">=</span> <span class="n">MappingProxyType</span><span class="p">({</span><span class="s1">'one'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">'two'</span><span class="p">:</span> <span class="mi">2</span><span class="p">})</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">read_only</span><span class="p">[</span><span class="s1">'one'</span><span class="p">]</span> <span class="mi">1</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">read_only</span><span class="p">[</span><span class="s1">'one'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">23</span> <span class="ne">TypeError</span><span class="p">:</span> <span class="s2">"'mappingproxy' object does not support item assignment"</span> </pre></div> </section><section><h2>Using Dictionaries in Python: Conclusion</h2> <p>All of the Python hashmap implementations I listed in this tutorial are valid implementations built into the Python standard library.</p> <p>If you’re looking for a general recommendation on which mapping type to use in your Python programs, then I’d point you to the built-in <code>dict</code> data type. It’s a versatile and optimized dictionary implementation that’s built directly into the core language.</p> <p>Only if you have special requirements that go beyond what’s provided by <code>dict</code> would I recommend that you use one of the other data types listed here. Yes, I still believe they’re valid options—but usually your code will be clearer and easier to maintain by other developers if it relies on standard Python dictionaries most of the time.</p> <p><em><a href="fundamental-data-structures-in-python">Read the full “Fundamental Data Structures in Python” article series here.</a> This article is missing something or you found an error? Help a brother out and leave a comment below.</em></p></section><footer></footer></article>https://dbader.org/blog/python-dictionaries-maps-and-hashtablesTue, 18 Apr 2017 00:00:00 GMTPriority Queues in Pythonhttps://dbader.org/blog/priority-queues-in-python<article><header><h1>Priority Queues in Python</h1> <p>What are the various ways you can implement a priority queue in Python? Read on and find out what the Python standard library has to offer.</p> </header><section><figure><img alt="" src="/blog/figures/priority-queues-in-python.png" width="1280" height="720"></figure> <p>A priority queue is a container data structure that manages a set of records with <a href="https://en.wikipedia.org/wiki/Total_order" target="_blank">totally-ordered</a> keys (for example, a numeric <em>weight</em> value) to provide quick access to the record with the <em>smallest</em> or <em>largest</em> key in the set.</p> <p>You can think of a priority queue as a modified queue: instead of retrieving the next element by insertion time, it retrieves the <em>highest-priority</em> element. The priority of individual elements is decided by the ordering applied to their keys.</p> <p>Priority queues are commonly used for dealing with scheduling problems. For example, to give precedence to tasks with higher urgency.</p> <p>For example, let’s take an operating system task scheduler—ideally, high-priority tasks on the system (e.g. playing a real-time game) should take precedence over lower-priority tasks (e.g. downloading updates in the background). By organizing pending tasks in a priority queue that uses the task urgency as the key, the task scheduler can allow the highest-priority tasks to run first.</p> <p>Let’s take a look at a few options for how you can implement Priority Queues in Python using built-in data structures or data structures that ship with Python’s standard library. They each have their up- and downsides, but in my mind there’s a clear winner for most common scenarios. But see for yourself:</p> </section><section><h2>⛔ Keeping a Manually Sorted List</h2> <p>You can use a sorted <code>list</code> to quickly identify and delete the smallest or largest element. The downside is that inserting new elements into a list is a slow <em>O(n)</em> operation.</p> <p>While the insertion point can be found in <em>O(log n)</em> time using <a href="https://docs.python.org/3/library/bisect.html#bisect.insort" target="_blank"><code>bisect.insort</code></a> in the standard library, this is always dominated by the slow insertion step.</p> <p>Maintaining the order by appending to the list and re-sorting also takes at least <em>O(n log n)</em> time.</p> <p>Therefore sorted lists are <strong>only suitable when there will be few insertions</strong> into the priority queue.</p> <div class="codehilite"><pre><span></span><span class="n">q</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'code'</span><span class="p">))</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'eat'</span><span class="p">))</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="mi">3</span><span class="p">,</span> <span class="s1">'sleep'</span><span class="p">))</span> <span class="c1"># NOTE: Remember to re-sort every time</span> <span class="c1"># a new element is inserted, or use</span> <span class="c1"># bisect.insort().</span> <span class="n">q</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">reverse</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="k">while</span> <span class="n">q</span><span class="p">:</span> <span class="n">next_item</span> <span class="o">=</span> <span class="n">q</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="n">next_item</span><span class="p">)</span> <span class="c1"># Result:</span> <span class="c1"># (1, 'eat')</span> <span class="c1"># (2, 'code')</span> <span class="c1"># (3, 'sleep')</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/heapq.html" target="_blank">heapq</a> Module</h2> <p>This is a binary heap implementation usually backed by a plain <code>list</code> and it supports insertion and extraction of the smallest element in <em>O(log n)</em> time.</p> <p>This module is a good choice for implementing priority queues in Python. Because <code>heapq</code> technically only provides a min-heap implementation, extra steps must be taken to ensure sort stability and other features <a href="https://docs.python.org/3/library/heapq.html#priority-queue-implementation-notes" target="_blank">typically expected from a “practical” priority queue</a>.</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">heapq</span> <span class="n">q</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">heapq</span><span class="o">.</span><span class="n">heappush</span><span class="p">(</span><span class="n">q</span><span class="p">,</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'code'</span><span class="p">))</span> <span class="n">heapq</span><span class="o">.</span><span class="n">heappush</span><span class="p">(</span><span class="n">q</span><span class="p">,</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'eat'</span><span class="p">))</span> <span class="n">heapq</span><span class="o">.</span><span class="n">heappush</span><span class="p">(</span><span class="n">q</span><span class="p">,</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="s1">'sleep'</span><span class="p">))</span> <span class="k">while</span> <span class="n">q</span><span class="p">:</span> <span class="n">next_item</span> <span class="o">=</span> <span class="n">heapq</span><span class="o">.</span><span class="n">heappop</span><span class="p">(</span><span class="n">q</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">next_item</span><span class="p">)</span> <span class="c1"># Result:</span> <span class="c1"># (1, 'eat')</span> <span class="c1"># (2, 'code')</span> <span class="c1"># (3, 'sleep')</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/queue.html#queue.PriorityQueue" target="_blank">queue.PriorityQueue</a> Class</h2> <p>This priority queue implementation uses <code>heapq</code> internally and shares the same time and space complexities.</p> <p>The difference is that <code>PriorityQueue</code> is synchronized and provides locking semantics to support multiple concurrent producers and consumers.</p> <p>Depending on your use case this might be helpful, or just incur unneeded overhead. In any case you might prefer its class-based interface over using the function-based interface provided by <code>heapq</code>.</p> <div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">queue</span> <span class="kn">import</span> <span class="n">PriorityQueue</span> <span class="n">q</span> <span class="o">=</span> <span class="n">PriorityQueue</span><span class="p">()</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'code'</span><span class="p">))</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">((</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'eat'</span><span class="p">))</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">((</span><span class="mi">3</span><span class="p">,</span> <span class="s1">'sleep'</span><span class="p">))</span> <span class="k">while</span> <span class="ow">not</span> <span class="n">q</span><span class="o">.</span><span class="n">empty</span><span class="p">():</span> <span class="n">next_item</span> <span class="o">=</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="n">next_item</span><span class="p">)</span> <span class="c1"># Result:</span> <span class="c1"># (1, 'eat')</span> <span class="c1"># (2, 'code')</span> <span class="c1"># (3, 'sleep')</span> </pre></div> </section><section><h2>A good default choice: <code>queue.PriorityQueue</code></h2> <p>Now which priority queue implementation should you use in your Python programs? They each have slightly different use cases. But in my mind <code>queue.PriorityQueue</code> is a good default choice.</p> <p>Sure, it might incur some unnecessary locking overhead—but it has a nice object-oriented interface and a name that states its intent clearly.</p> <p><em><a href="fundamental-data-structures-in-python">Read the full “Fundamental Data Structures in Python” article series here.</a> This article is missing something or you found an error? Help a brother out and leave a comment below.</em></p></section><footer></footer></article>https://dbader.org/blog/priority-queues-in-pythonWed, 12 Apr 2017 00:00:00 GMTFundamental Data Structures in Pythonhttps://dbader.org/blog/fundamental-data-structures-in-python<article><header><h1>Fundamental Data Structures in Python</h1> <p>In this article series we’ll take a tour of some fundamental data structures and implementations of abstract data types (ADTs) available in Python’s standard library.</p> </header><section><figure><img alt="" src="/blog/figures/fundamental-data-structures-in-python.png" width="1280" height="720"></figure> <p>Data structures are the fundamental constructs around which you build your applications. Each data structure provides a particular way of organizing data so it can be accessed efficiently, depending on the use case at hand.</p> <p>Python ships with an extensive set of data structures in its standard library. However due to naming differences it’s often unclear how even well-known “abstract data types” correspond to a specific implementation in Python.</p> <p>Other languages like Java stick to a more “computer sciencey” and explicit naming scheme for their standard data structures. For example, a list isn’t just a “list” in Java—it’s either a <code>LinkedList</code> or an <code>ArrayList</code>. This makes it easier to recognize the computational complexity of these types.</p> <p>Python favors a simpler and more “human” naming scheme. The downside is that to a Python initiate it’s unclear whether the built-in <code>list</code> type is implemented as a linked list or a dynamic array.</p> <p>My goal with this article series is to clarify how the most common abstract data types map to Python’s naming scheme and to provide a brief description for each. This information will also help you in Python coding interviews.</p> <p>If you’re looking for a good book to brush up your data structures knowledge I highly recommend Steven S. Skiena’s <a href="http://amzn.to/1oeOte3" target="_blank">The Algorithm Design Manual</a>.</p> <p>It strikes a great balance between teaching you fundamental (and more advanced) data structures and then showing you how to put them to practical use in various algorithms. Steve’s book was a great help in the writing of this series.</p> <p>Alright, let’s get started. This article serves as the “hub” for the individual data structure tutorials that I’ll link in the list below:</p> </section><section><h2>Python Data Structures Tutorials</h2> <ul> <li><a href="python-dictionaries-maps-and-hashtables">Dictionaries, Maps, and Hash Tables in Python</a></li> <li><a href="sets-and-multiset-in-python">Sets and Multisets in Python</a></li> <li><a href="stacks-in-python">Stacks in Python</a></li> <li><a href="queues-in-python">Queues in Python</a></li> <li><a href="priority-queues-in-python">Priority Queues in Python</a></li> <li><a href="records-structs-and-data-transfer-objects-in-python">Records, Structs, and Data Transfer Objects in Python</a></li> <li><a href="python-linked-list">Linked Lists in Python</a></li> <li>Arrays in Python (<em>Coming Soon</em>)</li> </ul> <p>By the way, I’m always looking to improve these tutorials so if you find an error or would like to suggest an addition—please leave a comment on the article or reach out to me via email or <a href="https://twitter.com/@dbader_org" target="_blank">Twitter</a>.</p></section><footer></footer></article>https://dbader.org/blog/fundamental-data-structures-in-pythonTue, 11 Apr 2017 00:00:00 GMTPython Decorators: A Step-By-Step Introductionhttps://dbader.org/blog/python-decorators<article><header><h1>Python Decorators: A Step-By-Step Introduction</h1> <p>Understanding decorators is a milestone for any serious Python programmer. Here’s your step-by-step guide to how decorators can help you become a more efficient and productive Python developer.</p> </header><section><figure><img alt="Understanding Python Decorators" src="/blog/figures/python-decorators.png" width="1280" height="720"></figure> <p>Python’s decorators allow you to extend and modify the behavior of a callable (functions, methods, and classes) <em>without</em> permanently modifying the callable itself.</p> <p>Any sufficiently generic functionality you can “tack on” to an existing class or function’s behavior makes a great use case for decoration. This includes:</p> <ul> <li>logging,</li> <li>enforcing access control and authentication,</li> <li>instrumentation and timing functions,</li> <li>rate-limiting,</li> <li>caching; and more.</li> </ul> </section><section><h2>Why Should I Master Decorators in Python?</h2> <p>That’s a fair question. After all, what I just mentioned sounded quite abstract and it might be difficult to see <strong>how decorators can benefit you in your day-to-day work as a Python developer</strong>. Here’s an example:</p> <p>Imagine you’ve got 30 functions with business logic in your report-generating program. One rainy Monday morning your boss walks up to your desk and says:</p> <blockquote> <p>“Happy Monday! Remember those TPS reports? I need you to add input/output logging to each step in the report generator. XYZ Corp needs it for auditing purposes. Oh, and I told them we can ship this by Wednesday.”</p> </blockquote> <p>Depending on whether or not you’ve got a solid grasp on Python’s decorators, this request request will either send your blood pressure spiking—or leave you relatively calm.</p> <p><strong>Without decorators</strong> you might be spending the next three days scrambling to modify each of those 30 functions and clutter them up with manual logging calls. Fun times.</p> <p><strong>If you do know your decorators</strong>, you’ll calmly smile at your boss and say:</p> <blockquote> <p>“Don’t worry Jim, I’ll get it done by 2pm today.”</p> </blockquote> <p>Right after that you’ll type the code for a generic <code>@audit_log</code> decorator (that’s only about 10 lines long) and quickly paste it in front of each function definition. Then you’ll commit your code and grab another cup of coffee.</p> <p>I’m dramatizing here. But only a little. Decorators <em>can be</em> that powerful 🙂</p> <p>I’d go as far as to say that understanding decorators is a milestone for any serious Python programmer. They require a solid grasp of several advanced concepts in the language—including the <a href="python-first-class-functions">properties of <em>first-class functions</em></a>.</p> <p>But:</p> </section><section><h2>Understanding Decorators <u>Is Worth It</u> 💡</h2> <p>The payoff for understanding how decorators work in Python is huge.</p> <p>Sure, decorators are relatively complicated to wrap your head around for the first time—but they’re a highly useful feature that you’ll often encounter in third-party frameworks and the Python standard library.</p> <p>Explaining decorators is also a <em>make or break</em> moment for any good Python tutorial. I’ll do my best here to introduce you to them step by step.</p> <p>Before you dive in, now would be an excellent moment to refresh your memory on the properties of <em>first-class functions</em> in Python. I wrote <a href="python-first-class-functions">a tutorial on them here on dbader.org</a> and I would encourage you to take a few minutes to review it. The most important “first-class functions” takeaways for understanding decorators are:</p> <ul> <li><strong>Functions are objects</strong>—they can be assigned to variables and passed to and returned from other functions; and</li> <li><strong>Functions can be defined inside other functions</strong>—and a child function can capture the parent function’s local state (lexical closures.)</li> </ul> <p>Alright, ready to do this? Let’s start with some:</p> </section><section><h2>Python Decorator Basics</h2> <p>Now, what are decorators really? They “decorate” or “wrap” another function and let you execute code before and after the wrapped function runs.</p> <p>Decorators allow you to define reusable building blocks that can change or extend the behavior of other functions. And they let you do that without permanently modifying the wrapped function itself. The function’s behavior changes only when it’s <em>decorated</em>.</p> <p>Now what does the implementation of a simple decorator look like? In basic terms, a decorator is <em>a callable that takes a callable as input and returns another callable</em>.</p> <p>The following function has that property and could be considered the simplest decorator one could possibly write:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">null_decorator</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="k">return</span> <span class="n">func</span> </pre></div> <p>As you can see, <code>null_decorator</code> is a callable (it’s a function), it takes another callable as its input, and it returns the same input callable without modifying it.</p> <p>Let’s use it to <em>decorate</em> (or <em>wrap</em>) another function:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">greet</span><span class="p">():</span> <span class="k">return</span> <span class="s1">'Hello!'</span> <span class="n">greet</span> <span class="o">=</span> <span class="n">null_decorator</span><span class="p">(</span><span class="n">greet</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="p">()</span> <span class="s1">'Hello!'</span> </pre></div> <p>In this example I’ve defined a <code>greet</code> function and then immediately decorated it by running it through the <code>null_decorator</code> function. I know this doesn’t look very useful yet (I mean we specifically designed the null decorator to be useless, right?) but in a moment it’ll clarify how Python’s decorator syntax works.</p> <p>Instead of explicitly calling <code>null_decorator</code> on <code>greet</code> and then reassigning the <code>greet</code> variable, you can use Python’s <code>@</code> syntax for decorating a function in one step:</p> <div class="codehilite"><pre><span></span><span class="nd">@null_decorator</span> <span class="k">def</span> <span class="nf">greet</span><span class="p">():</span> <span class="k">return</span> <span class="s1">'Hello!'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="p">()</span> <span class="s1">'Hello!'</span> </pre></div> <p>Putting an <code>@null_decorator</code> line in front of the function definition is the same as defining the function first and then running through the decorator. Using the <code>@</code> syntax is just <em>syntactic sugar</em>, and a shortcut for this commonly used pattern.</p> <p>Note that using the <code>@</code> syntax decorates the function immediately at definition time. This makes it difficult to access the undecorated original without brittle hacks. Therefore you might choose to decorate some functions manually in order to retain the ability to call the undecorated function as well.</p> <p>So far, so good. Let’s see how:</p> </section><section><h2>Decorators Can Modify Behavior</h2> <p>Now that you’re a little more familiar with the decorator syntax, let’s write another decorator that <em>actually does something</em> and modifies the behavior of the decorated function.</p> <p>Here’s a slightly more complex decorator which converts the result of the decorated function to uppercase letters:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">uppercase</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="k">def</span> <span class="nf">wrapper</span><span class="p">():</span> <span class="n">original_result</span> <span class="o">=</span> <span class="n">func</span><span class="p">()</span> <span class="n">modified_result</span> <span class="o">=</span> <span class="n">original_result</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span> <span class="k">return</span> <span class="n">modified_result</span> <span class="k">return</span> <span class="n">wrapper</span> </pre></div> <p>Instead of simply returning the input function like the null decorator did, this <code>uppercase</code> decorator defines a new function on the fly (a closure) and uses it to <em>wrap</em> the input function in order to modify its behavior at call time.</p> <p>The <code>wrapper</code> closure has access to the undecorated input function and it is free to execute additional code before and after calling the input function. (Technically, it doesn’t even need to call the input function at all.)</p> <p>Note how up until now the decorated function has never been executed. Actually calling the input function at this point wouldn’t make any sense—you’ll want the decorator to be able to modify the behavior of its input function when it gets called eventually.</p> <p>Time to see the <code>uppercase</code> decorator in action. What happens if you decorate the original <code>greet</code> function with it?</p> <div class="codehilite"><pre><span></span><span class="nd">@uppercase</span> <span class="k">def</span> <span class="nf">greet</span><span class="p">():</span> <span class="k">return</span> <span class="s1">'Hello!'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="p">()</span> <span class="s1">'HELLO!'</span> </pre></div> <p>I hope this was the result you expected. Let’s take a closer look at what just happened here. Unlike <code>null_decorator</code>, our <code>uppercase</code> decorator returns a <em>different function object</em> when it decorates a function:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span> <span class="o">&lt;</span><span class="n">function</span> <span class="n">greet</span> <span class="n">at</span> <span class="mh">0x10e9f0950</span><span class="o">&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">null_decorator</span><span class="p">(</span><span class="n">greet</span><span class="p">)</span> <span class="o">&lt;</span><span class="n">function</span> <span class="n">greet</span> <span class="n">at</span> <span class="mh">0x10e9f0950</span><span class="o">&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">uppercase</span><span class="p">(</span><span class="n">greet</span><span class="p">)</span> <span class="o">&lt;</span><span class="n">function</span> <span class="n">uppercase</span><span class="o">.&lt;</span><span class="nb">locals</span><span class="o">&gt;.</span><span class="n">wrapper</span> <span class="n">at</span> <span class="mh">0x10da02f28</span><span class="o">&gt;</span> </pre></div> <p>And as you saw earlier, it needs to do that in order to modify the behavior of the decorated function when it finally gets called. The <code>uppercase</code> decorator is a function itself. And the only way to influence the “future behavior” of an input function it decorates is to replace (or <em>wrap</em>) the input function with a closure.</p> <p>That’s why <code>uppercase</code> defines and returns another function (the closure) that can then be called at a later time, run the original input function, and modify its result.</p> <p>Decorators modify the behavior of a callable through a wrapper so you don’t have to permanently modify the original. The callable isn’t permanently modified—its behavior changes only when decorated.</p> <p>This let’s you “tack on” reusable building blocks, like logging and other instrumentation, to existing functions and classes. It’s what makes decorators such a powerful feature in Python that’s frequently used in the standard library and in third-party packages.</p> <div class="update-box"> <h3>⏰ A Quick Intermission</h3> <p>By the way, if you feel like you need a quick coffee break at this point—that’s totally normal. In my opinion closures and decorators are some of the most difficult concepts to understand in Python. Take your time and don’t worry about figuring this out immediately. Playing through the code examples in an interpreter session one by one often helps make things sink in.</p> <p>I know you can do it 🙂</p> </div> </section><section><h2>Applying Multiple Decorators to a Single Function</h2> <p>Perhaps not surprisingly, you can apply more than one decorator to a function. This accumulates their effects and it’s what makes decorators so helpful as reusable building blocks.</p> <p>Here’s an example. The following two decorators wrap the output string of the decorated function in HTML tags. By looking at how the tags are nested you can see which order Python uses to apply multiple decorators:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">strong</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="k">def</span> <span class="nf">wrapper</span><span class="p">():</span> <span class="k">return</span> <span class="s1">'&lt;strong&gt;'</span> <span class="o">+</span> <span class="n">func</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'&lt;/strong&gt;'</span> <span class="k">return</span> <span class="n">wrapper</span> <span class="k">def</span> <span class="nf">emphasis</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="k">def</span> <span class="nf">wrapper</span><span class="p">():</span> <span class="k">return</span> <span class="s1">'&lt;em&gt;'</span> <span class="o">+</span> <span class="n">func</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'&lt;/em&gt;'</span> <span class="k">return</span> <span class="n">wrapper</span> </pre></div> <p>Now let’s take these two decorators and apply them to our <code>greet</code> function at the same time. You can use the regular <code>@</code> syntax for that and just “stack” multiple decorators on top of a single function:</p> <div class="codehilite"><pre><span></span><span class="nd">@strong</span> <span class="nd">@emphasis</span> <span class="k">def</span> <span class="nf">greet</span><span class="p">():</span> <span class="k">return</span> <span class="s1">'Hello!'</span> </pre></div> <p>What output do you expect to see if you run the decorated function? Will the <code>@emphasis</code> decorator add its <code>&lt;em&gt;</code> tag first or does <code>@strong</code> have precedence? Here’s what happens when you call the decorated function:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="p">()</span> <span class="s1">'&lt;strong&gt;&lt;em&gt;Hello!&lt;/em&gt;&lt;/strong&gt;'</span> </pre></div> <p>This clearly shows in what order the decorators were applied: from <em>bottom to top</em>. First, the input function was wrapped by the <code>@emphasis</code> decorator, and then the resulting (decorated) function got wrapped again by the <code>@strong</code> decorator.</p> <p>To help me remember this bottom to top order I like to call this behavior <em>decorator stacking</em>. You start building the stack at the bottom and then keep adding new blocks on top to work your way upwards.</p> <p>If you break down the above example and avoid the <code>@</code> syntax to apply the decorators, the chain of decorator function calls looks like this:</p> <div class="codehilite"><pre><span></span><span class="n">decorated_greet</span> <span class="o">=</span> <span class="n">strong</span><span class="p">(</span><span class="n">emphasis</span><span class="p">(</span><span class="n">greet</span><span class="p">))</span> </pre></div> <p>Again you can see here that the <code>emphasis</code> decorator is applied first and then the resulting wrapped function is wrapped again by the <code>strong</code> decorator.</p> <p>This also means that deep levels of decorator stacking will have an effect on performance eventually because they keep adding nested function calls. Usually this won’t be a problem in practice, but it’s something to keep in mind if you’re working on performance intensive code.</p> </section><section><h2>Decorating Functions That Accept Arguments</h2> <p>All examples so far only decorated a simple <em>nullary</em> <code>greet</code> function that didn’t take any arguments whatsoever. So the decorators you saw here up until now didn’t have to deal with forwarding arguments to the input function.</p> <p>If you try to apply one of these decorators to a function that takes arguments it will not work correctly. How do you decorate a function that takes arbitrary arguments?</p> <p>This is where <a href="https://www.youtube.com/watch?v=WcTXxX3vYgY" target="_blank">Python’s <code>*args</code> and <code>**kwargs</code> feature</a> for dealing with variable numbers of arguments comes in handy. The following <code>proxy</code> decorator takes advantage of that:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">proxy</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="k">def</span> <span class="nf">wrapper</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="k">return</span> <span class="n">func</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="k">return</span> <span class="n">wrapper</span> </pre></div> <p>There are two notable things going on with this decorator:</p> <ul> <li> <p>It uses the <code>*</code> and <code>**</code> operators in the <code>wrapper</code> closure definition to collect all positional and keyword arguments and stores them in variables (<code>args</code> and <code>kwargs</code>).</p> </li> <li> <p>The <code>wrapper</code> closure then forwards the collected arguments to the original input function using the <code>*</code> and <code>**</code> “argument unpacking” operators.</p> </li> </ul> <p>(It’s a bit unfortunate that the meaning of the star and double-star operators is overloaded and changes depending on the context they’re used in. But I hope you get the idea.)</p> <p>Let’s expand the technique laid out by the <code>proxy</code> decorator into a more useful practical example. Here’s a <code>trace</code> decorator that logs function arguments and results during execution time:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">trace</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="k">def</span> <span class="nf">wrapper</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'TRACE: calling {func.__name__}() '</span> <span class="n">f</span><span class="s1">'with {args}, {kwargs}'</span><span class="p">)</span> <span class="n">original_result</span> <span class="o">=</span> <span class="n">func</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'TRACE: {func.__name__}() '</span> <span class="n">f</span><span class="s1">'returned {original_result!r}'</span><span class="p">)</span> <span class="k">return</span> <span class="n">original_result</span> <span class="k">return</span> <span class="n">wrapper</span> </pre></div> <p>Decorating a function with <code>trace</code> and then calling it will print the arguments passed to the decorated function and its return value. This is still somewhat of a toy example—but in a pinch it makes a great debugging aid:</p> <div class="codehilite"><pre><span></span><span class="nd">@trace</span> <span class="k">def</span> <span class="nf">say</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">line</span><span class="p">):</span> <span class="k">return</span> <span class="n">f</span><span class="s1">'{name}: {line}'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">say</span><span class="p">(</span><span class="s1">'Jane'</span><span class="p">,</span> <span class="s1">'Hello, World'</span><span class="p">)</span> <span class="s1">'TRACE: calling say() with ("Jane", "Hello, World"), {}'</span> <span class="s1">'TRACE: say() returned "Jane: Hello, World"'</span> <span class="s1">'Jane: Hello, World'</span> </pre></div> <p>Speaking of debugging—there are some things you should keep in mind when debugging decorators:</p> </section><section><h2>How to Write “Debuggable” Decorators</h2> <p>When you use a decorator, really what you’re doing is replacing one function with another. One downside of this process is that it “hides” some of the metadata attached to the original (undecorated) function.</p> <p>For example, the original function name, its docstring, and parameter list are hidden by the wrapper closure:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">greet</span><span class="p">():</span> <span class="sd">"""Return a friendly greeting."""</span> <span class="k">return</span> <span class="s1">'Hello!'</span> <span class="n">decorated_greet</span> <span class="o">=</span> <span class="n">uppercase</span><span class="p">(</span><span class="n">greet</span><span class="p">)</span> </pre></div> <p>If you try to access any of that function metadata you’ll see the wrapper closure’s metadata instead:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="o">.</span><span class="vm">__name__</span> <span class="s1">'greet'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="o">.</span><span class="vm">__doc__</span> <span class="s1">'Return a friendly greeting.'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">decorated_greet</span><span class="o">.</span><span class="vm">__name__</span> <span class="s1">'wrapper'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">decorated_greet</span><span class="o">.</span><span class="vm">__doc__</span> <span class="bp">None</span> </pre></div> <p>This makes debugging and working with the Python interpreter awkward and challenging. Thankfully there’s a quick fix for this: the <a href="https://docs.python.org/3/library/functools.html#functools.wraps" target="_blank"><code>functools.wraps</code> decorator</a> included in Python’s standard library.</p> <p>You can use <code>functools.wraps</code> in your own decorators to copy over the lost metadata from the undecorated function to the decorator closure. Here’s an example:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">functools</span> <span class="k">def</span> <span class="nf">uppercase</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="nd">@functools.wraps</span><span class="p">(</span><span class="n">func</span><span class="p">)</span> <span class="k">def</span> <span class="nf">wrapper</span><span class="p">():</span> <span class="k">return</span> <span class="n">func</span><span class="p">()</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span> <span class="k">return</span> <span class="n">wrapper</span> </pre></div> <p>Applying <code>functools.wraps</code> to the wrapper closure returned by the decorator carries over the docstring and other metadata of the input function:</p> <div class="codehilite"><pre><span></span><span class="nd">@uppercase</span> <span class="k">def</span> <span class="nf">greet</span><span class="p">():</span> <span class="sd">"""Return a friendly greeting."""</span> <span class="k">return</span> <span class="s1">'Hello!'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="o">.</span><span class="vm">__name__</span> <span class="s1">'greet'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="o">.</span><span class="vm">__doc__</span> <span class="s1">'Return a friendly greeting.'</span> </pre></div> <p>As a best practice I’d recommend that you use <code>functools.wraps</code> in all of the decorators you write yourself. It doesn’t take much time and it will save you (and others) debugging headaches down the road.</p> </section><section><h2>Python Decorators – Key Takeaways</h2> <ul> <li>Decorators define reusable building blocks you can apply to a callable to modify its behavior without permanently modifying the callable itself.</li> <li>The <code>@</code> syntax is just a shorthand for calling the decorator on an input function. Multiple decorators on a single function are applied bottom to top (<em>decorator stacking</em>).</li> <li>As a debugging best practice, use the <a href="https://docs.python.org/3/library/functools.html#functools.wraps" target="_blank"><code>functools.wraps</code></a> helper in your own decorators to carry over metadata from the undecorated callable to the decorated one.</li> </ul> <p>Was this tutorial helpful? Got any suggestions on how it could be improved that could help other learners? Leave a comment below and share your thoughts.</p></section><footer></footer></article>https://dbader.org/blog/python-decoratorsTue, 04 Apr 2017 00:00:00 GMTFinding and Choosing Quality Python Packageshttps://dbader.org/blog/finding-and-choosing-quality-python-packages<article><header><h1>Finding and Choosing Quality Python Packages</h1> <p>PyPI, the Python packaging repository, just crossed 100,000 third-party packages in total the other week. That’s an <em>overwhelming</em> number of packages to choose from.</p> </header><section><figure><img alt="Finding and Choosing Quality Python Packages" src="/blog/figures/finding-and-choosing-python-packages.png" width="1280" height="720"></figure> </section><section><h2>The Quest for the Perfect Python Package</h2> <p>Back when I got “serious” about building my Python skills, mastering the syntax of the language was <em>not</em> the hardest part. Python’s syntax seemed quite clear and intuitive by comparison, and there was a (relatively) obvious path to learning it from books and other resources.</p> <p>But when it came to Python’s <em>tens of thousands</em> of libraries and frameworks that was simply an overwhelming number to choose from. Memorizing them was (and still is) an impossible task.</p> <p>And this feeling of overwhelm and “choice paralysis” is exactly what held me back earlier on in my Python career.</p> </section><section><h2>Mastering Python ≠ Mastering the Syntax</h2> <p>What tripped me up as a fledgling Pythonista was this: I had the basics of Python under my belt, but I struggled when it came to adopting the right workflows and tools of the “ecosystem” surrounding the core language.</p> <p>Thus, I wasted time reinventing existing solutions left and right—sometimes I spent <em>days</em> writing my own (terrible) versions of common building blocks like config file parsers, data validators, or visualization tools.</p> <p>Now, sure I learned quite a bit from doing that…</p> </section><section><h2>Overcoming “Reinventing the Wheel Disease”</h2> <p>But I kept repeating the same mistake and was “reinventing the wheel” even when under a tight deadline. In hindsight, my ignorance caused me a ton of undue stress and sleep deprivation.</p> <p>Part of it was overconfidence in my abilities, and another part was a lack of experience using “bread and butter” tools like the <code>pip</code> package manager, virtual environments, and requirements files.</p> <p>Once I got the hang of Python’s dependency management tools and workflows I was able to overcome my “reinventing the wheel disease” quickly.</p> </section><section><h2>Dependency Management Skills Are <u>Key</u></h2> <p>Mastering those tools and coming up with strategies for identifying high-quality Python packages opened up a whole new world to me:</p> <p>By leveraging Python’s packaging ecosystem I was suddenly coding at a higher level of abstraction—and it had a <em>massive</em> impact on my productivity and efficiency. Saying it allowed me to “10X” my output would <em>not</em> be too far off.</p> <p>If you use Python and you’re wondering how to go from “writing scripts” to “building applications”—then there’s a good chance you could benefit from focusing on your dependency management skills.</p> <p>You might be ready for a similar “quantum leap” in your productivity.</p> <p>To discover the strategies and exact steps I used to break through this barrier, check out my new “Managing Python Dependencies” course:</p> <p><a href="/products/managing-python-dependencies/"><strong>Click to Learn More About “Managing Python Dependencies” →</strong></a></p></section><footer></footer></article>https://dbader.org/blog/finding-and-choosing-quality-python-packagesTue, 28 Mar 2017 00:00:00 GMTPython’s Functions Are First-Classhttps://dbader.org/blog/python-first-class-functions<article><header><h1>Python’s Functions Are First-Class</h1> <p>Python’s functions are first-class objects. You can assign them to variables, store them in data structures, pass them as arguments to other functions, and even return them as values from other functions.</p> </header><section><figure><img alt="" src="/blog/figures/python-first-class-functions.png" width="1280" height="720"></figure> <p>Grokking these concepts intuitively will make understanding advanced features in Python like <a href="/blog/python-lambda-functions">lambdas</a> and decorators much easier. It also puts you on a path towards functional programming techniques.</p> <p>In this tutorial I’ll guide you through a number of examples to help you develop this intuitive understanding. The examples will build on top of one another, so you might want to read them in sequence and even to try out some of them in a Python interpreter session as you go along.</p> <p>Wrapping your head around the concepts we’ll be discussing here might take a little longer than expected. Don’t worry—that’s completely normal. I’ve been there. You might feel like you’re banging your head against the wall, and then suddenly things will “click” and fall into place when you’re ready.</p> <p>Throughout this tutorial I’ll be using this <code>yell</code> function for demonstration purposes. It’s a simple toy example with easily recognizable output:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">yell</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="k">return</span> <span class="n">text</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'!'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">yell</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span> <span class="s1">'HELLO!'</span> </pre></div> </section><section><h2>Functions Are Objects</h2> <p>All data in a Python program is <a href="https://docs.python.org/3/reference/datamodel.html#objects-values-and-types" target="_blank">represented by objects or relations between objects</a>. Things like strings, lists, modules, and functions are all objects. There’s nothing particularly special about functions in Python.</p> <p>Because the <code>yell</code> function is an <em>object</em> in Python you can assign it to another variable, just like any other object:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">bark</span> <span class="o">=</span> <span class="n">yell</span> </pre></div> <p>This line doesn’t call the function. It takes the function object referenced by <code>yell</code> and creates a second name pointing to it, <code>bark</code>. You could now also execute the same underlying function object by calling <code>bark</code>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">bark</span><span class="p">(</span><span class="s1">'woof'</span><span class="p">)</span> <span class="s1">'WOOF!'</span> </pre></div> <p>Function objects and their names are two separate concerns. Here’s more proof: You can delete the function’s original name (<code>yell</code>). Because another name (<code>bark</code>) still points to the underlying function you can still call the function through it:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">del</span> <span class="n">yell</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">yell</span><span class="p">(</span><span class="s1">'hello?'</span><span class="p">)</span> <span class="ne">NameError</span><span class="p">:</span> <span class="s2">"name 'yell' is not defined"</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">bark</span><span class="p">(</span><span class="s1">'hey'</span><span class="p">)</span> <span class="s1">'HEY!'</span> </pre></div> <p>By the way, Python attaches a string identifier to every function at creation time for debugging purposes. You can access this internal identifier with the <code>__name__</code> attribute:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">bark</span><span class="o">.</span><span class="vm">__name__</span> <span class="s1">'yell'</span> </pre></div> <p>While the function’s <code>__name__</code> is still “yell” that won’t affect how you can access it from your code. This identifier is merely a debugging aid. A <em>variable pointing to a function</em> and the <em>function itself</em> are two separate concerns.</p> <p>(<a href="https://www.python.org/dev/peps/pep-3155/" target="_blank">Since Python 3.3</a> there’s also <code>__qualname__</code> which serves a similar purpose and provides a <em>qualified name</em> string to disambiguate function and class names.)</p> </section><section><h2>Functions Can Be Stored In Data Structures</h2> <p>As functions are first-class citizens you can store them in data structures, just like you can with other objects. For example, you can add functions to a list:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">funcs</span> <span class="o">=</span> <span class="p">[</span><span class="n">bark</span><span class="p">,</span> <span class="nb">str</span><span class="o">.</span><span class="n">lower</span><span class="p">,</span> <span class="nb">str</span><span class="o">.</span><span class="n">capitalize</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">funcs</span> <span class="p">[</span><span class="o">&lt;</span><span class="n">function</span> <span class="n">yell</span> <span class="n">at</span> <span class="mh">0x10ff96510</span><span class="o">&gt;</span><span class="p">,</span> <span class="o">&lt;</span><span class="n">method</span> <span class="s1">'lower'</span> <span class="n">of</span> <span class="s1">'str'</span> <span class="n">objects</span><span class="o">&gt;</span><span class="p">,</span> <span class="o">&lt;</span><span class="n">method</span> <span class="s1">'capitalize'</span> <span class="n">of</span> <span class="s1">'str'</span> <span class="n">objects</span><span class="o">&gt;</span><span class="p">]</span> </pre></div> <p>Accessing the function objects stored inside the list works like it would with any other type of object:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">funcs</span><span class="p">:</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">f</span><span class="p">(</span><span class="s1">'hey there'</span><span class="p">))</span> <span class="o">&lt;</span><span class="n">function</span> <span class="n">yell</span> <span class="n">at</span> <span class="mh">0x10ff96510</span><span class="o">&gt;</span> <span class="s1">'HEY THERE!'</span> <span class="o">&lt;</span><span class="n">method</span> <span class="s1">'lower'</span> <span class="n">of</span> <span class="s1">'str'</span> <span class="n">objects</span><span class="o">&gt;</span> <span class="s1">'hey there'</span> <span class="o">&lt;</span><span class="n">method</span> <span class="s1">'capitalize'</span> <span class="n">of</span> <span class="s1">'str'</span> <span class="n">objects</span><span class="o">&gt;</span> <span class="s1">'Hey there'</span> </pre></div> <p>You can even call a function object stored in the list without assigning it to a variable first. You can do the lookup and then immediately call the resulting “disembodied” function object within a single expression:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">funcs</span><span class="p">[</span><span class="mi">0</span><span class="p">](</span><span class="s1">'heyho'</span><span class="p">)</span> <span class="s1">'HEYHO!'</span> </pre></div> </section><section><h2>Functions Can Be Passed To Other Functions</h2> <p>Because functions are objects you can pass them as arguments to other functions. Here’s a <code>greet</code> function that formats a greeting string using the function object passed to it and then prints it:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">greet</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="n">greeting</span> <span class="o">=</span> <span class="n">func</span><span class="p">(</span><span class="s1">'Hi, I am a Python program'</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">greeting</span><span class="p">)</span> </pre></div> <p>You can influence the resulting greeting by passing in different functions. Here’s what happens if you pass the <code>yell</code> function to <code>greet</code>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="p">(</span><span class="n">yell</span><span class="p">)</span> <span class="s1">'HI, I AM A PYTHON PROGRAM!'</span> </pre></div> <p>Of course you could also define a new function to generate a different flavor of greeting. For example, the following <code>whisper</code> function might work better if you don’t want your Python programs to sound like Optimus Prime:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">whisper</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="k">return</span> <span class="n">text</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'...'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="p">(</span><span class="n">whisper</span><span class="p">)</span> <span class="s1">'hi, i am a python program...'</span> </pre></div> <p>The ability to pass function objects as arguments to other functions is powerful. It allows you to abstract away and pass around <em>behavior</em> in your programs. In this example, the <code>greet</code> function stays the same but you can influence its output by passing in different <em>greeting behaviors</em>.</p> <p>Functions that can accept other functions as arguments are also called <em>higher-order functions</em>. They are a necessity for the functional programming style.</p> <p>The classical example for higher-order functions in Python is the built-in <code>map</code> function. It takes a function and an iterable and calls the function on each element in the iterable, yielding the results as it goes along.</p> <p>Here’s how you might format a sequence of greetings all at once by <em>mapping</em> the <code>yell</code> function to them:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">list</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="n">yell</span><span class="p">,</span> <span class="p">[</span><span class="s1">'hello'</span><span class="p">,</span> <span class="s1">'hey'</span><span class="p">,</span> <span class="s1">'hi'</span><span class="p">]))</span> <span class="p">[</span><span class="s1">'HELLO!'</span><span class="p">,</span> <span class="s1">'HEY!'</span><span class="p">,</span> <span class="s1">'HI!'</span><span class="p">]</span> </pre></div> <p><code>map</code> has gone through the entire list and applied the <code>yell</code> function to each element.</p> </section><section><h2>Functions Can Be Nested</h2> <p>Python allows functions to be defined inside other functions. These are often called <em>nested functions</em> or <em>inner functions</em>. Here’s an example:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">speak</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="k">def</span> <span class="nf">whisper</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'...'</span> <span class="k">return</span> <span class="n">whisper</span><span class="p">(</span><span class="n">text</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">speak</span><span class="p">(</span><span class="s1">'Hello, World'</span><span class="p">)</span> <span class="s1">'hello, world...'</span> </pre></div> <p>Now, what’s going on here? Every time you call <code>speak</code> it defines a new inner function <code>whisper</code> and then calls it.</p> <p>And here’s the kicker—<code>whisper</code> <em>does not exist</em> outside <code>speak</code>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">whisper</span><span class="p">(</span><span class="s1">'Yo'</span><span class="p">)</span> <span class="ne">NameError</span><span class="p">:</span> <span class="s2">"name 'whisper' is not defined"</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">speak</span><span class="o">.</span><span class="n">whisper</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="s2">"'function' object has no attribute 'whisper'"</span> </pre></div> <p>But what if you really wanted to access that nested <code>whisper</code> function from outside <code>speak</code>? Well, functions are objects—you can <em>return</em> the inner function to the caller of the parent function.</p> <p>For example, here’s a function defining two inner functions. Depending on the argument passed to top-level function it selects and returns one of the inner functions to the caller:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">get_speak_func</span><span class="p">(</span><span class="n">volume</span><span class="p">):</span> <span class="k">def</span> <span class="nf">whisper</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="k">return</span> <span class="n">text</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'...'</span> <span class="k">def</span> <span class="nf">yell</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="k">return</span> <span class="n">text</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'!'</span> <span class="k">if</span> <span class="n">volume</span> <span class="o">&gt;</span> <span class="mf">0.5</span><span class="p">:</span> <span class="k">return</span> <span class="n">yell</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="n">whisper</span> </pre></div> <p>Notice how <code>get_speak_func</code> doesn’t actually <em>call</em> one of its inner functions—it simply selects the appropriate function based on the <code>volume</code> argument and then returns the function object:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">get_speak_func</span><span class="p">(</span><span class="mf">0.3</span><span class="p">)</span> <span class="o">&lt;</span><span class="n">function</span> <span class="n">get_speak_func</span><span class="o">.&lt;</span><span class="nb">locals</span><span class="o">&gt;.</span><span class="n">whisper</span> <span class="n">at</span> <span class="mh">0x10ae18</span><span class="o">&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">get_speak_func</span><span class="p">(</span><span class="mf">0.7</span><span class="p">)</span> <span class="o">&lt;</span><span class="n">function</span> <span class="n">get_speak_func</span><span class="o">.&lt;</span><span class="nb">locals</span><span class="o">&gt;.</span><span class="n">yell</span> <span class="n">at</span> <span class="mh">0x1008c8</span><span class="o">&gt;</span> </pre></div> <p>Of course you could then go on and call the returned function, either directly or by assigning it to a variable name first:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">speak_func</span> <span class="o">=</span> <span class="n">get_speak_func</span><span class="p">(</span><span class="mf">0.7</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">speak_func</span><span class="p">(</span><span class="s1">'Hello'</span><span class="p">)</span> <span class="s1">'HELLO!'</span> </pre></div> <p>Let that sink in for a second here… This means not only can functions <em>accept behaviors</em> through arguments but they can also <em>return behaviors</em>. How cool is that?</p> <p>You know what, this is starting to get a little loopy here. I’m going to take a quick coffee break before I continue writing (and I suggest you do the same.)</p> </section><section><h2>Functions Can Capture Local State</h2> <p>You just saw how functions can contain inner functions and that it’s even possible to return these (otherwise hidden) inner functions from the parent function.</p> <p>Best put on your seat belts on now because it’s going to get a little crazier still—we’re about to enter even deeper functional programming territory. (You had that coffee break, right?)</p> <p>Not only can functions return other functions, these inner functions can also <em>capture and carry some of the parent function’s state</em> with them.</p> <p>I’m going to slightly rewrite the previous <code>get_speak_func</code> example to illustrate this. The new version takes a “volume” <em>and</em> a “text” argument right away to make the returned function immediately callable:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">get_speak_func</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">volume</span><span class="p">):</span> <span class="k">def</span> <span class="nf">whisper</span><span class="p">():</span> <span class="k">return</span> <span class="n">text</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'...'</span> <span class="k">def</span> <span class="nf">yell</span><span class="p">():</span> <span class="k">return</span> <span class="n">text</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'!'</span> <span class="k">if</span> <span class="n">volume</span> <span class="o">&gt;</span> <span class="mf">0.5</span><span class="p">:</span> <span class="k">return</span> <span class="n">yell</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="n">whisper</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">get_speak_func</span><span class="p">(</span><span class="s1">'Hello, World'</span><span class="p">,</span> <span class="mf">0.7</span><span class="p">)()</span> <span class="s1">'HELLO, WORLD!'</span> </pre></div> <p>Take a good look at the inner functions <code>whisper</code> and <code>yell</code> now. Notice how they no longer have a <code>text</code> parameter? But somehow they can still access the <code>text</code> parameter defined in the parent function. In fact, they seem to <em>capture</em> and “remember” the value of that argument.</p> <p>Functions that do this are called <a href="https://en.wikipedia.org/wiki/Closure_(computer_programming)" target="_blank"><em>lexical closures</em></a> (or just <em>closures</em>, for short). A closure remembers the values from its enclosing lexical scope even when the program flow is no longer in that scope.</p> <p>In practical terms this means not only can functions <em>return behaviors</em> but they can also <em>pre-configure those behaviors</em>. Here’s another bare-bones example to illustrate this idea:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">make_adder</span><span class="p">(</span><span class="n">n</span><span class="p">):</span> <span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="n">x</span><span class="p">):</span> <span class="k">return</span> <span class="n">x</span> <span class="o">+</span> <span class="n">n</span> <span class="k">return</span> <span class="n">add</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_3</span> <span class="o">=</span> <span class="n">make_adder</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_5</span> <span class="o">=</span> <span class="n">make_adder</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_3</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span> <span class="mi">7</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_5</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span> <span class="mi">9</span> </pre></div> <p>In this example <code>make_adder</code> serves as a <em>factory</em> to create and configure “adder” functions. Notice how the “adder” functions can still access the <code>n</code> argument of the <code>make_adder</code> function (the enclosing scope).</p> </section><section><h2>Objects Can Behave Like Functions</h2> <p>Object’s aren’t functions in Python. But they can be made <em>callable</em>, which allows you to <em>treat them like functions</em> in many cases.</p> <p>If an object is callable it means you can use round parentheses <code>()</code> on it and pass function call arguments to it. Here’s an example of a callable object:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Adder</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">n</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">n</span> <span class="o">=</span> <span class="n">n</span> <span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">n</span> <span class="o">+</span> <span class="n">x</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_3</span> <span class="o">=</span> <span class="n">Adder</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_3</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span> <span class="mi">7</span> </pre></div> <p>Behind the scenes, “calling” an object instance as a function attempts to execute the object’s <code>__call__</code> method.</p> <p>Of course not all objects will be callable. That’s why there’s a built-in <code>callable</code> function to check whether an object appears callable or not:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">callable</span><span class="p">(</span><span class="n">plus_3</span><span class="p">)</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">callable</span><span class="p">(</span><span class="n">yell</span><span class="p">)</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">callable</span><span class="p">(</span><span class="bp">False</span><span class="p">)</span> <span class="bp">False</span> </pre></div> </section><section><h2>Key Takeaways</h2> <ul> <li>Everything in Python is an object, including functions. You can assign them to variables, store them in data structures, and pass or return them to and from other functions (first-class functions.)</li> <li>First-class functions allow you to abstract away and pass around behavior in your programs.</li> <li>Functions can be nested and they can capture and carry some of the parent function’s state with them. Functions that do this are called <em>closures</em>.</li> <li>Objects can be made callable which allows you to treat them like functions in many cases.</li> </ul></section><footer></footer></article>https://dbader.org/blog/python-first-class-functionsTue, 21 Mar 2017 00:00:00 GMTHow to Make Your Python Loops More Pythonichttps://dbader.org/blog/pythonic-loops<article><header><h1>How to Make Your Python Loops More Pythonic</h1> <p>Pythonize your C-style “for” and “while” loops by refactoring them using generators and other techniques.</p> </header><section><figure><img alt="Writing Pythonic Loops" src="/blog/figures/writing-pythonic-loops.png" width="1280" height="720"></figure> <p>One of the easiest ways to spot a developer with a background in C-style languages who only recently picked up Python is to look at how they write loops.</p> <p>For example, whenever I see a code snippet like this, that’s <strong>an example of someone trying to write Python like it’s C or Java</strong>:</p> <div class="codehilite"><pre><span></span><span class="n">my_items</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'a'</span><span class="p">,</span> <span class="s1">'b'</span><span class="p">,</span> <span class="s1">'c'</span><span class="p">]</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">while</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">my_items</span><span class="p">):</span> <span class="k">print</span><span class="p">(</span><span class="n">my_items</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span> </pre></div> </section><section><h2>Now, what’s so “unpythonic” about this code?</h2> <p>Two things could be improved in this code example:</p> <ul> <li>First, it keeps track of the index <code>i</code> manually—initializing it to zero and then carefully incrementing it upon every loop iteration.</li> <li>And second, it uses <code>len()</code> to get the size of a container in order to determine how often to iterate.</li> </ul> <p>In Python you can write loops that handle both of these responsibilities automatically. It’s a great idea to take advantage of that.</p> <p>If your code doesn’t have to keep track of a running index it’s much harder to write accidental infinite loops, for example. It also makes the code more concise and therefore more readable.</p> </section><section><h2>How to track the loop index <em>automatically</em></h2> <p>To refactor the <code>while</code>-loop in the code example, I’ll start by removing the code that manually updates the index. A good way to do that is with a <code>for</code>-loop in Python.</p> <p>Using Python’s <a href="https://docs.python.org/3/library/functions.html#func-range" target="_blank"><code>range()</code> built-in</a> I can generate the indexes automatically (without having to increment a running counter variable):</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">my_items</span><span class="p">))</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">]</span> </pre></div> <p>The <code>range</code> type represents an immutable sequence of numbers. It’s advantage over a regular <code>list</code> is that it always takes the same small amount of memory.</p> <p>Range objects don’t actually store the individual values representing the number sequence—instead they function as iterators and calculate the sequence values on the fly.</p> <p>(This is true for Python 3. In <a href="https://docs.python.org/2/library/functions.html#xrange" target="_blank">Python 2 you’ll need to use the <code>xrange()</code> built-in</a> to get this memory-saving behavior, as <code>range()</code> will construct a list object containing all the values.)</p> <p>Now, instead of incrementing <code>i</code> on each loop iteration, I could write a refactored version of that loop like this:</p> <div class="codehilite"><pre><span></span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">my_items</span><span class="p">)):</span> <span class="k">print</span><span class="p">(</span><span class="n">my_items</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> </pre></div> <p>This is better. However it still isn’t <em>super</em> Pythonic—in most cases when you see code that uses <code>range(len(...))</code> to iterate over a container it can be improved and simplified even further.</p> <p>Let’s have a look at how you might do that in practice.</p> </section><section><h2>💡 Python’s “for” loops are “for-each” loops</h2> <p>As I mentioned, <code>for</code>-loops in Python are really “for-each” loops that can iterate over items from a container or sequence directly, without having to look them up by index. I can take advantage of that and simplify my loop even further:</p> <div class="codehilite"><pre><span></span><span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">my_items</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> </pre></div> <p>I would consider this solution to be quite Pythonic. It’s nice and clean and almost reads like pseudo code from a text book. I don’t have to keep track of the container’s size or a running index to access elements.</p> <p>The container itself takes care of handing out the elements so they can be processed. If the container is <em>ordered</em>, so will be the resulting sequence of elements. If the container is <em>not ordered</em> it will return its elements in an arbitrary order but the loop will still cover all of them.</p> </section><section><h2>What if I <em>need</em> the item index?</h2> <p>Now, of course you won’t always be able to rewrite your loops like that. What if you <em>need</em> the item index, for example? There’s a Pythonic way to keep a running index that avoids the <code>range(len(...))</code> construct I recommended against.</p> <p>The <a href="https://docs.python.org/3/library/functions.html#enumerate" target="_blank"><code>enumerate()</code> built-in</a> is helpful in this case:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">item</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">my_items</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'{i}: {item}'</span><span class="p">)</span> <span class="mi">0</span><span class="p">:</span> <span class="n">a</span> <span class="mi">1</span><span class="p">:</span> <span class="n">b</span> <span class="mi">2</span><span class="p">:</span> <span class="n">c</span> </pre></div> <p>You see, iterators in Python can return more than just one value. They can return tuples with an arbitrary number of values that can then be unpacked right inside the <code>for</code>-statement.</p> <p>This is very powerful. For example, you can use the same technique to iterate over the keys and values of a dictionary at the same time:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">emails</span> <span class="o">=</span> <span class="p">{</span> <span class="o">...</span> <span class="s1">'Bob'</span><span class="p">:</span> <span class="s1">'bob@example.com'</span><span class="p">,</span> <span class="o">...</span> <span class="s1">'Alice'</span><span class="p">:</span> <span class="s1">'alice@example.com'</span><span class="p">,</span> <span class="o">...</span> <span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">email</span> <span class="ow">in</span> <span class="n">emails</span><span class="o">.</span><span class="n">items</span><span class="p">():</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'{name} → {email}'</span><span class="p">)</span> <span class="s1">'Bob → bob@example.com'</span> <span class="s1">'Alice → alice@example.com'</span> </pre></div> </section><section><h2>Okay, what if I just <em>have</em> to write a C-style loop?</h2> <p>There’s one more example I’d like to show you. What if you absolutely, positively need to write a C-style loop. For example, what if you must control the step size for the index? Imagine you had the following original C or Java <code>for</code>-loop:</p> <div class="codehilite"><pre><span></span><span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="n">a</span><span class="o">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">n</span><span class="o">;</span> <span class="n">i</span><span class="o">+=</span><span class="n">s</span><span class="o">)</span> <span class="o">{</span> <span class="c1">// ...</span> <span class="o">}</span> </pre></div> <p>How would this pattern translate to Python?</p> <p>The <code>range()</code> function comes to our rescue again—it can accept extra parameters to control the start value for the loop (<code>a</code>), the stop value (<code>n</code>), and the step size (<code>s</code>).</p> <p>Therefore our example C-style loop could be implemented as follows:</p> <div class="codehilite"><pre><span></span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">s</span><span class="p">):</span> <span class="c1"># ...</span> </pre></div> </section><section><h2>Main Takeaways</h2> <ul> <li>Writing C-style loops in Python is considered unpythonic. Avoid managing loop indexes and stop conditions manually if possible.</li> <li>Python’s <code>for</code>-loops are really “for-each” loops that can iterate over items from a container or sequence directly.</li> </ul> </section><section><h2>📺 Watch a video tutorial based on this article</h2> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/URxQGA9f_AA?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div></section><footer></footer></article>https://dbader.org/blog/pythonic-loopsTue, 14 Mar 2017 00:00:00 GMT5 Python Development Setup Tips to Boost Your Productivityhttps://dbader.org/blog/python-development-setup-tips-to-boost-your-productivity<article><header><h1>5 Python Development Setup Tips to Boost Your Productivity</h1> <p>I struggled with setting up an effective development environment as a new Python developer. It was difficult to build the right habits and to find a set of tools I enjoyed to use.</p> </header><section><figure><img alt="" src="/blog/figures/python-dev-setup-tips.png" width="1280" height="720"></figure> <p>Back then I didn’t understand how much this impacted my productivity. I didn’t even know some of the most valuable practices and tools I’m using today existed!</p> <p>As my experience grew I understood this was a common pain among Python developers. No matter who I spoke with—colleagues, strangers at a conference, or developers on web forums and mailing lists—I saw similar struggles.</p> <p>Today I believe entry-level Python programmers can make leaps in their productivity by adopting a few key practices and tools into their workflow.</p> <p>This article helps you identify and fix 5 common issues in your Python development setup. I experienced them all myself and in some cases helped others through them as a colleague and a team lead. If you can avoid these issues you’ll become a happier and more effective Python developer.</p> </section><section><h2>#1 – Don’t waste time doing the compiler’s job</h2> <p>When developer brains do what computer brains can do much better then that’s usually a costly mistake. One example is programmers spending time hunting bugs that could be spotted just as well by automated tools.</p> <p>For some reason, maybe because of Python’s dynamic nature and earlier status as a “scripting” language, it’s still rare to see it used with static code analysis tools and linters.</p> <p>But these tools are fantastic. They can help detect and avoid certain bugs and classes of errors completely. For example, they can catch functional bugs like misspelled identifiers or reveal code quality issues like unused variables and imports.</p> <p>I won’t say code analysis tools are a miracle cure—but they can help reduce debugging and code review time with a small initial time investment.</p> <p>If you’re looking for just one tool that will improve the quality of your Python code without getting in the way with false positives and verbose messages, then I’d recommend the <a href="https://pypi.python.org/pypi/pyflakes" target="_blank">Pyflakes</a> code linter. Pyflakes is open-source, available for free, and easy to set up.</p> <p>To get immediate feedback and catch bugs early I recommend you integrate Pyflakes with your code editor and build server. Automatic linting for code changes as part of your continuous integration process makes your life easier. It ensures all developers on your team use the same settings and no uncaught warnings slip through the cracks.</p> <p><strong>Tip 1: Use static code analysis tools like Pyflakes.</strong></p> </section><section><h2>#2 – Avoid fruitless code style discussions</h2> <p>Your team does code reviews? Great! But be aware that a common mistake among inexperienced code reviewers is to spend too much time on feedback that automated tools could give for them. I’m talking about code style issues.</p> <p>It’s easy for development teams to get into a habit where they mostly talk about code style issues in code reviews: “We need an extra space character here.” or “Class names should use camel case.”</p> <p>This is a form of bikeshedding that prevents developers from looking at the real issues. The ones that cost money and cause maintenance problems later on.</p> <p>A quick fix here is to pick one of the Python style guides available on the internet, like <a href="https://www.python.org/dev/peps/pep-0008/" target="_blank">PEP 8</a> or <a href="https://google.github.io/styleguide/pyguide.html" target="_blank">Google’s Python Style Guide</a>, and to put automated tools in place that make sure committed code follows the style guide.</p> <p>I recommend using <a href="https://www.python.org/dev/peps/pep-0008/" target="_blank">PEP 8</a> as a style guide in combination with the <a href="https://github.com/PyCQA/pycodestyle" target="_blank">Pycodestyle</a> or <a href="https://pypi.python.org/pypi/flake8" target="_blank">flake8</a> code style checker. This will help avoid most code style discussions and allows your team to focus on the issues that matter.</p> <p><strong>Tip 2: Pick a code style (PEP 8) and enforce it with automated tools.</strong></p> </section><section><h2>#3 – Micro delays and death by a thousand cuts</h2> <p>Usability research shows the large effect website page load time has on <a href="https://www.nngroup.com/articles/website-response-times/" target="_blank">user abandonment</a>: If people get bored waiting for something to happen it increases the chances that they’ll abandon the original task they had in mind.</p> <p>As software developers, waiting on tools to complete their job is a normal part of our day to day workflow. We’re always waiting for a module to install, a test to run, or a commit to finish (<a href="https://xkcd.com/303/" target="_blank">“It’s compiling!”</a>). Of course we’re not “abandoning” our work every time we have to wait a few seconds for a tool to run—keeping focused on the task at hand is part of our job after all.</p> <p>Yet, keeping that focus costs mental energy that we might then lack in other areas of our work: We get tired a little quicker in the afternoon, or introduce a tiny little extra bug with our latest commit.</p> <p>In my experience even small forced pauses and delays add up. Switching files in a slow editor or jumping between apps on a slow computer is frustrating! We can even apply this at a microscopic level and look into <a href="https://pavelfatin.com/typing-with-pleasure/" target="_blank">editor typing latencies</a>. I believe these micro delays add up, too. They cost us productivity and cause frustration.</p> <p>Got time for a little thought experiment? Let’s say you’re waiting for a task to complete for about 1 out of every 10 seconds you spend on productive work. That adds up to half a day per week, or 2 days a month, or a whole month of productive work you might be losing over the course of a year.</p> <p>Maybe this estimate is too high—but what if you could get an additional week of productive time a year just by spending an afternoon on optimizing your tools? I’d say that’s worth a try!</p> <p><strong>Tip 3: Your development tools should be fast. Favor simplicity.</strong></p> </section><section><h2>#4 – Don’t work with an unpleasant editing environment</h2> <p>Working with tools that I don’t enjoy crushes my productivity. You might know the feeling. Some tools are so frustrating to work with they zap your energy levels and motivation.</p> <p>What’s the most important tool that you work with every day as a developer? For me it’s my code editor. For some developers it might be their email client or a team chat app—but let’s hope that a large part of your day is spent writing code.</p> <p>This means it pays off in terms of productivity (and happiness!) to invest into an enjoyable code editing environment.</p> <p>As Python developers we have many editors and IDEs to choose from: <a href="http://www.vim.org/" target="_blank">Vim</a>, <a href="https://www.gnu.org/software/emacs/" target="_blank">Emacs</a>, <a href="https://www.jetbrains.com/pycharm/" target="_blank">PyCharm</a>, <a href="https://wingware.com/" target="_blank">Wing IDE</a>, <a href="https://atom.io/" target="_blank">Atom</a>, <a href="http://www.pydev.org/" target="_blank">Eclipse PyDev</a>, <a href="https://www.sublimetext.com/" target="_blank">Sublime Text</a>—just to name a few.</p> <p>I spent much time fine-tuning my editing environment over the years. After trying other editors and IDEs I <a href="https://dbader.org/blog/setting-up-sublime-text-for-python-development">eventually settled on Sublime Text</a>. I like its speed, simplicity, and stability. It just feels right for my programming workflow. And I arrived at this choice by trying as many other options as I could.</p> <p>Your choice might be different. The point I’m trying to make is you need to find out which tool works best for yourself and your unique needs. Go and try out some editors and see which one you enjoy the most. Your productivity will thank you for it.</p> <p><strong>Tip 4: Find the right editor and tailor it to your needs.</strong></p> </section><section><h2>#5 – Invest in your setup</h2> <p>I once worked with someone who used a commercial editor to write code. But that developer didn’t want to spend the money to purchase a license for it. Instead they used the trial version of the editor for months on end.</p> <p>The trial version of this particular editor has a nag screen that pops up every few minutes when you save a file, asking you to buy the full version. This developer constantly saved files out of habit and therefore got to see that nag screen hundreds of times a day…</p> <p>A license that would’ve removed the nag screen cost about $70. I love a frugal mindset but this was ridiculous! Trying to save some money on a critical tool you use all day was the wrong choice—I’m sure the nag screens and the subtle frustrations they caused added up to more than $70 of lost productivity.</p> <p>If you’re working for yourself then these license costs will be a business expense you can deduct from your taxes. If you’re working for a company I’m sure they’ll gladly invest in your tools if you explain how they make you more productive and more valuable as an employee.</p> <p>License costs for software development tools are low compared to what graphic designers or architects have to put up with, for example. Some of the best tools and editors are even available for free. Invest money in the right tools where it makes sense and your life (and career) will be better for it.</p> <p><strong>Tip 5: Invest in tools that make you happy and more effective.</strong></p> </section><section><h2>Where to start?</h2> <p>I showed you five common development setup issues that can harm your productivity as a Python programmer. Luckily most of them are easy to fix with the right approach:</p> <ul> <li><strong>Tip 1: Use static code analysis tools like Pyflakes.</strong></li> <li><strong>Tip 2: Pick a code style (PEP 8) and enforce it with automated tools.</strong></li> <li><strong>Tip 3: Your development tools should be fast. Favor simplicity.</strong></li> <li><strong>Tip 4: Find the right editor and tailor it to your needs.</strong></li> <li><strong>Tip 5: Invest in tools that make you happy and more effective.</strong></li> </ul> <p>Here’s a good way to start: Find the one problem that irritates you the most. You’ll want to <em>divide and conquer</em> instead of trying to achieve perfection immediately. Fix one small thing at a time. Then iterate and keep making improvements from there.</p> <p>Think of it as an investment—even small changes will compound over time and give you a nice long-term productivity gain. In my experience, success is all about building the right habits and a mindset of continuous improvement.</p> <p>A great development environment makes you feel confident and productive. When you feel right at home in your setup programming Python becomes even more enjoyable and fun. Good luck!</p> <p><em>(This article was originally published on <a href="https://techbeacon.com/productive-python-development-how-tackle-5-common-issues" target="_blank">TechBeacon</a>.)</em></p></section><footer></footer></article>https://dbader.org/blog/python-development-setup-tips-to-boost-your-productivityTue, 07 Mar 2017 00:00:00 GMTContext Managers and the “with” Statement in Pythonhttps://dbader.org/blog/python-context-managers-and-with-statement<article><header><h1>Context Managers and the “with” Statement in Python</h1> <p>The “with” statement in Python is regarded as an obscure feature by some. But when you peek behind the scenes of the underlying Context Manager protocol you’ll see there’s little “magic” involved.</p> </header><section><div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/iba-I4CrmyA?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p>So what’s the <code>with</code> statement good for? It helps simplify some common resource management patterns by abstracting their functionality and allowing them to be factored out and reused.</p> <p>In turn this helps you write more expressive code and makes it easier to avoid resource leaks in your programs.</p> <p>A good way to see this feature used effectively is by looking at examples in the Python standard library. A well-known example involves the <code>open()</code> function:</p> <div class="codehilite"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'hello.txt'</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'hello, world!'</span><span class="p">)</span> </pre></div> <p>Opening files using the <code>with</code> statement is generally recommended because it ensures that open file descriptors are closed automatically after program execution leaves the context of the <code>with</code> statement. Internally, the above code sample translates to something like this:</p> <div class="codehilite"><pre><span></span><span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'hello.txt'</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">try</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'hello, world'</span><span class="p">)</span> <span class="k">finally</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> </pre></div> <p>You can already tell that this is quite a bit more verbose. Note that the <code>try...finally</code> statement is significant. It wouldn’t be enough to just write something like this:</p> <div class="codehilite"><pre><span></span><span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'hello.txt'</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'hello, world'</span><span class="p">)</span> <span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> </pre></div> <p>This implementation won’t guarantee the file is closed if there’s an exception during the <code>f.write()</code> call—and therefore our program might leak a file descriptor. That’s why the <code>with</code> statement is so useful. It makes acquiring and releasing resources <em>properly</em> a breeze.</p> <p>Another good example where the <code>with</code> statement is used effectively in the Python standard library is the <code>threading.Lock</code> class:</p> <div class="codehilite"><pre><span></span><span class="n">some_lock</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Lock</span><span class="p">()</span> <span class="c1"># Harmful:</span> <span class="n">some_lock</span><span class="o">.</span><span class="n">acquire</span><span class="p">()</span> <span class="k">try</span><span class="p">:</span> <span class="c1"># Do something...</span> <span class="k">finally</span><span class="p">:</span> <span class="n">some_lock</span><span class="o">.</span><span class="n">release</span><span class="p">()</span> <span class="c1"># Better:</span> <span class="k">with</span> <span class="n">some_lock</span><span class="p">:</span> <span class="c1"># Do something...</span> </pre></div> <p>In both cases using a <code>with</code> statement allows you to abstract away most of the resource handling logic. Instead of having to write an explicit <code>try...finally</code> statement each time, <code>with</code> takes care of that for us.</p> <p>The <code>with</code> statement can make code dealing with system resources more readable. It also helps avoid bugs or leaks by making it almost impossible to forget cleaning up or releasing a resource after we’re done with it.</p> </section><section><h2>Supporting <code>with</code> in Your Own Objects</h2> <p>Now, there’s nothing special or magical about the <code>open()</code> function or the <code>threading.Lock</code> class and the fact that they can be used with a <code>with</code> statement. You can provide the same functionality in your own classes and functions by implementing so-called <a href="https://docs.python.org/3/reference/datamodel.html#context-managers" target="_blank">context managers</a>.</p> <p>What’s a context manager? It’s a simple “protocol” (or interface) that your object needs to follow so it can be used with the <code>with</code> statement. Basically all you need to do is add <code>__enter__</code> and <code>__exit__</code> methods to an object if you want it to function as a context manager. Python will call these two methods at the appropriate times in the resource management cycle.</p> <p>Let’s take a look at what this would look like in practical terms. Here’s how a simple implementation of the <code>open()</code> context manager might look like:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">ManagedFile</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span> <span class="k">def</span> <span class="fm">__enter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">file</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">file</span> <span class="k">def</span> <span class="fm">__exit__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">exc_type</span><span class="p">,</span> <span class="n">exc_val</span><span class="p">,</span> <span class="n">exc_tb</span><span class="p">):</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">file</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">file</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> </pre></div> <p>Our <code>ManagedFile</code> class follows the context manager protocol and now supports the <code>with</code> statement, just like the original <code>open()</code> example did:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">with</span> <span class="n">ManagedFile</span><span class="p">(</span><span class="s1">'hello.txt'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="o">...</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'hello, world!'</span><span class="p">)</span> <span class="o">...</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'bye now'</span><span class="p">)</span> </pre></div> <p>Python calls <code>__enter__</code> when execution <em>enters</em> the context of the <code>with</code> statement and it’s time to acquire the resource. When execution <em>leaves</em> the context again, Python calls <code>__exit__</code> to free up the resource.</p> <p>Writing a class-based context manager isn’t the only way to support the <code>with</code> statement in Python. The <a href="https://docs.python.org/3/library/contextlib.html" target="_blank"><code>contextlib</code> utility module in the standard library</a> provides a few more abstractions built on top of the basic context manager protocol. This can make your life a little easier if your use cases matches what’s offered by <code>contextlib</code>.</p> <p>For example, you can use the <code>contextlib.contextmanager</code> decorator to define a generator-based <em>factory function</em> for a resource that will then automatically support the <code>with</code> statement. Here’s what rewriting our <code>ManagedFile</code> context manager with this technique looks like:</p> <div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">contextlib</span> <span class="kn">import</span> <span class="n">contextmanager</span> <span class="nd">@contextmanager</span> <span class="k">def</span> <span class="nf">managed_file</span><span class="p">(</span><span class="n">name</span><span class="p">):</span> <span class="k">try</span><span class="p">:</span> <span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">yield</span> <span class="n">f</span> <span class="k">finally</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">with</span> <span class="n">managed_file</span><span class="p">(</span><span class="s1">'hello.txt'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="o">...</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'hello, world!'</span><span class="p">)</span> <span class="o">...</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'bye now'</span><span class="p">)</span> </pre></div> <p>In this case, <code>managed_file()</code> is a generator that first acquires the resource. Then it temporarily suspends its own executing and <em>yields</em> the resource so it can be used by the caller. When the caller leaves the <code>with</code> context, the generator continues to execute so that any remaining clean up steps can happen and the resource gets released back to the system.</p> <p>Both the class-based implementations and the generator-based are practically equivalent. Depending on which one you find more readable you might prefer one over the other.</p> <p>A downside of the <code>@contextmanager</code>-based implementation might be that it requires understanding of advanced Python concepts, like decorators and generators.</p> <p>Once again, making the right choice here comes down to what you and your team are comfortable using and find the most readable.</p> </section><section><h2>Writing Pretty APIs With Context Managers</h2> <p>Context managers are quite flexible and if you use the <code>with</code> statement creatively you can define convenient APIs for your modules and classes.</p> <p>For example, what if the “resource” we wanted to manage was text indentation levels in some kind of report generator program? What if we could write code like this to do it:</p> <div class="codehilite"><pre><span></span><span class="k">with</span> <span class="n">Indenter</span><span class="p">()</span> <span class="k">as</span> <span class="n">indent</span><span class="p">:</span> <span class="n">indent</span><span class="o">.</span><span class="k">print</span><span class="p">(</span><span class="s1">'hi!'</span><span class="p">)</span> <span class="k">with</span> <span class="n">indent</span><span class="p">:</span> <span class="n">indent</span><span class="o">.</span><span class="k">print</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span> <span class="k">with</span> <span class="n">indent</span><span class="p">:</span> <span class="n">indent</span><span class="o">.</span><span class="k">print</span><span class="p">(</span><span class="s1">'bonjour'</span><span class="p">)</span> <span class="n">indent</span><span class="o">.</span><span class="k">print</span><span class="p">(</span><span class="s1">'hey'</span><span class="p">)</span> </pre></div> <p>This almost reads like a domain-specific language (DSL) for indenting text. Also, notice how this code enters and leaves the same context manager multiple times to change indentation levels. Running this code snippet should lead to the following output and print neatly formatted text:</p> <div class="codehilite"><pre><span></span>hi! hello bonjour hey </pre></div> <p>How would you implement a context manager to support this functionality?</p> <p>By the way, this could be a great exercise to wrap your head around how context managers work. So before you check out my implementation below you might take some time and try to implement this yourself as a learning exercise.</p> <p>Ready? Here’s how we might implement this functionality using a class-based context manager:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Indenter</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">level</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">def</span> <span class="fm">__enter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">level</span> <span class="o">+=</span> <span class="mi">1</span> <span class="k">return</span> <span class="bp">self</span> <span class="k">def</span> <span class="fm">__exit__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">exc_type</span><span class="p">,</span> <span class="n">exc_val</span><span class="p">,</span> <span class="n">exc_tb</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">level</span> <span class="o">-=</span> <span class="mi">1</span> <span class="k">def</span> <span class="nf">print</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">text</span><span class="p">):</span> <span class="k">print</span><span class="p">(</span><span class="s1">' '</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">level</span> <span class="o">+</span> <span class="n">text</span><span class="p">)</span> </pre></div> <p>Another good exercise would be trying to refactor this code to be generator-based.</p> </section><section><h2>Things to Remember</h2> <ul> <li>The <code>with</code> statement simplifies exception handling by encapsulating standard uses of <code>try/finally</code> statements in so-called Context Managers.</li> <li>Most commonly it is used to manage the safe acquisition and release of system resources. Resources are acquired by the <code>with</code> statement and released automatically when execution leaves the <code>with</code> context.</li> <li>Using <code>with</code> effectively can help you avoid resource leaks and make your code easier to read.</li> </ul></section><footer></footer></article>https://dbader.org/blog/python-context-managers-and-with-statementTue, 28 Feb 2017 00:00:00 GMTInstalling Python and Pip on Windowshttps://dbader.org/blog/installing-python-and-pip-on-windows-10<article><header><h1>Installing Python and Pip on Windows</h1> <p>In this tutorial you’ll learn how to set up Python and the Pip package manager on Windows 10, completely from scratch.</p> </header><section><figure><img alt="" src="/blog/figures/python-windows-setup.png" width="1280" height="720"></figure> </section><section><h2>Step 1: Download the Python Installer</h2> <p>The best way to install Python on Windows is by downloading the official Python installer from the Python website at <a href="https://python.org" target="_blank">python.org</a>.</p> <p>To do so, open a browser and navigate to <a href="https://python.org/" target="_blank">https://python.org/</a>. After the page has finished loading, click <strong>Downloads.</strong></p> <ul> <li>The website should detect that you’re on Windows and offer you to download the latest version of Python 3 or Python 2. If you don’t know which version of Python to use then I recommend <strong>Python 3</strong>. If you know you’ll need to work with legacy Python 2 code <em>only then</em> should you pick Python 2.</li> </ul> <p>Under <strong>Downloads → Download for Windows</strong>, click the <strong>“Python 3.X.X”</strong> (or “Python 2.X.X”) button to begin downloading the installer.</p> <figure><img alt="" src="/blog/figures/windows-setup-download-the-python-installer.jpg" width="1212" height="942"></figure> <div class="update-box"> <h2>Sidebar: 64-bit Python vs 32-bit Python</h2> <p>If you’re wondering whether you should use a 32-bit or a 64-bit version of Python then you might want to go with the 32-bit version.</p> <p>It’s sometimes still problematic to find binary extensions for 64-bit Python on Windows, which means that some third-party modules might not install correctly with a 64-bit version of Python.</p> <p>My thinking is that it’s best to go with the version currently recommended on python.org. If you click the Python 3 or Python 2 button under “Download for Windows” you’ll get just that.</p> <p>Remember that if you get this choice wrong and you’d like to switch to another version of Python you can just uninstall Python and then re-install it by downloading another installer from <a href="https://python.org" target="_blank">python.org</a>.</p> </div> </section><section><h2>Step 2: Run the Python Installer</h2> <p>Once the Python installer file has finished downloading, <strong>launch it</strong> by double-clicking on it in order to begin the installation.</p> <p>Be sure to select the <strong>Add Python X.Y to PATH</strong> checkbox in the setup wizard.</p> <ul> <li>Please make sure the “Add Python X.Y to PATH” checkbox was enabled in the installer because otherwise you will have problems accessing your Python installation from the command line. If you accidentally installed Python without checking the box, <a href="http://www.anthonydebarros.com/2015/08/16/setting-up-python-in-windows-10/" target="_blank">follow this tutorial</a> to add <code>python.exe</code> to your system PATH.</li> </ul> <p>Click <strong>Install Now</strong> to begin the installation process. The installation should finish quickly and then Python will be ready to go on your system. We’re going to make sure everything was set up correctly in the next step.</p> <figure><img alt="" src="/blog/figures/windows-setup-run-the-python-installer.jpg" width="684" height="427"></figure> </section><section><h2>Step 3: Verify Python Was Installed Correctly</h2> <p>After the Python installer finished its work Python should be installed on your system. Let’s make sure everything went correctly by testing if Python can be accessed from the Windows Command Prompt:</p> <ol> <li>Open the Windows Command Prompt by launching <strong>cmd.exe</strong></li> <li>Type <strong>pip</strong> and hit <strong>Return</strong></li> <li>You should see the help text from Python’s “pip” package manager. If you get an error message running <strong>pip</strong> go through the Python install steps again to make sure you have a working Python installation. Most issues you will encounter here will have something to do with the PATH not being set correctly. Re-installing and making sure that the “Add Python to PATH” option is enabled in the installer should resolve this.</li> </ol> <figure><img alt="" src="/blog/figures/windows-setup-verify-python-was-installed-correctly.jpg" width="1351" height="1028"></figure> </section><section><h2>What Now?</h2> <p>Assuming everything went well and you saw the output from Pip in your command prompt window—Congratulations, you just installed Python on your system!</p> <p>Wondering where to go from here? <a href="/python-basics">Click here to get some pointers</a> for Python beginners.</p></section><footer></footer></article>https://dbader.org/blog/installing-python-and-pip-on-windows-10Sat, 25 Feb 2017 00:00:00 GMTSublime Text Settings for Writing Clean Pythonhttps://dbader.org/blog/sublime-text-settings-for-writing-clean-python<article><header><h1>Sublime Text Settings for Writing Clean Python</h1> <p>How to write beautiful and clean Python by tweaking your Sublime Text settings so that they make it easier to adhere to the PEP 8 style guide recommendations.</p> </header><section><figure><img alt="" src="/blog/figures/sublime-settings-pep8-python.png" width="1280" height="720"></figure> <p>There are a few settings you can change to make it easier for you to write PEP 8 compliant Python with Sublime Text 3. <a href="https://www.python.org/dev/peps/pep-0008/" target="_blank">PEP 8 is the most common Python style guide</a> and widely used in the Python community.</p> <p>The tweaks I describe in this article mainly deal with getting the placement of whitespace correct so that you don’t have to manage this (boring) aspect yourself.</p> <p>I’ll also show you how to get visual indicators for the maximum allowed line-lengths in your editor window so that your lines can be concise and beautifully PEP 8 compliant—just like Guido wants them to be 🙂</p> </section><section><h2>Optional: Opening Sublime’s Syntax-Specific Settings for Python</h2> <p>The settings we’re changing now are specific to Python. Feel free to place them in your User settings, that will work just fine. However if you’d like to apply some or all of the settings in this chapter only to Python code then here’s how you can do that:</p> <ol> <li>Open a Python file in Sublime Text (or create a new file, open the Command Palette and execute the “Set Syntax: Python” command)</li> <li>Click on <strong>Sublime Text → Preferences → Settings – More → Syntax Specific – User</strong> to open your Python-specific user settings. Make sure this opens a new editor tab called <strong>Python.sublime-settings</strong>. That’s the one you want!</li> </ol> <figure><img alt="" src="/blog/figures/sublime-pep8-syntax-specific-settings.png" width="926" height="564"></figure> <p>If you’d like like to learn more about how Sublime Text’s preferences system works, then <a href="sublime-text-preferences-tutorial">check out this tutorial I wrote</a>.</p> </section><section><h2>Better Whitespace Handling</h2> <p>The following changes you can make to your (<strong>Syntax Specific</strong>) <strong>User Settings</strong> will help you keep the whitespace in your Python code clean and consistent:</p> <div class="codehilite"><pre><span></span><span class="s2">"tab_size"</span><span class="o">:</span> <span class="mi">4</span><span class="p">,</span> <span class="s2">"translate_tabs_to_spaces"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="s2">"trim_trailing_white_space_on_save"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="s2">"ensure_newline_at_eof_on_save"</span><span class="o">:</span> <span class="kc">true</span> </pre></div> <p>A <strong>tab_size</strong> of <strong>4</strong> is the general recommendation for writing Python. You’ll also want to enable <strong>translate_tabs_to_spaces</strong> to ensure that you don’t have a mixture of tabs and spaces in your Python files, which should be avoided.</p> <p>The <strong>trim_trailing_white_space_on_save</strong> option will remove superfluous whitespace at the end of lines or on empty lines. I highly recommend enabling this because it can save headaches and merge conflicts when working with Git and other forms of source control.</p> <p>PEP 8 recommends that Python files should end with a blank line to ensure that <a href="https://stackoverflow.com/questions/729692/why-should-text-files-end-with-a-newline" target="_blank">POSIX tools can process the file correctly</a>. If you want to never have to worry about this again then turn on the <strong>ensure_newline_at_eof_on_save</strong> setting as this will make sure that your Python files end with a newline automatically.</p> </section><section><h2>Enable PEP 8 Line-Length Indicators</h2> <p>Another setting that’s really handy for writing PEP 8 compliant code is the “rulers” feature. It enables visual indicators in the editor area that show you the preferred maximum line length.</p> <p>You can enable several rulers with different line lengths at the same time. This helps you follow the PEP 8 recommendations of limiting your docstrings to 72 characters and limiting all other lines to 79 characters.</p> <p>Here’s how to set up the rulers feature for Python development. Open your (<strong>Syntax Specific</strong>) <strong>User Settings</strong> and add the following setting:</p> <div class="codehilite"><pre><span></span><span class="s2">"rulers"</span><span class="o">:</span> <span class="p">[</span> <span class="mi">72</span><span class="p">,</span> <span class="mi">79</span> <span class="p">]</span> </pre></div> <p>This will add two line-length indicators—one at 72 characters for docstrings, and one at 79 characters for regular lines. You can see them in the screenshot as vertical lines on the right-hand side of the editor area.</p> <figure><img alt="" src="/blog/figures/sublime-pep8-line-length-indicators.png" width="992" height="488"></figure> </section><section><h2>Turn On Word Wrapping</h2> <p>I like enabling Sublime’s word-wrapping feature when I’m writing Python. Most of my projects follow the PEP 8 style guide and therefore use a maximum line length of 79 characters.</p> <p>I don’t want to get into an argument why that’s a good idea or not—but one benefit I found from limiting the lengths of my lines is that I can comfortably fit several files on my screen at once using Sublime’s “split layouts” feature.</p> <p>This is especially useful if you’re following a test-heavy development process because you can see and edit the test and the production code at the same time.</p> <p>Of course sometimes you’ll encounter a file that uses line-lengths above the 79 characters recommended by PEP 8. If I’m using split layouts with multiple editor panes at the same time it impacts my productivity if I have to scroll around horizontally.</p> <p>The idea is to see <em>all of the code</em> at once. So, how can we fix that?</p> <p>The best way I found to handle this is to enable Sublime’s word-wrap feature. This will visually break apart lines that are longer than the maximum line length. It might look a little odd sometimes but it’s still light years better than having to scroll around horizontally.</p> <p>Here’s how you enable word wrapping. Open your (<strong>Syntax Specific</strong>) <strong>User Settings</strong> and add (or modify) the following options:</p> <div class="codehilite"><pre><span></span><span class="s2">"word_wrap"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="s2">"wrap_width"</span><span class="o">:</span> <span class="mi">80</span> </pre></div> <p>I’m setting the <strong>wrap_width</strong> to 80 which is one character past the 79 characters recommended by PEP 8. Therefore any line that goes beyond the PEP 8 recommendations will get wrapped.</p> <figure><img alt="" src="/blog/figures/sublime-pep8-word-wrapping.png" width="964" height="308"></figure></section><footer></footer></article>https://dbader.org/blog/sublime-text-settings-for-writing-clean-pythonTue, 21 Feb 2017 00:00:00 GMTWriting Clean Python With Namedtupleshttps://dbader.org/blog/writing-clean-python-with-namedtuples<article><header><h1>Writing Clean Python With Namedtuples</h1> <p>Python comes with a specialized “namedtuple” container type that doesn’t seem to get the attention it deserves. It’s one of these amazing features in Python that’s hidden in plain sight.</p> </header><section><figure><img alt="" src="/blog/figures/namedtuples.png" width="1280" height="720"></figure> <p>Namedtuples can be a great alternative to defining a class manually and they have some other interesting features that I want to introduce you to in this article.</p> <p>Now, what’s a namedtuple and what makes it so special? A good way to think about namedtuples is to view them as an extension of the built-in tuple data type.</p> <p>Python’s tuples are a simple data structure for grouping arbitrary objects. Tuples are also immutable—they cannot be modified once they’ve been created.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">tup</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'hello'</span><span class="p">,</span> <span class="nb">object</span><span class="p">(),</span> <span class="mi">42</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">tup</span> <span class="p">(</span><span class="s1">'hello'</span><span class="p">,</span> <span class="o">&lt;</span><span class="nb">object</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x105e76b70</span><span class="o">&gt;</span><span class="p">,</span> <span class="mi">42</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">tup</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="mi">42</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">tup</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="mi">23</span> <span class="ne">TypeError</span><span class="p">:</span> <span class="s2">"'tuple' object does not support item assignment"</span> </pre></div> <p>A downside of plain tuples is that the data you store in them can only be pulled out by accessing it through integer indexes. You can’t give names to individual properties stored in a tuple. This can impact code readability.</p> <p>Also, a tuple is always an ad-hoc structure. It’s hard to ensure that two tuples have the same number of fields and the same properties stored on them. This makes it easy to introduce “slip-of-the-mind” bugs by mixing up the field order.</p> </section><footer><h3>Namedtuples to the Rescue</h3> <p>Namedtuples aim to solve these two problems.</p> <p>First of all, namedtuples are immutable just like regular tuples. Once you store something in them you can’t modify it.</p> <p>Besides that, namedtuples are, well…<em>named tuples</em>. Each object stored in them can be accessed through a unique (human-readable) identifier. This frees you from having to remember integer indexes, or resorting to workarounds like defining integer constants as mnemonics for your indexes.</p> <p>Here’s what a namedtuple looks like:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">namedtuple</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">Car</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span><span class="s1">'Car'</span> <span class="p">,</span> <span class="s1">'color mileage'</span><span class="p">)</span> </pre></div> <p>To use namedtuples you need to import the <code>collections</code> module. They were added to the standard library in Python 2.6. In the above example we defined a simple “Car” data type with two fields: “color” and “mileage”.</p> <p>You might find the syntax a little weird here—Why are we passing the fields as a string encoding them <code>"color mileage"</code>?</p> <p>The answer is that namedtuple’s factory function calls <code>split()</code> on the field names string, so this is really just a shorthand to say the following:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="s1">'color mileage'</span><span class="o">.</span><span class="n">split</span><span class="p">()</span> <span class="p">[</span><span class="s1">'color'</span><span class="p">,</span> <span class="s1">'mileage'</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">Car</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span><span class="s1">'Car'</span><span class="p">,</span> <span class="p">[</span><span class="s1">'color'</span><span class="p">,</span> <span class="s1">'mileage'</span><span class="p">])</span> </pre></div> <p>Of course you can also pass a list with string field names directly if you prefer how that looks. The advantage of using a proper list is that it’s easier to reformat this code if you need to split it across multiple lines:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">Car</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span><span class="s1">'Car'</span><span class="p">,</span> <span class="p">[</span> <span class="o">...</span> <span class="s1">'color'</span><span class="p">,</span> <span class="o">...</span> <span class="s1">'mileage'</span><span class="p">,</span> <span class="o">...</span> <span class="p">])</span> </pre></div> <p>However you decide, you can now create new “car” objects with the <code>Car</code> factory function. It behaves as if you had defined a <code>Car</code> class manually and given it a constructor accepting a “color” and a “mileage” value:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span> <span class="o">=</span> <span class="n">Car</span><span class="p">(</span><span class="s1">'red'</span><span class="p">,</span> <span class="mf">3812.4</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span><span class="o">.</span><span class="n">color</span> <span class="s1">'red'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span><span class="o">.</span><span class="n">mileage</span> <span class="mf">3812.4</span> </pre></div> <p>Tuple unpacking and the <a href="https://www.youtube.com/watch?v=YWY4BZi_o28" target="_blank"><code>*</code>-operator for function argument unpacking</a> also work as expected:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">color</span><span class="p">,</span> <span class="n">mileage</span> <span class="o">=</span> <span class="n">my_car</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="n">color</span><span class="p">,</span> <span class="n">mileage</span><span class="p">)</span> <span class="n">red</span> <span class="mf">3812.4</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="o">*</span><span class="n">my_car</span><span class="p">)</span> <span class="n">red</span> <span class="mf">3812.4</span> </pre></div> <p>Besides accessing the values stored in a namedtuple by their identifiers, you can still access them by their index. That way namedtuples can be used as a drop-in replacement for regular tuples:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="s1">'red'</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">my_car</span><span class="p">)</span> <span class="p">(</span><span class="s1">'red'</span><span class="p">,</span> <span class="mf">3812.4</span><span class="p">)</span> </pre></div> <p>You’ll even get a nice string representation for free, which saves some typing and redundancy:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span> <span class="n">Car</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="s1">'red'</span> <span class="p">,</span> <span class="n">mileage</span><span class="o">=</span><span class="mf">3812.4</span><span class="p">)</span> </pre></div> <p>Like tuples, namedtuples are immutable. When you try to overwrite one of their fields you’ll get an <code>AttributeError</code> exception:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span><span class="o">.</span><span class="n">color</span> <span class="o">=</span> <span class="s1">'blue'</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="s2">"can't set attribute"</span> </pre></div> <p>Namedtuple objects are implemented as regular Python classes internally. When it comes to memory usage they are also “better” than regular classes and just as memory efficient as regular tuples.</p> <p>A good way to view them is to think that <em>namedtuples are a memory-efficient shortcut to defining an immutable class in Python manually</em>.</p> <h3>Subclassing Namedtuples</h3> <p>Because they are built on top of regular classes you can even add methods to a namedtuple’s class. For example, you can extend the class like any other class and add methods and new properties to it that way. Here’s an example:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">Car</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span><span class="s1">'Car'</span><span class="p">,</span> <span class="s1">'color mileage'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">class</span> <span class="nc">MyCarWithMethods</span><span class="p">(</span><span class="n">Car</span><span class="p">):</span> <span class="o">...</span> <span class="k">def</span> <span class="nf">hexcolor</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="o">...</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">color</span> <span class="o">==</span> <span class="s1">'red'</span><span class="p">:</span> <span class="o">...</span> <span class="k">return</span> <span class="s1">'#ff0000'</span> <span class="o">...</span> <span class="k">else</span><span class="p">:</span> <span class="o">...</span> <span class="k">return</span> <span class="s1">'#000000'</span> </pre></div> <p>We can now create <code>MyCarWithMethods</code> objects and call their <code>hexcolor()</code> method, just as expected:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">c</span> <span class="o">=</span> <span class="n">MyCarWithMethods</span><span class="p">(</span><span class="s1">'red'</span><span class="p">,</span> <span class="mi">1234</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">c</span><span class="o">.</span><span class="n">hexcolor</span><span class="p">()</span> <span class="s1">'#ff0000'</span> </pre></div> <p>However, this might be a little clunky. It might be worth doing if you want a class with immutable properties. But it’s also easy to shoot yourself in the foot here.</p> <p>For example, adding a new <em>immutable</em> field is tricky because of how namedtuples are structured internally. The easiest way to create hierarchies of namedtuples is to use the base tuple’s <code>._fields</code> property:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">Car</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span><span class="s1">'Car'</span><span class="p">,</span> <span class="s1">'color mileage'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ElectricCar</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span> <span class="o">...</span> <span class="s1">'ElectricCar'</span><span class="p">,</span> <span class="n">Car</span><span class="o">.</span><span class="n">_fields</span> <span class="o">+</span> <span class="p">(</span><span class="s1">'charge'</span><span class="p">,))</span> </pre></div> <p>This gives the desired result:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">ElectricCar</span><span class="p">(</span><span class="s1">'red'</span><span class="p">,</span> <span class="mi">1234</span><span class="p">,</span> <span class="mf">45.0</span><span class="p">)</span> <span class="n">ElectricCar</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="s1">'red'</span><span class="p">,</span> <span class="n">mileage</span><span class="o">=</span><span class="mi">1234</span><span class="p">,</span> <span class="n">charge</span><span class="o">=</span><span class="mf">45.0</span><span class="p">)</span> </pre></div> <h3>Built-in Helper Methods</h3> <p>Besides the <code>_fields</code> property each namedtuple instance also provides a few more helper methods you might find useful. Their names all start with an underscore character (<code>_</code>) which usually signals that a method or property is “private” and not part of the stable public interface of a class or module.</p> <p>With namedtuples the underscore naming convention has a different meaning though: These helper methods and properties <em>are</em> part of namedtuple’s public interface. The helpers were named that way to avoid naming collisions with user-defined tuple fields. So go ahead and use them if you need them!</p> <p>I want to show you a few scenarios where the namedtuple helper methods might come in handy. Let’s start with the <code>_asdict()</code> helper. It returns the contents of a namedtuple as a dictionary:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span><span class="o">.</span><span class="n">_asdict</span><span class="p">()</span> <span class="n">OrderedDict</span><span class="p">([(</span><span class="s1">'color'</span><span class="p">,</span> <span class="s1">'red'</span><span class="p">),</span> <span class="p">(</span><span class="s1">'mileage'</span><span class="p">,</span> <span class="mf">3812.4</span><span class="p">)])</span> </pre></div> <p>This is great for avoiding typos when generating JSON-output, for example:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">my_car</span><span class="o">.</span><span class="n">_asdict</span><span class="p">())</span> <span class="s1">'{"color": "red", "mileage": 3812.4}'</span> </pre></div> <p>Another useful helper is the <code>_replace()</code> function. It creates a (shallow) copy of a tuple and allows you to selectively replace some of its fields:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span><span class="o">.</span><span class="n">_replace</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="s1">'blue'</span><span class="p">)</span> <span class="n">Car</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="s1">'blue'</span><span class="p">,</span> <span class="n">mileage</span><span class="o">=</span><span class="mf">3812.4</span><span class="p">)</span> </pre></div> <p>Lastly, the <code>_make()</code> classmethod can be used to create new instances of a namedtuple from a sequence or iterable:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">Car</span><span class="o">.</span><span class="n">_make</span><span class="p">([</span><span class="s1">'red'</span><span class="p">,</span> <span class="mi">999</span><span class="p">])</span> <span class="n">Car</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="s1">'red'</span><span class="p">,</span> <span class="n">mileage</span><span class="o">=</span><span class="mi">999</span><span class="p">)</span> </pre></div> <h3>When to Use Namedtuples</h3> <p>Namedtuples can be an easy way to clean up your code and to make it more readable by enforcing a better structure for your data.</p> <p>I find, for example, that going from ad-hoc data types like dictionaries with a fixed format to namedtuples helps me express my intentions more clearly. Often when I attempt this refactoring I magically come up with a better solution for the problem I’m facing.</p> <p>Using namedtuples over unstructured tuples and dicts can also make my coworkers’ lives easier because they make the data being passed around “self-documenting” (to a degree).</p> <p>On the other hand I try not to use namedtuples for their own sake if they don’t help me write “cleaner”, more readable, and more maintainable code. Too much of a good thing can be a bad thing.</p> <p>However if you use them with care, namedtuples can undoubtedly make your Python code better and more expressive.</p> <h3>Things to Remember</h3> <ul> <li><code>collection.namedtuple</code> is a memory-efficient shortcut to defining an immutable class in Python manually.</li> <li>Namedtuples can help clean up your code by enforcing an easier to understand structure on your data.</li> <li>Namedtuples provide a few useful helper methods that all start with an <code>_</code> underscore—but are part of the public interface. It’s okay to use them.</li> </ul></footer></article>https://dbader.org/blog/writing-clean-python-with-namedtuplesTue, 14 Feb 2017 00:00:00 GMTLambda Functions in Python: What Are They Good For?https://dbader.org/blog/python-lambda-functions<article><header><h1>Lambda Functions in Python: What Are They Good For?</h1> <p>An introduction to “lambda” expressions in Python: What they’re good for, when you should use them, and when it’s best to avoid them.</p> </header><section><figure><img alt="Lambda expressions in Python" src="/blog/figures/python-lambda-functions.png" width="1280" height="720"></figure> <p>The <code>lambda</code> keyword in Python provides a shortcut for declaring small anonymous functions. Lambda functions behave just like regular functions declared with the <code>def</code> keyword. They can be used whenever function objects are required.</p> <p>For example, this is how you’d define a simple lambda function carrying out an addition:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">add</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">add</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="mi">8</span> </pre></div> <p>You could declare the same <code>add</code> function with the <code>def</code> keyword:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span> <span class="o">...</span> <span class="k">return</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">add</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="mi">8</span> </pre></div> <p>Now you might be wondering: <strong>Why the big fuss about lambdas?</strong> If they’re just a slightly more terse version of declaring functions with <code>def</code>, what’s the big deal?</p> <p>Take a look at the following example and keep the words <em>function expression</em> in your head while you do that:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span><span class="p">)(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="mi">8</span> </pre></div> <p>Okay, what happened here? I just used <code>lambda</code> to define an “add” function inline and then immediately called it with the arguments <code>5</code> and <code>3</code>.</p> <p>Conceptually the <em>lambda expression</em> <code>lambda x, y: x + y</code> is the same as declaring a function with <code>def</code>, just written inline. The difference is I didn’t bind it to a name like <code>add</code> before I used it. I simply stated the expression I wanted to compute and then immediately evaluated it by calling it like a regular function.</p> <p>Before you move on, you might want to play with the previous code example a little to really let the meaning of it sink in. I still remember this took me a while to wrap my head around. So don’t worry about spending a few minutes in an interpreter session.</p> <p>There’s another syntactic difference between lambdas and regular function definitions: Lambda functions are restricted to a single expression. This means a lambda function can’t use statements or annotations—not even a <code>return</code> statement.</p> <p>How do you return values from lambdas then? Executing a lambda function evaluates its expression and then automatically returns its result. So there’s always an <em>implicit</em> return statement. That’s why some people refer to lambdas as <em>single expression functions</em>.</p> </section><footer><h3>Lambdas You Can Use</h3> <p>When should you use lambda functions in your code? Technically, any time you’re expected to supply a function object you can use a lambda expression. And because a lambda expression can be anonymous, you don’t even need to assign it to a name.</p> <p>This can provide a handy and “unbureaucratic” shortcut to defining a function in Python. My most frequent use case for lambdas is writing short and concise <em>key funcs</em> for <a href="python-min-max-and-nested-lists">sorting iterables by an alternate key</a>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">sorted</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="o">-</span><span class="mi">5</span><span class="p">,</span> <span class="mi">6</span><span class="p">),</span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="o">-</span><span class="mi">4</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="o">-</span><span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">]</span> </pre></div> <p>Like regular nested functions, lambdas also work as <em>lexical closures</em>.</p> <p>What’s a lexical closure? Just a fancy name for a function that remembers the values from the enclosing lexical scope even when the program flow is no longer in that scope. Here’s a (fairly academic) example to illustrate the idea:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">make_adder</span><span class="p">(</span><span class="n">n</span><span class="p">):</span> <span class="o">...</span> <span class="k">return</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span> <span class="o">+</span> <span class="n">n</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_3</span> <span class="o">=</span> <span class="n">make_adder</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_5</span> <span class="o">=</span> <span class="n">make_adder</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_3</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span> <span class="mi">7</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_5</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span> <span class="mi">9</span> </pre></div> <p>In the above example the <code>x + n</code> lambda can still access the value of <code>n</code> even though it was defined in the <code>make_adder</code> function (the enclosing scope).</p> <p>Sometimes, using a lambda function instead of a nested function declared with <code>def</code> can express one’s intent more clearly. But to be honest this isn’t a common occurrence—at least in the kind of code that I like to write.</p> <h3>But Maybe You Shouldn’t…</h3> <p>Now on the one hand I’m hoping this article got you interested in exploring Python’s lambda functions. On the other hand I feel like it’s time to put up another caveat: Lambda functions should be used sparingly and with extraordinary care.</p> <p>I know I wrote my fair share of code using lambdas that looked “cool” but was actually a liability for me and my coworkers. If you’re tempted to use a lambda spend a few seconds (or minutes) to think if this is really the cleanest and most maintainable way to achieve the desired result.</p> <p>For example, doing something like this to save two lines of code is just silly. Sure, it technically works and it’s a nice enough “trick”. But it’s also going to confuse the next gal or guy having to ship a bugfix under a tight deadline:</p> <div class="codehilite"><pre><span></span><span class="c1"># Harmful:</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">class</span> <span class="nc">Car</span><span class="p">:</span> <span class="o">...</span> <span class="n">rev</span> <span class="o">=</span> <span class="k">lambda</span> <span class="bp">self</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s1">'Wroom!'</span><span class="p">)</span> <span class="o">...</span> <span class="n">crash</span> <span class="o">=</span> <span class="k">lambda</span> <span class="bp">self</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s1">'Boom!'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span> <span class="o">=</span> <span class="n">Car</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span><span class="o">.</span><span class="n">crash</span><span class="p">()</span> <span class="s1">'Boom!'</span> </pre></div> <p>I feel similarly about complicated <code>map()</code> or <code>filter()</code> constructs using lambdas. Usually it’s much cleaner to go with a list comprehension or generator expression:</p> <div class="codehilite"><pre><span></span><span class="c1"># Harmful:</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">list</span><span class="p">(</span><span class="nb">filter</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">,</span> <span class="nb">range</span><span class="p">(</span><span class="mi">16</span><span class="p">)))</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">14</span><span class="p">]</span> <span class="c1"># Better:</span> <span class="o">&gt;&gt;&gt;</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span> <span class="k">if</span> <span class="n">x</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">]</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">14</span><span class="p">]</span> </pre></div> <p>If you find yourself doing anything remotely complex with a lambda expression, consider defining a real function with a proper name instead.</p> <p>Saving a few keystrokes won’t matter in the long run. Your colleagues (and your future self) will appreciate clean and readable code more than terse wizardry.</p> <h3>Things to Remember</h3> <ul> <li>Lambda functions are single-expression functions that are not necessarily bound to a name (anonymous).</li> <li>Lambda functions can’t use regular Python statements and always include an implicit <code>return</code> statement.</li> <li>Always ask yourself: <em>Would using a regular (named) function or a list/generator expression offer more clarity?</em></li> </ul></footer></article>https://dbader.org/blog/python-lambda-functionsTue, 07 Feb 2017 00:00:00 GMTWhy Learn Python? Here Are 8 Data-Driven Reasonshttps://dbader.org/blog/why-learn-python<article><header><h1>Why Learn Python? Here Are 8 Data-Driven Reasons</h1> <p>Is Python worth learning? We’ve interviewed experts and surveyed the job market to identify the key reasons why you should learn Python today.</p> </header><section><figure><img alt="Why Learn Python? Here Are 8 Data-Driven Reasons" src="/blog/figures/python-worth-learning-header.png" width="1280" height="720"></figure> <p>Python had a great year in 2016. The latest <a href="http://stackoverflow.com/research/developer-survey-2016" target="_blank">Stack Overflow Developer Survey</a> ranked Python as the 6th most popular and the 4th most wanted technology of the year.</p> <p>Python is also one of the hottest skills to have according to <a href="http://insights.dice.com/2016/02/01/whats-hot-and-not-in-tech-skills/" target="_blank">research by Dice</a>, and the 2nd most popular programming language in the world based on the <a href="http://pypl.github.io/PYPL.html" target="_blank">PYPL Popularity of Programming Language Index</a>.</p> <p>So why the hype? What makes Python so popular? Should you stop what you’re doing and start learning Python right now? I’ve searched far and wide to find out why Python is one of the world’s most loved and most used technologies. Without further ado, here’s why Python is worth learning in 2017 and the years ahead:</p> </section><section><h2>1. You Can Use Python for Pretty Much Anything</h2> <p>One significant advantage of learning Python is that it’s a general-purpose language that can be applied in a large variety of projects. Below are just some of the most common fields where Python has found its use:</p> <ul> <li>Data science</li> <li>Scientific and mathematical computing</li> <li>Web development</li> <li>Finance and trading</li> <li>System automation and administration</li> <li>Computer graphics</li> <li>Basic game development</li> <li>Security and penetration testing</li> <li>General and application-specific scripting</li> <li>Mapping and geography (GIS software)</li> </ul> <p>In preparation for this post, I posted the question <em>“Is Python worth learning?”</em> on Google+, Quora, and LinkedIn in order to collect some professional opinions on the matter. Here’s one of the responses I got that supports my point:</p> <blockquote> <p>“I had the opportunity to start learning Python 6 years ago. Since that time, I’ve used Python for everything from work related stuff to home automation tasks, and I have never stumbled upon a problem that can’t be solved with Python.”</p> </blockquote> <p>— <a href="https://www.linkedin.com/in/abensrhir" target="_blank">Anass Bensrhir</a>, Senior Data Scientist and Managing Director at <a href="http://www.bolddata.net/" target="_blank">Bold Data</a></p> </section><section><h2>2. Python Is Widely Used in Data Science</h2> <figure><img alt="Python developer job roles" src="/blog/figures/python-worth-learning-1.png" width="825" height="575"></figure> <p>(<a href="https://www.packtpub.com/skill-up-2016" target="_blank">Source</a>)</p> <p>Python’s application in data science and data engineering is what’s really fuelling its popularity today. Pandas, NumPy, SciPy, and other tools combined with the ability to prototype quickly and then “glue” systems together enable data engineers to maintain high efficiency when using Python.</p> <p><a href="https://plus.google.com/u/0/+JustinMcGrath" target="_blank">Justin McGrath</a>, a researcher at the University of Illinois, Champaign-Urbana agrees:</p> <blockquote> <p>“<strong>Python is probably going to become the de facto standard for scientific and statistical analyses</strong>. If you’re going into those fields, it’s certainly worth learning.”</p> </blockquote> </section><section><h2>3. Python Pays Well</h2> <p>It’s all well and good, but what about the pay, I hear you ask? It turns out <a href="https://www.daxx.com/article/python-developer-salary-usa" target="_blank">Python engineers have some of the highest salaries in the industry</a>, at least in the US.</p> <p>At nearly $103,500 per year, Python is the second best-paying programming language in the country (beating out Java, C++, and JavaScript) according to <a href="https://gooroo.io/" target="_blank">Gooroo</a>, a skill and salary analytics platform.</p> <figure><img alt="Average salary for Python developers" src="/blog/figures/python-worth-learning-2.png" width="591" height="369"></figure> <p><a href="http://www.indeed.com/salaries/Python-Developer-Salaries?from=salaries-search" target="_blank">Indeed’s salary calculator</a> gives an even larger figure—a whopping $116,000 per year. Of course, tech salaries differ greatly from one state to another. So to add some context, here’s a breakdown of how much Python engineers make in the states featured on Indeed:</p> <figure><img alt="Average Python developer salaries in the USA" src="/blog/figures/python-worth-learning-3.jpg" width="1240" height="711"></figure> <p>(<a href="figures/python-worth-learning-3.jpg" target="_blank">Click to view a larger version of the above image.</a>)</p> <figure><img alt="Python developer salaries in the USA (ranking)" src="/blog/figures/python-worth-learning-4.png" width="705" height="806"></figure> <p>(<a href="figures/python-worth-learning-4.png" target="_blank">Click to view a larger version of the above image.</a>)</p> </section><section><h2>4. Demand for Python Developers Is High (And Growing)</h2> <p>Based on <a href="https://www.indeed.com/jobtrends/q-python.html" target="_blank">Indeed’s job trends</a>, it looks like having Python under your belt can help you land a job in very short terms. The graph below displays a steady growth in the number of job postings featuring Python since 2012, and there has been a strong spike in popularity over the last six months.</p> <figure><img alt="Python developer job postings" src="/blog/figures/python-worth-learning-5.png" width="754" height="527"></figure> <p>What’s more, the demand for Python skills clearly outstrips jobseeker interest. The job market outlook for Python developers is excellent at the moment.</p> <figure><img alt="Python developer jobseeker interest" src="/blog/figures/python-worth-learning-6.png" width="754" height="527"></figure> </section><section><h2>5. Python Saves Time</h2> <p>I’m pretty sure that the majority of the developers who’ve used Python would agree that making anything with this language takes a lot less time <em>and</em> code than most other technologies.</p> <p>Even the classic “Hello, world” program illustrates this point:</p> <div class="codehilite"><pre><span></span><span class="k">print</span><span class="p">(</span><span class="s2">"Hello, world"</span><span class="p">)</span> </pre></div> <p>For comparison, this is what the same program looks like in Java:</p> <div class="codehilite"><pre><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">HelloWorld</span> <span class="o">{</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span> <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Hello, world"</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> </pre></div> </section><section><h2>6. Python Is Beginner Friendly</h2> <p>Following up from the previous point, Python’s famously clean and readable syntax makes it newbie-friendly. A well-written Python program can look like it came straight out of an algorithms text book. There’s little superfluous boilerplate, allowing beginners and experts alike to focus on the job at hand—instead of the code.</p> <p>Python’s efficiency and readability have also made it <a href="http://cacm.acm.org/blogs/blog-cacm/176450-python-is-now-the-most-popular-introductory-teaching-language-at-top-u-s-universities/fulltext" target="_blank">the number one most commonly taught introductory language at top US universities</a>. This will have ramifications on the future job market and likely make Python an even more popular technology choice.</p> </section><section><h2>7. All the Big Names Use Python</h2> <p>Ever wanted to work for a tech giant like Google or Facebook? Python could be your way in, as these companies, as well as YouTube, IBM, Yahoo, Dropbox, Quora, Mozilla, Instagram, and many others all use Python for a wide array of purposes, and are constantly hiring Python developers.</p> <figure><img alt="Programming Languages Company Size Breakdown" src="/blog/figures/python-worth-learning-7.png" width="715" height="580"></figure> <p>(<a href="https://www.packtpub.com/skill-up-2016" target="_blank">Source</a>)</p> <p>Dropbox’s code base, for instance, uses Python for almost everything, including analytics, the server backend, the API backend, and the desktop clients.</p> </section><section><h2>8. Python Has an Amazing Ecosystem</h2> <p>Last but not least, there’s a huge number of resources developed for Python that keep getting updated, including <a href="https://docs.python.org/3/library/" target="_blank">an impressive standard library</a> with built-in functionality, a built-in <a href="https://docs.python.org/2.7/library/unittest.html" target="_blank">unit testing framework</a>, and more than enough frameworks and environments that allow you to focus on writing the website or app at hand.</p> <p><a href="https://www.djangoproject.com/" target="_blank">Django</a> is the most commonly used Python web framework, but there’s also <a href="http://flask.pocoo.org/" target="_blank">Flask</a>, <a href="https://trypyramid.com" target="_blank">Pyramid</a>, <a href="http://www.web2py.com/" target="_blank">web2py</a>, <a href="https://pypi.python.org/pypi/Zope2" target="_blank">Zope 2</a>, and a few more.</p> </section><section><h2>What Do Python Community Leaders Think?</h2> <p>I thought it would also be a good idea to top things up with a few expert opinions on the advantages of Python as well as its future. It’s always a good idea to get a second (or third) opinion. So I reached out to several influencers and leaders in the Python space.</p> <p>I asked each of these experts three questions:</p> <ol> <li>What advantages does Python have over other programming languages?</li> <li>What future do you see for Python in 3–5 years?</li> <li>What will the job market look like for a Python developer in the coming years?</li> </ol> <p>Here’s what I was able to learn:</p> <div class="testimonial testimonial-box"> <p></p><div class="testimonial-attribution" style="margin-top: 0;"> <img src="/img/testimonial-mk.jpg" width="512" height="512"> <div> <span class="name">Michael Kennedy</span> <span class="job">Python Coach and Host of the Talk Python and Python Bytes Podcasts</span> </div> </div> <hr> <h3>“You start easy but you rarely outgrow Python like you do other easy to learn languages”</h3> <hr> <p><strong>What advantages does Python have over other programming languages?</strong></p> <p>I often think of programming languages as falling into two buckets.</p> <p>The first group would be the “With great power comes great responsibility” type of languages. This would be C, C++, and to a lesser degree C# and Java. The others are “I just need to ship something, don’t waste my time with minutia” languages. Visual Basic (pre-VB.NET) and JavaScript seem solidly in this camp, although JavaScript appears to be trying to escape with the massive decoupling seen in typical Node.js code and TypeScript.</p> <p>You choose C++ or C# if you need to really control the system and build large professional software. Is it mission critical enterprise software running the company with 100k lines of code? You might choose these. If you need a quick app to get the job done, like writing that “forms over data” app for something internal, VB 6 used to be a great answer for finishing that in a week, but coding yourself into a box if it grows too big or needs low level capabilities.</p> <p>Python is one of the few languages that is:</p> <ul> <li>Easy to learn</li> <li>Solves that “Don’t waste my time” set of problems well</li> <li>Yet, is also well designed with OOP and solid modern language features</li> <li>Can grow in power to match the powerful languages in capabilities</li> </ul> <p>In short, it’s one of the few languages that spans the spectrum of these capabilities. You start easy but you rarely outgrow Python like you do other easy to learn languages.</p> <p>We could also go into things like data science, scientific computing, web development, microcontrollers, things like Raspberry Pi, and how Python spans more technologies and areas of focus than most programming languages do.</p> <p>But the full spectrum aspect is the most powerful to me.</p> <p><strong>What future do you see for Python in 3–5 years?</strong></p> <p>In terms of predictions, I’m willing to make a few:</p> <ol> <li>Python will continue to expand into new areas of computing. It will be the primary IoT programming language.</li> <li>We will see Python interpreters/runtimes evolve and innovate. The YouTube team just released a <a href="https://talkpython.fm/episodes/show/95/grumpy-running-python-on-go" target="_blank">project</a> running Python on the Go runtime for example.</li> <li>The Python 3 vs Python 2 schism that has turned off countless new developers and generally been a cloud over community will be closed, and Python 3 will be just “Python”.</li> </ol> <p><strong>What will the job market look like for a Python developer in the coming years?</strong></p> <p>Given the growth numbers as well as the wide areas of computing that Python occupies, I think the job perspectives for Python developers are very solid.</p> <p>Some folks may feel Python is kind of a niche language or a small time scripting language. But very major applications are written in Python, including Dropbox and Youtube.</p> <p>Other areas outside web development where Python shines are places like the Large Hadron Collider where the team that found the Higgs Boson and won the Nobel Prize made heavy use of Python. Netflix uses Python to manage their AWS servers which cumulatively handle up to 35% of the bandwidth of the United States during the evenings.</p> <p>You’ll find that some locations in the world are more Python-centric than others. But there are many opportunities for Python developers.</p> <hr> <p><a href="https://twitter.com/mkennedy" target="_blank"><strong>Michael Kennedy</strong></a> is a Python coach and host of the popular <a href="https://talkpython.fm/" target="_blank">Talk Python</a> and <a href="https://pythonbytes.fm/" target="_blank">Python Bytes</a> podcasts.</p> </div> <div class="testimonial testimonial-box"> <p></p><div class="testimonial-attribution" style="margin-top: 0;"> <img src="/img/sublime-python/testimonial-ag.jpg" width="400" height="400"> <div> <span class="name">Ankur Gupta</span> <span class="job">Curator at ImportPython</span> </div> </div> <hr> <h3>“There is a demand-supply mismatch for Python developers with 2 to 6 years of experience”</h3> <hr> <p><strong>What advantages does Python have over other programming languages?</strong></p> <p>Python is an easier language to learn compared to, say, C++, C, C#, or Java, but that’s not it. We often tend to credit syntax, core team, feature roadmap, etc for the success of a certain language.</p> <p>They’re beyond doubt important, but when it comes to Python, it’s the global, diverse, and vibrant community that make it so widely adopted. Initiatives like <a href="https://djangogirls.org" target="_blank">Django Girls</a> and the scale at which they operate are unique. There are at least three dozen free books on Python, thousands of free videos to learn from, as well as the PyCon events all around the world.</p> <p>Active local and online regional Python communities are the biggest advantage that Python has over other languages. It’s the people behind the language that make it special.</p> <p><strong>What future do you see for Python in 3–5 years?</strong></p> <p>10 years ago, mentioning Python was guaranteed to invite blank stares. But today, Python is a pretty mainstream language. I think Python is here to stay.</p> <p>In 3–5 years I foresee:</p> <ul> <li>2.x codebase becoming a minority</li> <li>Python developers being available in abundance thanks to schools and colleges that teach Python as an introductory language</li> <li>People using different Python runtime interpreters instead of just CPython</li> </ul> <p><strong>What will the job market look like for a Python developer in the coming years?</strong></p> <p>Back in 2007–2008, I’d get no more than 3–4 calls a month concerning Python job openings, and most of those calls had to do with Python scripting for test automation (India). But if I were to look for a job today, I’m sure my phone would ring multiple times per day.</p> <p>There is a demand-supply mismatch for Python developers with 2 to 6 years of experience because of all these companies wanting to use Python for data science, data processing, machine learning, web application development, and so on.</p> <p>This situation will be gradually improving over the next couple of years, which means today is definitely the best time to be a Python developer.</p> <hr> <p><a href="https://twitter.com/originalankur" target="_blank"><strong>Ankur Gupta</strong></a> is the curator of the <a href="http://importpython.com/newsletter/" target="_blank">weekly newsletter over at ImportPython.com</a>, which keeps you updated on everything happening in the world of Python programming.</p> </div> <div class="testimonial testimonial-box"> <p></p><div class="testimonial-attribution" style="margin-top: 0;"> <img src="/img/testimonial-sv.jpg" width="460" height="460"> <div> <span class="name">Sebastian Vetter</span> <span class="job">Python Engineer at Eventbase, PyCon Speaker and Meetup Host</span> </div> </div> <hr> <h3>“The community around Python is the most welcoming and inclusive one out of all those that I’ve experienced”</h3> <hr> <p><strong>What advantages does Python have over other programming languages?</strong></p> <ul> <li><strong>Community.</strong> The community around Python is the most welcoming and inclusive one out of all those that I’ve experienced. Many times I’ve been inspired by the progressive effort at meetups and conferences to be inclusive to newcomers, underrepresented groups and minorities.</li> <li><strong>Readability.</strong> A lot of effort has gone into developing Python as a language that has readability as one of its main features, rather than considering it as an afterthought. As Robert C. Martin wrote in <em>Clean Code</em>, “the ratio of time spent reading versus writing is well over 10 to 1.”</li> <li><strong>Consistency.</strong> One of the things that I’ve always loved about Python is the fact that it uses whitespace to determine blocks instead of using various types of brackets. Although this is a little unintuitive when starting out, in my opinion, the advantage is that it ensures that Python code is relatively similar across different projects. It improves consistency and readability.</li> </ul> <p><strong>What future do you see for Python in 3–5 years?</strong></p> <p>In my opinion, the use of Python and the number of developers working with it will grow significantly in scientific fields. The number of science-related topics at Python conferences (and beyond) and releases of new tools to help the scientific community will make it easier to adopt the language. This will give the scientific community access to a very inclusive and welcoming developer community that will help improve the quality of development and simplify the tooling for scientific and research-related applications.</p> <p>The mobile space is going to be very interesting in about 3–5 years. As Russell Keith-Magee pointed out in his presentation “Python on the Move: the State of Mobile Python” at PyCon AU 2015, the future of Python as a language will most likely depend in part on how the community moves into the mobile development space. Although the Python community is very diverse and the language is used in a lot of different fields, we currently don’t have any decent support for mobile platforms. Looking at Russell’s efforts to bridge this gap with his project under the BeeWare umbrella, I’m confident that this gap will be closed within the next few years, and we’ll be able to maintain a strong position even in these new areas.</p> <p>Over the last several years, there’s been a lot of disagreement over Python 3 and whether it’s a step in the right direction. I do understand some of the critical arguments made against Python 3. Several highly qualified Pythonistas with vastly more experience than myself have raised valid concerns and pointed out flaws. Regardless of these concerns, I’m convinced that the adoption of Python 3 will pick up steam over the next two or three years, moving faster towards it being the mainstream version. This is indicated by projects like Django dropping support for Python 2.7 within 2017 with their release of Django 2.0 and the broader adoption of asyncio and coroutine-based frameworks and libraries.</p> <p>Making the Python community a more inclusive space for individuals of underrepresented groups such as women and other minorities will help us build a community made up of all different types of people. I’m sure that over the next 5 years, we’ll see the first major benefits of these initiatives contributing to a much stronger community. Making everyone welcome and embracing the differences in perspectives and experiences will serve as a model for companies, proving that such an environment results in better software and happier employees. I also think that individuals from within the Python community who’ve experienced this atmosphere will impact their employers by demanding a similar environment in their professional lives, drawing from the support of the community.</p> <p><strong>What will the job market look like for a Python developer in the coming years?</strong></p> <p>The next few years will most likely see a much more diverse landscape of Python jobs. With the increased application of Python in scientific fields, more research positions will become available. In addition, I think the growing need of programming skills within the scientific community will lead to having a combination of researchers and programmers to produce a skilled workforce that is capable in the scientific aspect as well as development best practises and tooling.</p> <p>The position of Data Scientist is going to become more and more important in the tech industry and will therefore increase the demand within the Python community specifically. We already have a large number of scientists use Python as their main language for their research in our community. Their skills in statistics and the use of the language will make them prime candidates for positions that are related to data-driven systems. With the demand for such systems growing fast, there will be a high demand for these individuals, and anybody within the Python community willing to level up on either the development aspects or the scientific skills.</p> <p>The Python community is strongly committed to improving its inclusiveness and diversity. Mandating and enforcing codes of conduct at conferences and meetups as well as openly stating the inclusive nature of communities around projects like the Django framework are helping to improve the representation of underprivileged individuals within the community. I hope and believe that this will, over the next few years, help make the community a place that will thrive, because individuals from these underrepresented groups will feel safe and welcome. This will make the Python community an exceptional pool to tap into for companies that are making an effort to improve the diversity of their development and science teams.</p> <hr> <p><a href="http://www.roadsi.de" target="_blank"><strong>Sebastian Vetter</strong></a> is a Senior Python Engineer at <a href="https://www.eventbase.com/" target="_blank">Eventbase</a>, PyCon speaker and Python meet-up host.</p> </div></section><footer></footer></article>https://dbader.org/blog/why-learn-pythonTue, 31 Jan 2017 00:00:00 GMTThe 4 Major Ways to Do String Formatting in Pythonhttps://dbader.org/blog/python-string-formatting<article><header><h1>The 4 Major Ways to Do String Formatting in Python</h1> <p>Remember the Zen of Python and how there should be “one obvious way to do something in Python”? You might scratch your head when you find out that there are *four* major ways to do string formatting in Python.</p> </header><section><figure><img alt="String Formatting in Python (updated for Python 3.6 and above)" src="/blog/figures/python-string-formatting-36.png" width="1280" height="720"></figure> <p>In this article I’ll demonstrate how these four string formatting approaches work and what their respective strengths and weaknesses are. I’ll also give you my simple “rule of thumb” for how I pick the best general purpose string formatting approach.</p> <p>Let’s jump right in, as we’ve got a lot to cover. In order to have a simple toy example for experimentation, let’s assume we’ve got the following variables (or constants, really) to work with:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">errno</span> <span class="o">=</span> <span class="mi">50159747054</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">'Bob'</span> </pre></div> <p>And based on these variables we’d like to generate an output string containing a simple error message:</p> <div class="codehilite"><pre><span></span><span class="s1">'Hey Bob, there is a 0xbadc0ffee error!'</span> </pre></div> <p>Hey… now <em>that</em> error could really spoil a dev’s Monday morning. But we’re here to discuss string formatting. So let’s get to work.</p> </section><section><h2><a class="anchor" name="old-style"></a>#1 – “Old Style” String Formatting (%-operator)</h2> <p>Strings in Python have a unique built-in operation that can be accessed with the <code>%</code>-operator. This lets you do simple positional formatting very easily. If you’ve ever worked with a <code>printf</code>-style function in C you’ll recognize how this works instantly. Here’s a simple example:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="s1">'Hello, </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">name</span> <span class="s2">"Hello, Bob"</span> </pre></div> <p>I’m using the <code>%s</code> format specifier here to tell Python where to substitute the value of <code>name</code>, represented as a string.</p> <p>There are other format specifiers available that let you control the output format. For example it’s possible to convert numbers to hexadecimal notation or to add whitespace padding to generate nicely formatted tables and reports (<a href="https://docs.python.org/3/library/stdtypes.html#old-string-formatting" target="_blank">cf. Python Docs: “printf-style String Formatting”</a>).</p> <p>Here, we can use the <code>%x</code> format specifier to convert an int value to a string and to represent it as a hexadecimal number:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="s1">'</span><span class="si">%x</span><span class="s1">'</span> <span class="o">%</span> <span class="n">errno</span> <span class="s1">'badc0ffee'</span> </pre></div> <p>The “old style” string formatting syntax changes slightly if you want to make multiple substitutions in a single string. Because the <code>%</code>-operator only takes one argument you need to wrap the right-hand side in a tuple, like so:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="s1">'Hey </span><span class="si">%s</span><span class="s1">, there is a 0x</span><span class="si">%x</span><span class="s1"> error!'</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">errno</span><span class="p">)</span> <span class="s1">'Hey Bob, there is a 0xbadc0ffee error!'</span> </pre></div> <p>It’s also possible to refer to variable substitutions by name in your format string, if you pass a mapping to the <code>%</code>-operator:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="s1">'Hey </span><span class="si">%(name)s</span><span class="s1">, there is a 0x</span><span class="si">%(errno)x</span><span class="s1"> error!'</span> <span class="o">%</span> <span class="p">{</span> <span class="o">...</span> <span class="s2">"name"</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span> <span class="s2">"errno"</span><span class="p">:</span> <span class="n">errno</span> <span class="p">}</span> <span class="s1">'Hey Bob, there is a 0xbadc0ffee error!'</span> </pre></div> <p>This makes your format strings easier to maintain and easier to modify in the future. You don’t have to worry about making sure the order you’re passing in the values matches up with the order the values are referenced in the format string. Of course the downside is that this technique requires a little more typing.</p> <p>I’m sure you’ve been wondering why this <code>printf</code>-style formatting is called “old style” string formatting. It was technically superseded by “new style” formatting, which we’re going to talk about in a minute.</p> </section><section><h2><a class="anchor" name="new-style"></a>#2 – “New Style” String Formatting (str.format)</h2> <p>Python 3 introduced a new way to do string formatting that was also later back-ported to Python 2.7. This “new style” string formatting gets rid of the <code>%</code>-operator special syntax and makes the syntax for string formatting more regular. Formatting is now handled by calling a <code>format()</code> function on a string object (<a href="https://docs.python.org/3/library/stdtypes.html#str.format" target="_blank">cf. Python Docs: “str.format”</a>).</p> <p>You can use the <code>format()</code> function to do simple positional formatting, just like you could with “old style” formatting:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="s1">'Hello, {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="s1">'Hello, Bob'</span> </pre></div> <p>Or, you can refer to your variable substitutions by name and use them in any order you want. This is quite a powerful feature as it allows for re-arranging the order of display without changing the arguments passed to the format function:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="s1">'Hey {name}, there is a 0x{errno:x} error!'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span> <span class="o">...</span> <span class="n">name</span><span class="o">=</span><span class="n">name</span><span class="p">,</span> <span class="n">errno</span><span class="o">=</span><span class="n">errno</span><span class="p">)</span> <span class="s1">'Hey Bob, there is a 0xbadc0ffee error!'</span> </pre></div> <p>This also shows that the syntax to format an int variable as a hexadecimal string has changed. Now we need to pass a <em>format spec</em> by adding a <code>:x</code> suffix. The format string syntax has become more powerful without complicating the simpler use cases. It pays off to read up on this <em>string formatting mini-language</em> in the Python documentation (<a href="https://docs.python.org/3/library/string.html#string-formatting" target="_blank">cf. Python Docs: “Format String Syntax”</a>).</p> <p>In Python 3, this “new style” string formatting is to be preferred over <code>%</code>-style formatting. While <a href="https://docs.python.org/3/library/stdtypes.html?highlight=sprintf#printf-style-string-formatting" target="_blank">“old style” formatting has been de-emphasized</a> it has <em>not</em> been deprecated. It is still supported in the latest versions of Python. According to <a href="https://lists.gt.net/python/dev/969817" target="_blank">this discussion on the Python dev email list</a> and <a href="https://bugs.python.org/issue14123" target="_blank">this issue on the Python dev bug tracker</a>, <code>%</code>-formatting is going to stick around for a long time to come.</p> <p>Still, the official Python 3.X documentation doesn’t exactly recommend “old style” formatting or speak too fondly of it:</p> <blockquote> <p>The formatting operations described here exhibit a variety of quirks that lead to a number of common errors (such as failing to display tuples and dictionaries correctly). Using the newer formatted string literals or the str.format() interface helps avoid these errors. These alternatives also provide more powerful, flexible and extensible approaches to formatting text. (Source: <a href="https://docs.python.org/3/library/stdtypes.html?highlight=sprintf#printf-style-string-formatting" target="_blank">Python 3 Docs</a>)</p> </blockquote> <p>This is why I’d personally try to stick with <code>str.format</code> for new code moving forward. Starting with Python 3.6 there’s yet another way to format your strings. I’ll tell you all about it in the next section.</p> </section><section><h2><a class="anchor" name="interpolation"></a>#3 – Literal String Interpolation (Python 3.6+)</h2> <p><a href="cool-new-features-in-python-3-6">Python 3.6 adds a new string formatting approach</a> called <em>Formatted String Literals</em>. This new way of formatting strings lets you use embedded Python expressions inside string constants. Here’s a simple example to give you a feel for the feature:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">f</span><span class="s1">'Hello, {name}!'</span> <span class="s1">'Hello, Bob!'</span> </pre></div> <p>This new formatting syntax is powerful. Because you can embed arbitrary Python expressions you can even do inline arithmetic with it. See here for example:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">5</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">b</span> <span class="o">=</span> <span class="mi">10</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">f</span><span class="s1">'Five plus ten is {a + b} and not {2 * (a + b)}.'</span> <span class="s1">'Five plus ten is 15 and not 30.'</span> </pre></div> <p>Formatted string literals are a Python parser feature that converts f-strings into a series of string constants and expressions. They then get joined up to build the final string.</p> <p>Imagine we had the following <code>greet()</code> function that contains an f-string:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">greet</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">question</span><span class="p">):</span> <span class="o">...</span> <span class="k">return</span> <span class="n">f</span><span class="s2">"Hello, {name}! How's it {question}?"</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="p">(</span><span class="s1">'Bob'</span><span class="p">,</span> <span class="s1">'going'</span><span class="p">)</span> <span class="s2">"Hello, Bob! How's it going?"</span> </pre></div> <p>When we disassemble the function and inspect what’s going on behind the scenes we can see that the f-string in the function gets transformed into something similar to the following:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">greet</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">question</span><span class="p">):</span> <span class="o">...</span> <span class="k">return</span> <span class="s2">"Hello, "</span> <span class="o">+</span> <span class="n">name</span> <span class="o">+</span> <span class="s2">"! How's it "</span> <span class="o">+</span> <span class="n">question</span> <span class="o">+</span> <span class="s2">"?"</span> </pre></div> <p>The real implementation is slightly faster than that because it uses the <a href="https://bugs.python.org/issue27078" target="_blank"><code>BUILD_STRING</code> opcode as an optimization</a>. But functionally they’re the same:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">dis</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dis</span><span class="o">.</span><span class="n">dis</span><span class="p">(</span><span class="n">greet</span><span class="p">)</span> <span class="mi">2</span> <span class="mi">0</span> <span class="n">LOAD_CONST</span> <span class="mi">1</span> <span class="p">(</span><span class="s1">'Hello, '</span><span class="p">)</span> <span class="mi">2</span> <span class="n">LOAD_FAST</span> <span class="mi">0</span> <span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="mi">4</span> <span class="n">FORMAT_VALUE</span> <span class="mi">0</span> <span class="mi">6</span> <span class="n">LOAD_CONST</span> <span class="mi">2</span> <span class="p">(</span><span class="s2">"! How's it "</span><span class="p">)</span> <span class="mi">8</span> <span class="n">LOAD_FAST</span> <span class="mi">1</span> <span class="p">(</span><span class="n">question</span><span class="p">)</span> <span class="mi">10</span> <span class="n">FORMAT_VALUE</span> <span class="mi">0</span> <span class="mi">12</span> <span class="n">LOAD_CONST</span> <span class="mi">3</span> <span class="p">(</span><span class="s1">'?'</span><span class="p">)</span> <span class="mi">14</span> <span class="n">BUILD_STRING</span> <span class="mi">5</span> <span class="mi">16</span> <span class="n">RETURN_VALUE</span> </pre></div> <p>String literals also support the existing format string syntax of the <code>str.format()</code> method. That allows you to solve the same formatting problems we’ve discussed in the previous two sections:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">f</span><span class="s2">"Hey {name}, there's a {errno:#x} error!"</span> <span class="s2">"Hey Bob, there's a 0xbadc0ffee error!"</span> </pre></div> <p>Python’s new Formatted String Literals are similar to the JavaScript Template Literals added in ES2015. I think they’re quite a nice addition to the language and I’ve already started using them in my day to day (Python 3) work. You can learn more about Formatted String Literals in the official Python documentation (<a href="https://docs.python.org/3/reference/lexical_analysis.html#f-strings" target="_blank">cf. Python Docs: “Formatted string literals”</a>).</p> </section><section><h2><a class="anchor" name="template-strings"></a>#4 – Template Strings (standard library)</h2> <p>Here’s one more technique for string formatting in Python: Template Strings. It’s a simpler and less powerful mechanism, but in some cases this might be exactly what you’re looking for.</p> <p>Let’s take a look at a simple greeting example:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">string</span> <span class="kn">import</span> <span class="n">Template</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">t</span> <span class="o">=</span> <span class="n">Template</span><span class="p">(</span><span class="s1">'Hey, $name!'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">substitute</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">name</span><span class="p">)</span> <span class="s1">'Hey, Bob!'</span> </pre></div> <p>You see here that we need to import the <code>Template</code> class from Python’s built-in <code>string</code> module. Template strings are not a core language feature but they’re supplied by a module in the standard library.</p> <p>Another difference is that template strings don’t allow format specifiers. So in order to get our error string example to work we need to transform our int error number into a hex-string ourselves:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">templ_string</span> <span class="o">=</span> <span class="s1">'Hey $name, there is a $error error!'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">Template</span><span class="p">(</span><span class="n">templ_string</span><span class="p">)</span><span class="o">.</span><span class="n">substitute</span><span class="p">(</span> <span class="o">...</span> <span class="n">name</span><span class="o">=</span><span class="n">name</span><span class="p">,</span> <span class="n">error</span><span class="o">=</span><span class="nb">hex</span><span class="p">(</span><span class="n">errno</span><span class="p">))</span> <span class="s1">'Hey Bob, there is a 0xbadc0ffee error!'</span> </pre></div> <p>That worked great. So when should you use template strings in your Python programs? In my opinion the best use case for template strings is when you’re handling format strings generated by users of your program. Due to their reduced complexity template strings are a safer choice.</p> <p>The more complex formatting mini-languages of the other string formatting techniques might introduce security vulnerabilities to your programs. For example, it’s <a href="http://lucumr.pocoo.org/2016/12/29/careful-with-str-format/" target="_blank">possible for format strings to access arbitrary variables in your program</a>.</p> <p>That means, if a malicious user can supply a format string they can potentially leak secret keys and other sensible information! Here’s a simple proof of concept of how this attack might be used:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">SECRET</span> <span class="o">=</span> <span class="s1">'this-is-a-secret'</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">class</span> <span class="nc">Error</span><span class="p">:</span> <span class="o">...</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="o">...</span> <span class="k">pass</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">err</span> <span class="o">=</span> <span class="n">Error</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">user_input</span> <span class="o">=</span> <span class="s1">'{error.__init__.__globals__[SECRET]}'</span> <span class="c1"># Uh-oh...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">user_input</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">error</span><span class="o">=</span><span class="n">err</span><span class="p">)</span> <span class="s1">'this-is-a-secret'</span> </pre></div> <p>See how a hypothetical attacker was able to extract our secret string by accessing the <code>__globals__</code> dictionary? Scary, huh? Template Strings close this attack vector. And this makes them a safer choice if you’re handling format strings generated from user input:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">user_input</span> <span class="o">=</span> <span class="s1">'${error.__init__.__globals__[SECRET]}'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">Template</span><span class="p">(</span><span class="n">user_input</span><span class="p">)</span><span class="o">.</span><span class="n">substitute</span><span class="p">(</span><span class="n">error</span><span class="o">=</span><span class="n">err</span><span class="p">)</span> <span class="ne">ValueError</span><span class="p">:</span> <span class="s2">"Invalid placeholder in string: line 1, col 1"</span> </pre></div> </section><section><h2>Which String Formatting Method Should I Use?</h2> <p>I totally get that having so much choice for how to format your strings in Python can feel very confusing. This is an excellent cue to bust out this handy flowchart infographic I’ve put together for you:</p> <figure><img alt="String Formatting in Python -- Flowchart" src="/blog/figures/python-string-formatting-flowchart.png" width="706" height="989"></figure> <p>This flowchart is based on the following rule of thumb that I apply when I’m writing Python:</p> </section><footer><h3>Dan’s Python String Formatting Rule of Thumb:</h3> <blockquote> <p><em>If your format strings are user-supplied, use <a href="#template-strings">Template Strings (#4)</a> to avoid security issues. Otherwise, use <a href="#interpolation">Literal String Interpolation (#3)</a> if you’re on Python 3.6+, and <a href="#new-style">“New Style” str.format (#2)</a> if you’re not.</em></p> </blockquote></footer></article>https://dbader.org/blog/python-string-formattingTue, 24 Jan 2017 00:00:00 GMTAssert Statements in Pythonhttps://dbader.org/blog/python-assert-tutorial<article><header><h1>Assert Statements in Python</h1> <p>How to use assertions to help automatically detect errors in your Python programs in order to make them more reliable and easier to debug.</p> </header><section><figure><img alt="" src="/blog/figures/python-assert.png" width="1280" height="720"></figure> </section><section><h2>What Are Assertions &amp; What Are They Good For?</h2> <p>Python’s assert statement is a debugging aid that tests a condition. If the condition is true, it does nothing and your program just continues to execute. But if the assert condition evaluates to false, it raises an <code>AssertionError</code> exception with an optional error message.</p> <p>The proper use of assertions is to inform developers about <strong>unrecoverable</strong> errors in a program. They’re not intended to signal expected error conditions, like “file not found”, where a user can take corrective action or just try again.</p> <p>Another way to look at it is to say that assertions are <strong>internal self-checks</strong> for your program. They work by declaring some conditions as <em>impossible</em> in your code. If one of these conditions doesn’t hold that means there’s a bug in the program.</p> <p>If your program is bug-free, these conditions will never occur. But if they <em>do</em> occur the program will crash with an assertion error telling you exactly which “impossible” condition was triggered. This makes it much easier to track down and fix bugs in your programs.</p> <p><strong>To summarize</strong>: Python’s assert statement is a debugging aid, not a mechanism for handling run-time errors. The goal of using assertions is to let developers find the likely root cause of a bug more quickly. An assertion error should never be raised unless there’s a bug in your program.</p> </section><section><h2>Assert in Python — An Example</h2> <p>Here’s a simple example so you can see where assertions might come in handy. I tried to give this some semblance of a real world problem you might actually encounter in one of your programs.</p> <p>Suppose you were building an online store with Python. You’re working to add a discount coupon functionality to the system and eventually write the following <code>apply_discount</code> function:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">apply_discount</span><span class="p">(</span><span class="n">product</span><span class="p">,</span> <span class="n">discount</span><span class="p">):</span> <span class="n">price</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">product</span><span class="p">[</span><span class="s1">'price'</span><span class="p">]</span> <span class="o">*</span> <span class="p">(</span><span class="mf">1.0</span> <span class="o">-</span> <span class="n">discount</span><span class="p">))</span> <span class="k">assert</span> <span class="mi">0</span> <span class="o">&lt;=</span> <span class="n">price</span> <span class="o">&lt;=</span> <span class="n">product</span><span class="p">[</span><span class="s1">'price'</span><span class="p">]</span> <span class="k">return</span> <span class="n">price</span> </pre></div> <p>Notice the <code>assert</code> statement in there? It will guarantee that, no matter what, discounted prices cannot be lower than $0 and they cannot be higher than the original price of the product.</p> <p>Let’s make sure this actually works as intended if we call this function to apply a valid discount:</p> <div class="codehilite"><pre><span></span><span class="c1">#</span> <span class="c1"># Our example product: Nice shoes for $149.00</span> <span class="c1">#</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">shoes</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'name'</span><span class="p">:</span> <span class="s1">'Fancy Shoes'</span><span class="p">,</span> <span class="s1">'price'</span><span class="p">:</span> <span class="mi">14900</span><span class="p">}</span> <span class="c1">#</span> <span class="c1"># 25% off -&gt; $111.75</span> <span class="c1">#</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">apply_discount</span><span class="p">(</span><span class="n">shoes</span><span class="p">,</span> <span class="mf">0.25</span><span class="p">)</span> <span class="mi">11175</span> </pre></div> <p>Alright, this worked nicely. Now, let’s try to apply some <em>invalid</em> discounts:</p> <div class="codehilite"><pre><span></span><span class="c1">#</span> <span class="c1"># A "200% off" discount:</span> <span class="c1">#</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">apply_discount</span><span class="p">(</span><span class="n">shoes</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">)</span> <span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span> <span class="n">File</span> <span class="s2">"&lt;input&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o">&lt;</span><span class="n">module</span><span class="o">&gt;</span> <span class="n">apply_discount</span><span class="p">(</span><span class="n">prod</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">)</span> <span class="n">File</span> <span class="s2">"&lt;input&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">4</span><span class="p">,</span> <span class="ow">in</span> <span class="n">apply_discount</span> <span class="k">assert</span> <span class="mi">0</span> <span class="o">&lt;=</span> <span class="n">price</span> <span class="o">&lt;=</span> <span class="n">product</span><span class="p">[</span><span class="s1">'price'</span><span class="p">]</span> <span class="ne">AssertionError</span> <span class="c1">#</span> <span class="c1"># A "-30% off" discount:</span> <span class="c1">#</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">apply_discount</span><span class="p">(</span><span class="n">shoes</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.3</span><span class="p">)</span> <span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span> <span class="n">File</span> <span class="s2">"&lt;input&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o">&lt;</span><span class="n">module</span><span class="o">&gt;</span> <span class="n">apply_discount</span><span class="p">(</span><span class="n">prod</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.3</span><span class="p">)</span> <span class="n">File</span> <span class="s2">"&lt;input&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">4</span><span class="p">,</span> <span class="ow">in</span> <span class="n">apply_discount</span> <span class="k">assert</span> <span class="mi">0</span> <span class="o">&lt;=</span> <span class="n">price</span> <span class="o">&lt;=</span> <span class="n">product</span><span class="p">[</span><span class="s1">'price'</span><span class="p">]</span> <span class="ne">AssertionError</span> </pre></div> <p>As you can see, trying to apply an invalid discount raises an <code>AssertionError</code> exception that points out the line with the violated assertion condition. If we ever encounter one of these errors while testing our online store it will be easy to find out what happened by looking at the traceback.</p> <p>This is the power of assertions, in a nutshell.</p> </section><section><h2>Python’s Assert Syntax</h2> <p>It’s always a good idea to study up on how a language feature is actually implemented in Python before you start using it. So let’s take a quick look at the <a href="https://docs.python.org/3/reference/simple_stmts.html#the-assert-statement" target="_blank">syntax for the assert statement according to the Python docs</a>:</p> <div class="codehilite"><pre><span></span><span class="n">assert_stmt</span> <span class="p">::</span><span class="o">=</span> <span class="s2">"assert"</span> <span class="n">expression1</span> <span class="p">[</span><span class="s2">","</span> <span class="n">expression2</span><span class="p">]</span> </pre></div> <p>In this case <code>expression1</code> is the condition we test, and the optional <code>expression2</code> is an error message that’s displayed if the assertion fails.</p> <p>At execution time, the Python interpreter transforms each assert statement into roughly the following:</p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="n">__debug__</span><span class="p">:</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">expression1</span><span class="p">:</span> <span class="k">raise</span> <span class="ne">AssertionError</span><span class="p">(</span><span class="n">expression2</span><span class="p">)</span> </pre></div> <p>You can use <code>expression2</code> to pass an <strong>optional error message</strong> that will be displayed with the <code>AssertionError</code> in the traceback. This can simplify debugging even further—for example, I’ve seen code like this:</p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="n">cond</span> <span class="o">==</span> <span class="s1">'x'</span><span class="p">:</span> <span class="n">do_x</span><span class="p">()</span> <span class="k">elif</span> <span class="n">cond</span> <span class="o">==</span> <span class="s1">'y'</span><span class="p">:</span> <span class="n">do_y</span><span class="p">()</span> <span class="k">else</span><span class="p">:</span> <span class="k">assert</span> <span class="bp">False</span><span class="p">,</span> <span class="p">(</span><span class="s2">"This should never happen, but it does occasionally. "</span> <span class="s2">"We're currently trying to figure out why. "</span> <span class="s2">"Email dbader if you encounter this in the wild."</span><span class="p">)</span> </pre></div> <p>Is this ugly? Well, yes. But it’s definitely a valid and helpful technique if you’re faced with a <a href="https://en.wikipedia.org/wiki/Heisenbug" target="_blank">heisenbug-type issue</a> in one of your applications. 😉</p> </section><section><h2>Common Pitfalls With Using Asserts in Python</h2> <p>Before you move on, there are two important caveats with using assertions in Python that I’d like to call out.</p> <p>The first one has to do with introducing security risks and bugs into your applications, and the second one is about a syntax quirk that makes it easy to write <em>useless</em> assertions.</p> <p>This sounds (and potentially is) pretty horrible, so you might at least want to skim these two caveats or read their summaries below.</p> </section><section><h2>Caveat #1 – Don’t Use Asserts for Data Validation</h2> <p><strong>Asserts can be turned off globally in the Python interpreter. Don’t rely on assert expressions to be executed for data validation or data processing.</strong></p> <p>The biggest caveat with using asserts in Python is that <em><a href="https://docs.python.org/3/library/constants.html#__debug__" target="_blank">assertions can be globally disabled</a> with the <code>-O</code> and <code>-OO</code> command line switches, as well as the <code>PYTHONOPTIMIZE</code> environment variable in CPython</em>.</p> <p>This turns any assert statement into a null-operation: the assertions simply get compiled away and won’t be evaluated, which means that none of the conditional expressions will be executed.</p> <p>This is an intentional design decision used similarly by many other programming languages. As a side-effect it becomes extremely dangerous to use assert statements as a quick and easy way to validate input data.</p> <p>Let me explain—if your program uses asserts to check if a function argument contains a “wrong” or unexpected value this can backfire quickly and lead to bugs or security holes.</p> <p>Let’s take a look at a simple example. Imagine you’re building an online store application with Python. Somewhere in your application code there’s a function to delete a product as per a user’s request:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">delete_product</span><span class="p">(</span><span class="n">product_id</span><span class="p">,</span> <span class="n">user</span><span class="p">):</span> <span class="k">assert</span> <span class="n">user</span><span class="o">.</span><span class="n">is_admin</span><span class="p">(),</span> <span class="s1">'Must have admin privileges to delete'</span> <span class="k">assert</span> <span class="n">store</span><span class="o">.</span><span class="n">product_exists</span><span class="p">(</span><span class="n">product_id</span><span class="p">),</span> <span class="s1">'Unknown product id'</span> <span class="n">store</span><span class="o">.</span><span class="n">find_product</span><span class="p">(</span><span class="n">product_id</span><span class="p">)</span><span class="o">.</span><span class="n">delete</span><span class="p">()</span> </pre></div> <p>Take a close look at this function. What happens if assertions are disabled?</p> <p>There are two serious issues in this three-line function example, caused by the incorrect use of assert statements:</p> <ol> <li><strong>Checking for admin privileges with an assert statement is dangerous.</strong> If assertions are disabled in the Python interpreter, this turns into a null-op. Therefore <em>any user can now delete products</em>. The privileges check doesn’t even run. This likely introduces a security problem and opens the door for attackers to destroy or severely damage the data in your customer’s or company’s online store. Not good.</li> <li><strong>The <code>product_exists()</code> check is skipped when assertions are disabled.</strong> This means <code>find_product()</code> can now be called with invalid product ids—which could lead to more severe bugs depending on how our program is written. In the worst case this could be an avenue for someone to launch Denial of Service attacks against our store. If the store app crashes if we attempt to delete an unknown product, it might be possible for an attacker to bombard it with invalid delete requests and cause an outage.</li> </ol> <p>How might we avoid these problems? The answer is to not use assertions to do data validation. Instead we could do our validation with regular if-statements and raise validation exceptions if necessary. Like so:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">delete_product</span><span class="p">(</span><span class="n">product_id</span><span class="p">,</span> <span class="n">user</span><span class="p">):</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">user</span><span class="o">.</span><span class="n">is_admin</span><span class="p">():</span> <span class="k">raise</span> <span class="n">AuthError</span><span class="p">(</span><span class="s1">'Must have admin privileges to delete'</span><span class="p">)</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">store</span><span class="o">.</span><span class="n">product_exists</span><span class="p">(</span><span class="n">product_id</span><span class="p">):</span> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'Unknown product id'</span><span class="p">)</span> <span class="n">store</span><span class="o">.</span><span class="n">find_product</span><span class="p">(</span><span class="n">product_id</span><span class="p">)</span><span class="o">.</span><span class="n">delete</span><span class="p">()</span> </pre></div> <p>This updated example also has the benefit that instead of raising unspecific <code>AssertionError</code> exceptions, it now raises semantically correct exceptions like <code>ValueError</code> or <code>AuthError</code> (which <a href="python-custom-exceptions">we’d have to define ourselves</a>).</p> </section><section><h2>Caveat #2 – Asserts That Never Fail</h2> <p>It’s easy to accidentally write Python assert statements that always evaluate to true. I’ve been bitten by this myself in the past. I wrote <a href="catching-bogus-python-asserts">a longer article about this specific issue you can check out by clicking here</a>.</p> <p>Alternatively, here’s the executive summary:</p> <p><strong>When you pass a tuple as the first argument in an <code>assert</code> statement, the assertion always evaluates as true and therefore never fails.</strong></p> <p>For example, this assertion will never fail:</p> <div class="codehilite"><pre><span></span><span class="k">assert</span><span class="p">(</span><span class="mi">1</span> <span class="o">==</span> <span class="mi">2</span><span class="p">,</span> <span class="s1">'This should fail'</span><span class="p">)</span> </pre></div> <p>This has to do with non-empty tuples always being truthy in Python. If you pass a tuple to an assert statement it leads to the assert condition to always be true—which in turn leads to the above assert statement being useless because it can never fail and trigger an exception.</p> <p>It’s relatively easy to accidentally write bad multi-line asserts due to this unintuitive behavior. This quickly leads to broken test cases that give a false sense of security in our test code. Imagine you had this assertion somewhere in your unit test suite:</p> <div class="codehilite"><pre><span></span><span class="k">assert</span> <span class="p">(</span> <span class="n">counter</span> <span class="o">==</span> <span class="mi">10</span><span class="p">,</span> <span class="s1">'It should have counted all the items'</span> <span class="p">)</span> </pre></div> <p>Upon first inspection this test case looks completely fine. However, this test case would never catch an incorrect result: it always evaluates to <code>True</code>, regardless of the state of the counter variable.</p> <p>Like I said, it’s rather easy to shoot yourself in the foot with this (mine still hurts). Luckily, there are some countermeasures you can apply to prevent this syntax quirk from causing trouble:</p> <p><a href="catching-bogus-python-asserts">&gt;&gt; Read the full article on bogus assertions to get the dirty details.</a></p> </section><section><h2>Python Assertions — Summary</h2> <p>Despite these caveats I believe that Python’s assertions are a powerful debugging tool that’s frequently underused by Python developers.</p> <p>Understanding how assertions work and when to apply them can help you write more maintainable and easier to debug Python programs. It’s a great skill to learn that will help bring your Python to the next level and make you a more well-rounded Pythonista.</p></section><footer></footer></article>https://dbader.org/blog/python-assert-tutorialWed, 18 Jan 2017 00:00:00 GMTComprehending Python’s Comprehensionshttps://dbader.org/blog/list-dict-set-comprehensions-in-python<article><header><h1>Comprehending Python’s Comprehensions</h1> <p>One of my favorite features in Python are list comprehensions. They can seem a bit arcane at first but when you break them down they are actually a very simple construct.</p> </header><section><figure><img alt="Comprehending Python's List, Dict, Set Comprehensions" src="/blog/figures/comprehending-comprehensions.png" width="1280" height="720"></figure> <p>The key to understanding list comprehensions is that they’re just <code>for</code>-loops over a collection expressed in a more terse and compact syntax. Let’s take the following list comprehension as an example:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">squares</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span> <span class="o">*</span> <span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)]</span> </pre></div> <p>It computes a list of all integer square numbers from 0 to 9:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">squares</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">9</span><span class="p">,</span> <span class="mi">16</span><span class="p">,</span> <span class="mi">25</span><span class="p">,</span> <span class="mi">36</span><span class="p">,</span> <span class="mi">49</span><span class="p">,</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">81</span><span class="p">]</span> </pre></div> <p>If we wanted to build the same list using a plain <code>for</code>-loop we’d probably write something like this:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">squares</span> <span class="o">=</span> <span class="p">[]</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span> <span class="o">...</span> <span class="n">squares</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">x</span> <span class="o">*</span> <span class="n">x</span><span class="p">)</span> </pre></div> <p>That’s a pretty straightforward loop, right? If you try and generalize some of this structure you might end up with a template similar to this:</p> <div class="codehilite"><pre><span></span><span class="p">(</span><span class="n">values</span><span class="p">)</span> <span class="o">=</span> <span class="p">[</span> <span class="p">(</span><span class="n">expression</span><span class="p">)</span> <span class="k">for</span> <span class="p">(</span><span class="n">item</span><span class="p">)</span> <span class="ow">in</span> <span class="p">(</span><span class="n">collection</span><span class="p">)</span> <span class="p">]</span> </pre></div> <p>The above list comprehension is equivalent to the following plain <code>for</code>-loop:</p> <div class="codehilite"><pre><span></span><span class="p">(</span><span class="n">values</span><span class="p">)</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="p">(</span><span class="n">item</span><span class="p">)</span> <span class="ow">in</span> <span class="p">(</span><span class="n">collection</span><span class="p">):</span> <span class="p">(</span><span class="n">values</span><span class="p">)</span><span class="o">.</span><span class="n">append</span><span class="p">(</span> <span class="p">(</span><span class="n">expression</span><span class="p">)</span> <span class="p">)</span> </pre></div> <p>Again, a fairly simple cookiecutter pattern you can apply to most for loops. Now there’s one more useful element we need to add to this template, and that is element filtering with <em>conditions</em>.</p> <p>List comprehensions can filter values based on some arbitrary condition that decides whether or not the resulting value becomes a part of the output list. Here’s an example:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">even_squares</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span> <span class="o">*</span> <span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="k">if</span> <span class="n">x</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">]</span> </pre></div> <p>This list comprehension will compute a list of the squares of all even integers from 0 to 9.</p> <p>If you’re not familiar with what the <em>modulo</em> (<code>%</code>) operator does—it returns the remainder after division of one number by another. In this example the <code>%</code>-operator gives us an easy way to test if a number is even by checking the remainder after we divide the number by 2.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">even_squares</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">16</span><span class="p">,</span> <span class="mi">36</span><span class="p">,</span> <span class="mi">64</span><span class="p">]</span> </pre></div> <p>Similarly to the first example, this new list comprehension can be transformed into an equivalent <code>for</code>-loop:</p> <div class="codehilite"><pre><span></span><span class="n">even_squares</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span> <span class="k">if</span> <span class="n">x</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="n">even_squares</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">x</span> <span class="o">*</span> <span class="n">x</span><span class="p">)</span> </pre></div> <p>Let’s try and generalize the above <em>list comprehension to for-loop</em> transform again. This time we’re going to add a filter condition to our template to decide which values end up in the resulting list.</p> <p>Here’s the list comprehension template:</p> <div class="codehilite"><pre><span></span><span class="n">values</span> <span class="o">=</span> <span class="p">[</span><span class="n">expression</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">collection</span> <span class="k">if</span> <span class="n">condition</span><span class="p">]</span> </pre></div> <p>And we can transform this list comprehension into a <em>for</em>-loop with the following pattern:</p> <div class="codehilite"><pre><span></span><span class="n">values</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">collection</span><span class="p">:</span> <span class="k">if</span> <span class="n">condition</span><span class="p">:</span> <span class="n">values</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">expression</span><span class="p">)</span> </pre></div> <p>Again, this is a straightforward transformation—we simply apply our cookiecutter pattern again. I hope this dispelled some of the “magic” in how list comprehensions work. They’re really quite a useful tool.</p> <p>Before you move on I want to point out that Python not only supports <em>list</em> comprehensions but also has similar syntax for <em>sets</em> and <em>dictionaries</em>.</p> <p>Here’s what a <em>set comprehension</em> looks like:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="p">{</span> <span class="n">x</span> <span class="o">*</span> <span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="o">-</span><span class="mi">9</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="p">}</span> <span class="nb">set</span><span class="p">([</span><span class="mi">64</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">36</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">49</span><span class="p">,</span> <span class="mi">9</span><span class="p">,</span> <span class="mi">16</span><span class="p">,</span> <span class="mi">81</span><span class="p">,</span> <span class="mi">25</span><span class="p">,</span> <span class="mi">4</span><span class="p">])</span> </pre></div> <p>And this is a <em>dict comprehension</em>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="p">{</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span> <span class="o">*</span> <span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span> <span class="p">}</span> <span class="p">{</span><span class="mi">0</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">:</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">3</span><span class="p">:</span> <span class="mi">9</span><span class="p">,</span> <span class="mi">4</span><span class="p">:</span> <span class="mi">16</span><span class="p">}</span> </pre></div> <p>Both are useful tools in practice. There’s one caveat to Python’s comprehensions—as you get more proficient at using them it becomes easier and easier to write code that’s difficult to read. If you’re not careful you might have to deal with monstrous list, set, dict comprehensions soon. Remember, too much of a good thing is usually a bad thing.</p> <p>After much chagrin I’m personally drawing the line at one level of nesting for comprehensions. I found that in most cases it’s better (as in “more readable” and “easier to maintain”) to use <em>for</em>-loops beyond that point.</p> </section><section><h2>📺🐍 Learn More With This Video Tutorial</h2> <p>I recorded a step-by-step video tutorial that teaches you how list comprehensions work in Python to go along with the article. Watch it embedded below or <a href="https://www.youtube.com/watch?v=1HlyKKiGg-4" target="_blank">on my YouTube channel</a>:</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/1HlyKKiGg-4?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> </section><section><h2>Key Takeaways</h2> <ul> <li>Comprehensions are a key feature in Python. Understanding and applying them will make your code much more Pythonic.</li> <li>Comprehensions are just fancy syntax for a simple <code>for</code>-loop pattern. Once you understand the pattern, you’ll develop an intuitive understanding for comprehensions.</li> <li>There are more than just list comprehensions.</li> </ul></section><footer></footer></article>https://dbader.org/blog/list-dict-set-comprehensions-in-pythonWed, 11 Jan 2017 00:00:00 GMTThe Difference Between “is” and “==” in Pythonhttps://dbader.org/blog/difference-between-is-and-equals-in-python<article><header><h1>The Difference Between “is” and “==” in Python</h1> <p>Python has two operators for equality comparisons, “is” and “==” (equals). In this article I’m going to teach you the difference between the two and when to use each with a few simple examples.</p> </header><section><figure><img alt='When to use "==" and "is" in Python' src="/blog/figures/python-is-vs-equals.png" width="1280" height="720"></figure> <p>When I was a kid, our neighbors had two twin cats.</p> <p>Both cats looked seemingly identical—same charcoal fur, same piercing green eyes. Some personality quirks aside, you just couldn’t tell them apart just from looking at them. But of course they were two different cats, two separate beings, even though they looked exactly the same.</p> <p>There’s a difference in meaning between <em>equal</em> and <em>identical</em>. And this difference is important when you want to understand how Python’s <code>is</code> and <code>==</code> comparison operators behave.</p> <p><strong>The <code>==</code> operator compares by checking for <u>equality</u></strong>: If these cats were Python objects and we’d compare them with the <code>==</code> operator, we’d get “both cats are equal” as an answer.</p> <p><strong>The <code>is</code> operator, however, compares <u>identities</u></strong>: If we compared our cats with the <code>is</code> operator, we’d get “these are two different cats” as an answer.</p> <p>But before I get all tangled up in this ball of twine of a cat analogy, let’s take a look at some real Python code.</p> <p>First, we’ll create a new list object and name it <code>a</code>, and then define another variable <code>b</code> that points to the same list object:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">b</span> <span class="o">=</span> <span class="n">a</span> </pre></div> <p>Let’s inspect these two variables. We can see they point to identical looking lists:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">b</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> </pre></div> <p>Because the two list objects look the same we’ll get the expected result when we compare them for equality using the <code>==</code> operator:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="o">==</span> <span class="n">b</span> <span class="bp">True</span> </pre></div> <p>However, that doesn’t tell us whether <code>a</code> and <code>b</code> are actually pointing to the same object. Of course, we know they do because we assigned them earlier, but suppose we didn’t know—how might we find out?</p> <p>The answer is comparing both variables with the <code>is</code> operator. This confirms both variables are in fact pointing to one list object:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="ow">is</span> <span class="n">b</span> <span class="bp">True</span> </pre></div> <p>Let’s see what happens when we create an identical copy of our list object. We can do that by calling <code>list()</code> on the existing list to create a copy we’ll name <code>c</code>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">c</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> </pre></div> <p>Again you’ll see that the new list we just created looks identical to the list object pointed to by <code>a</code> and <code>b</code>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">c</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> </pre></div> <p>Now this is where it gets interesting—let’s compare our list copy <code>c</code> with the initial list <code>a</code> using the <code>==</code> operator. What answer do you expect to see?</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="o">==</span> <span class="n">c</span> <span class="bp">True</span> </pre></div> <p>Okay, I hope this was what you expected. What this result tells us is that <code>c</code> and <code>a</code> have the same contents. They’re considered equal by Python. But are they actually pointing to the same object? Let’s find out with the <code>is</code> operator:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="ow">is</span> <span class="n">c</span> <span class="bp">False</span> </pre></div> <p>Boom—this is where we get a different result. Python is telling us that <code>c</code> and <code>a</code> are pointing to two different objects, even though their contents might be the same.</p> <p>So, to recap let’s try and break the difference between <code>is</code> and <code>==</code> down to two short definitions:</p> <ul> <li> <p>An <code>is</code> expression evaluates to <code>True</code> if two variables point to the same (identical) object.</p> </li> <li> <p>An <code>==</code> expression evaluates to <code>True</code> if the objects referred to by the variables are equal (have the same contents).</p> </li> </ul> <p>Just remember, think of twin cats (dogs should work, too) whenever you need to decide between using <code>is</code> and <code>==</code> in Python. You’ll be fine.</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/CZ8bZPqtwU0?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div></section><footer></footer></article>https://dbader.org/blog/difference-between-is-and-equals-in-pythonWed, 28 Dec 2016 00:00:00 GMTHow to Speak at a Python Conferencehttps://dbader.org/blog/how-to-speak-at-a-python-conference<article><header><h1>How to Speak at a Python Conference</h1> <p>My tutorial on how you can get a first-time speaking gig at a tech conference like PyCon as a software developer.</p> </header><section><figure><img alt="How To Speak at a Python Conference" src="/blog/figures/how-to-speak-python-conference-header.png" width="1280" height="720"></figure> <p>I gave <a href="https://www.youtube.com/watch?v=Di50Y5b20p8" target="_blank">a talk at PyCon Germany this year</a> and I was just chatting with my friend Sergei who wants to get into presenting at tech conferences and is looking for a way to get started. Here’s Segei’s question:</p> <blockquote> <p>My goal for now it to get ready for the spring season of conferences, and pick one where I could present something.</p> <p>I want to try presenting stuff on conferences, as it is a great public speaking experience for me and… free attendance of a good conference :)</p> <p><strong>Where do you start if you want to present something?</strong></p> </blockquote> <p>If you’re thinking about speaking at a tech conference that’s a fantastic idea! It’s a lot of fun (it’s a lot of work, too)—but it’s totally worth it.</p> <p>It’s great for getting in touch with people, making new connections, finding a new or better job, and to improve your public speaking skills. Plus, you get tagged as a speaker on your conference badge—a great conversation starter 😜</p> </section><section><h2>Alright, I’m sold—How do I become a first-time conference speaker?</h2> <p>For most tech conferences you can become a speaker by applying with a talk proposal a few months before the conference takes place.</p> <p>Basically, all the legwork is on you. You’re picking the topic and pitching it to the conference organizers and they decide which talks they accept. If you’ve built a bit of a reputation for yourself then conference organizers might contact you and ask you to speak at their conference. But for first time speakers that’s unlikely.</p> <p>The key thing here is to apply early enough and before the “call for proposals” deadline. The deadline usually ends well in advance before the conference takes place. It absolutely helps and increases your chances to get in if you plan ahead of time.</p> </section><section><h2>Where can I find upcoming conferences to speak at?</h2> <p>For conferences related to Python there’s the Python Events calendar at:</p> <ul> <li><a href="http://www.pycon.org/" target="_blank">http://www.pycon.org/</a>; and</li> <li><a href="https://www.python.org/events/" target="_blank">https://www.python.org/events/</a>.</li> </ul> <p>Use the events calendar to find a conference that looks interesting. Then check out the conference website to see what the application process is like and when the call for proposals (submission deadline) ends.</p> <p>At this point you should know which event you want to speak at, who to send your talk proposal to, and when the submission deadline is. The next step is to put together and send in your talk proposal.</p> </section><section><h2>How do I write a good talk proposal?</h2> <p>The most important thing you need to realize about talk proposals is that <em>they’re all about the organizers</em>. You need to put yourself in the shoes of the organizers when you’re writing the proposal—that’s the biggest trick to getting your first talk accepted.</p> <p>Your goal is to convince the organizers that:</p> <ul> <li>inviting you as a speaker won’t embarrass them; and</li> <li>their audience is likely going to enjoy your talk.</li> </ul> <p>Keep this in mind while writing your proposal and it’ll increase your chances to get your talk accepted immensely. Whatever information you can give to convince the organizers your talk will do these two things for them helps.</p> <p>For example, be sure to mention the intended audience for your talk and to share your previous speaking experience (meetup/work/university presentations). Also mention your own expertise on the talk’s topic.</p> <p>If you don’t have previous public speaking experience it can be a bit tough to get into conferences at first. To “hack” this system you could also point people to a YouTube video tutorial you recorded. That way the organizers can get a sense for your presentation skills (I always include a link to my <a href="/screencasts">YouTube channel</a> for that reason.)</p> <p>To make this a bit easier for you I’ve put together a conference proposal template you can use. <a href="https://www.getdrip.com/forms/46062574/submissions/new" data-drip-show-form="46062574"><strong>Click here and I’ll email it over to you.</strong></a></p> <p>My template will help you cover all of the relevant info. I used this template to get a talk into PyCon Germany not long ago, for example. Good luck 😃</p> </section><section><h2>What happens after I send in my proposal?</h2> <p>After you’ve sent in your proposal all you need to do is wait for an answer. If you get accepted—congratulations, you’ll be speaking at a conference soon!</p> <p>Usually you won’t get feedback when your talk gets rejected (it’s kind of similar to a job interview that way). But don’t be too upset if your talk is rejected! You can always try again with the same talk at a different conference.</p> <p>My gut feeling is that the biggest factor in whether or not your talk gets accepted is “quality indicators” like your previous speaking experience—so be sure to talk about that in your proposal.</p> <p>It’ll get much easier to get your talks accepted once you’ve spoken a few times.</p> </section><section><h2>Can I attend for free if I’m speaking?</h2> <p>Most non-commercial tech conferences don’t waive the attendance fee even if you’re a speaker. I believe it’s different with keynote speakers but if you’re a regular speaker at a small to medium size conference you’ll often pay the full fee or a slightly reduced fee. Still worth it though 😊</p> </section><section><h2>Got any more tips?</h2> <p>Sure do! Watch this video I recorded to get some extra tips on how to bootstrap your speaking career (there’s more videos on <a href="https://www.youtube.com/playlist?list=PLP8GkvaIxJP37FpJZcj_NhP9-r1a_AkqJ" target="_blank">this playlist</a>):</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/0-BkCkAiso8?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> </section><section><h2>Extra Resources</h2> <ul> <li><a href="https://www.getdrip.com/forms/46062574/submissions/new" data-drip-show-form="46062574">PDF talk proposal template (got me accepted at PyCon Germany)</a></li> <li><a href="https://www.python.org/events/" target="_blank">Python Events Calendar</a></li> <li><a href="http://katemats.com/public-speaking-at-a-conference/" target="_blank">Kate Matsudaira: “So you want to speak at a conference”</a></li> <li><a href="http://wunder.schoenaberselten.com/2016/02/16/how-to-prepare-and-write-a-tech-conference-talk/" target="_blank">Lena Reinhard: “How to prepare and write a tech conference talk”</a></li> <li><a href="https://medium.com/@karen_meep/how-to-start-speaking-in-tech-conferences-f64a9f3a84a6" target="_blank">Karen Meep: “How to start speaking (in tech conferences)”</a></li> <li><a href="https://www.ashedryden.com/blog/what-you-need-to-know-about-speaking-at-conferences" target="_blank">Ashe Dryden: “What you need to know about speaking at conferences”</a></li> </ul></section><footer></footer></article>https://dbader.org/blog/how-to-speak-at-a-python-conferenceTue, 13 Dec 2016 00:00:00 GMTCool new features in Python 3.6https://dbader.org/blog/cool-new-features-in-python-3-6<article><header><h1>Cool new features in Python 3.6</h1> <p>Python 3.6 adds a couple of new features and improvements that’ll affect the day to day work of Python coders. In this article I’ll give you an overview of the new features I found the most interesting.</p> </header><section><figure><img alt="" src="/blog/figures/python-36-features.png" width="1280" height="720"></figure> </section><section><h2>Improved numeric literals</h2> <p>This is a syntactic tweak that makes numeric literals easier to read. You can now add underscores to numbers in order to group them to your liking. This is handy for expressing large quantities or constants in binary or hexadecimal:</p> <div class="codehilite"><pre><span></span>&gt;&gt;&gt; <span class="nv">six_figures</span> <span class="o">=</span> 100_000 &gt;&gt;&gt; six_figures <span class="m">100000</span> </pre></div> <div class="codehilite"><pre><span></span>&gt;&gt;&gt; <span class="nv">programmer_error</span> <span class="o">=</span> 0xbad_c0ffee &gt;&gt;&gt; <span class="nv">flags</span> <span class="o">=</span> 0b_0111_0101_0001_0101 </pre></div> <p>Remember, this change doesn’t introduce any new semantics. It’s just a way to represent numeric literals differently in your source code. A small but neat addition.</p> <p>You can learn more about this change in <a href="https://www.python.org/dev/peps/pep-0515/" target="_blank">PEP 515</a>.</p> </section><section><h2>String interpolation</h2> <p>Python 3.6 adds yet another way to format strings called <em>Formatted String Literals</em>. This new way of formatting strings lets you use embedded Python expressions inside string constants. Here’s are two simple examples to give you a feel for the feature:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">'Bob'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">f</span><span class="s1">'Hello, </span><span class="si">{name}</span><span class="s1">!'</span> <span class="s1">'Hello, Bob!'</span> </pre></div> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">5</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">b</span> <span class="o">=</span> <span class="mi">10</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">f</span><span class="s1">'Five plus ten is {a + b} and not {2 * (a + b)}.'</span> <span class="s1">'Five plus ten is 15 and not 30.'</span> </pre></div> <p>String literals also support the existing format string syntax of the <code>str.format()</code> method. That allows you to do things like:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">error</span> <span class="o">=</span> <span class="mi">50159747054</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">f</span><span class="s1">'Programmer Error: </span><span class="si">{error:#x}</span><span class="s1">'</span> <span class="s1">'Programmer Error: 0xbadc0ffee'</span> </pre></div> <p>Python’s new Formatted String Literals are similar to the <a href="https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Template_literals" target="_blank">JavaScript Template Literals added in ES2015/ES6</a>. I think they’re quite a nice addition to the language and I look forward to using them in my day to day work.</p> <p>You can learn more about this change in <a href="https://www.python.org/dev/peps/pep-0498/" target="_blank">PEP 498</a>.</p> </section><section><h2>Type annotations for variables</h2> <p>Starting with Python 3.5 you could add type annotations to functions and methods:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">my_add</span><span class="p">(</span><span class="n">a</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">b</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span> <span class="o">...</span> <span class="k">return</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span> </pre></div> <p>In Python 3.6 you can use a syntax similar to type annotations for function arguments to type-hint standalone variables:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">python_version</span> <span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mf">3.6</span> </pre></div> <p>Nothing has changed in terms of the semantics–CPython simply records the type as a type annotation but doesn’t validate or check types in any way.</p> <p>Type-checking is purely optional and you’ll need <a href="https://www.youtube.com/watch?v=2xWhaALHTvU" target="_blank">a tool like Mypy</a> for that, which basically works like a <a href="/blog/python-code-linting">code linter</a>.</p> <p>You can learn more about this change in <a href="https://www.python.org/dev/peps/pep-0526/" target="_blank">PEP 526</a>.</p> </section><section><h2>Watch a video summary of the best new features in Python 3.6</h2> <p></p><div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/klKdMxjDaa0?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> </section><section><h2>Other notable changes</h2> <p>I think Python 3.6 will be an interesting release. There are many more interesting additions and improvements that are worth checking out. You can learn more about them in the links below or by reading the <a href="https://docs.python.org/3.6/whatsnew/3.6.html" target="_blank">official “What’s New In Python 3.6” announcement</a>.</p> <ul> <li><a href="https://docs.python.org/3.6/whatsnew/3.6.html#whatsnew36-pep525" target="_blank">Syntax for asynchronous generators</a></li> <li><a href="https://docs.python.org/3.6/whatsnew/3.6.html#whatsnew36-pep530" target="_blank">Syntax for asynchronous comprehensions</a></li> <li><a href="https://docs.python.org/3.6/whatsnew/3.6.html#whatsnew36-compactdict" target="_blank">Dictionaries are faster and use 20% to 25% less memory</a></li> <li><a href="https://docs.python.org/3.6/whatsnew/3.6.html#whatsnew36-typing" target="_blank">The typing module is considered a stable API</a></li> <li><a href="https://docs.python.org/3.6/library/secrets.html#module-secrets" target="_blank">A new “secrets” module for generating cryptographically secure authentication tokens</a></li> <li><a href="https://docs.python.org/3.6/whatsnew/3.6.html" target="_blank">…and more</a></li> </ul></section><footer></footer></article>https://dbader.org/blog/cool-new-features-in-python-3-6Thu, 08 Dec 2016 00:00:00 GMT3 Reasons why you need a programming bloghttps://dbader.org/blog/3-reasons-why-you-need-a-programming-blog<article><header><h1>3 Reasons why you need a programming blog</h1> <p>One of the best things I ever did for my dev career: A little story and three reasons why you should start a programming portfolio website right now.</p> </header><section><figure><img alt="" src="/blog/figures/programming-blog-reasons.png" width="1280" height="720"></figure> <p>At PyCon Germany I chatted with Astrid, a freelance Python (Django) developer looking for ways to improve her career and to find more contracts.</p> <p>Astrid seemed quite frustrated with her situation—it was tough for her to get the contracts and jobs she really wanted.</p> <p>Often when she sent out her resume for more desirable gigs she wouldn’t even receive an answer. It sounded like she was stuck with a certain quality of clients and couldn’t really push past that invisible barrier.</p> <p>I always love to help a sister (or brother) out and went into full-on diagnosing mode. Usually I just end up spouting unsolicited advice in these situations but with Astrid I think I actually hit the nail on the head… 😉</p> <p>Eventually I asked Astrid if she had a website or blog as a “programmer portfolio” of sorts.</p> <p>She did not.</p> <p>And I think that was a BIG mistake –</p> <p>Looking back I’d say starting my personal website at dbader.org was probably the best thing I ever did for my programming career:</p> </section><section><h2>Reason #1: Employers loved it–it made it much easier to get interviews</h2> <p>In fact once I had my website up for a while companies started contacting me through it. And they were no longer the crappy recruiter emails I got through LinkedIn, but from managers and dev leads at companies that I found actually interesting.</p> </section><section><h2>Reason #2: It was easier to get started than I thought</h2> <p>I launched my site with just 3 articles I wrote over the holidays hanging out with my family one year. I was surprised to find I got more (not less) traffic over time even though I didn’t post new stuff constantly. More people started linking to my posts and they ranked higher in Google (also search engines seem to favor content that has been around for a while). It was incredibly fun to see that growth and to find new ways of reaching developers.</p> </section><section><h2>Reason #3: It put me in touch with so many fine folks (like you!)</h2> <p>Most of the places I lived in didn’t have strong software dev / meetup communities. Starting a website was a fantastic way to make friends with other developers around the world and to exchange ideas.</p> </section><section><h2>How you can get started today</h2> <p>I know it seems super difficult to get everything set up in the beginning. And the work involved can seem kind of boring at first… “it’s just a website”.</p> <p>What finally got me started with setting up my own website was turning it into a programming exercise.</p> <p>Instead of using a pre-fab framework like Wordpress I wrote my own Python framework for generating the website.</p> <p>I figured even if I wouldn’t follow through with the site I’d learn some web development skills in the process… And this was exactly true 😃</p> <p>Putting myself in Astrid’s shoes again I really believe every software developer should have a personal website. The time investment is so small in comparison to the awesome benefits and opportunities it can generate for you.</p> <p>If you’re sold on the idea of starting a programming blog but you don’t know how to go about it yet then check out this video I created for you.</p> <p>In the video embedded below I’m going over my own website as an example and how it looks very different today compared to when I started it in 2012.</p> <p>It doesn’t take much to get started with your own programming blog or portfolio website and the benefits can be huge.</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/wqpBeGeOMgM?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div></section><footer></footer></article>https://dbader.org/blog/3-reasons-why-you-need-a-programming-blogWed, 30 Nov 2016 00:00:00 GMTA Python Riddle: The Craziest Dict Expression in the Westhttps://dbader.org/blog/python-mystery-dict-expression<article><header><h1>A Python Riddle: The Craziest Dict Expression in the West</h1> <p>Let’s pry apart this slightly unintuitive Python dictionary expression to find out what’s going on in the uncharted depths of the Python interpreter.</p> </header><section><figure><img alt="" src="/blog/figures/python-dict-riddle.png" width="661" height="371"></figure> <p>A while ago I shared this Python one-liner as a “Python riddle” <a href="https://twitter.com/dbader_org/status/784108741373427718" target="_blank">on Twitter</a> and it got some interesting reactions:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="p">{</span><span class="bp">True</span><span class="p">:</span> <span class="s1">'yes'</span><span class="p">,</span> <span class="mi">1</span><span class="p">:</span> <span class="s1">'no'</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">:</span> <span class="s1">'maybe'</span><span class="p">}</span> </pre></div> <p>Take a quick moment to think about what this dict expression will evaluate to.</p> <p>I’ll wait here 😃</p> <p>Ok, ready?</p> <p>It’ll evaluate to this dictionary:</p> <div class="codehilite"><pre><span></span><span class="p">{</span><span class="bp">True</span><span class="p">:</span> <span class="s1">'maybe'</span><span class="p">}</span> </pre></div> <p>I’ll admit I was pretty surprised about this result the first time I saw it. But it all makes sense when you investigate what happens step by step.</p> </section><section><h2>Where baby dictionaries come from</h2> <p>Let’s think about why we get this (I want to say “slightly unintuitive”) result:</p> <p>When Python processes our dictionary expression it first constructs a new empty dictionary object; and then assigns the keys and values to it in the order given in the dict expression.</p> <p>When we break it down our dict expression is equivalent to this sequence of statements:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">xs</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">xs</span><span class="p">[</span><span class="bp">True</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'yes'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">xs</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'no'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">xs</span><span class="p">[</span><span class="mf">1.0</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'maybe'</span> </pre></div> <p>Now this gets a lot more interesting when we realize that <em>all keys</em> we’re using in this example are considered to be <em>equal</em> by Python:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="bp">True</span> <span class="o">==</span> <span class="mi">1</span> <span class="o">==</span> <span class="mf">1.0</span> <span class="bp">True</span> </pre></div> <p>Okay, but–wait a minute here. We can intuitively accept that <code>1.0 == 1</code>, but why would <code>True</code> be considered equal to <code>1</code> as well?</p> <p>The answer to that is Python treats <code>bool</code> as a subclass of <code>int</code><sup id="fnref:bool-indexes"><a class="footnote-ref" href="#fn:bool-indexes" rel="footnote">1</a></sup>. This is the case in <a href="https://docs.python.org/2/reference/datamodel.html#the-standard-type-hierarchy" target="_blank">Python 2</a> and <a href="https://docs.python.org/3/reference/datamodel.html#the-standard-type-hierarchy" target="_blank">Python 3</a>:</p> <blockquote> <p>The Boolean type is a subtype of the integer type, and Boolean values behave like the values 0 and 1, respectively, in almost all contexts, the exception being that when converted to a string, the strings “False” or “True” are returned, respectively.</p> </blockquote> <p>So, as far as Python is concerned, <code>True</code>, <code>1</code>, and <code>1.0</code> all represent <em>the same dictionary key</em>. As the interpreter evaluates the dictionary expression it repeatedly overwrites the value for the key <code>True</code>.</p> <p>But why do we still get <code>True</code> as the key in the final dictionary? Shouldn’t the key also change to <code>1.0</code> at the end through the repeated assignments?</p> <p>To explain this outcome we need to know that Python doesn’t replace the object instance for a key when a new value is assigned to it:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">ys</span> <span class="o">=</span> <span class="p">{</span><span class="mf">1.0</span><span class="p">:</span> <span class="s1">'no'</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ys</span><span class="p">[</span><span class="bp">True</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'yes'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ys</span> <span class="p">{</span><span class="mf">1.0</span><span class="p">:</span> <span class="s1">'yes'</span><span class="p">}</span> </pre></div> <p>This is presumably done as a performance optimization–if the keys are considered identical then why update the original. From this example we saw that the initial <code>True</code> object is never replaced as the key. Therefore the dictionary’s string representation still prints the key as <code>True</code> (instead of <code>1</code> or <code>1.0</code>).</p> </section><section><h2>Wait, what about the hash code?</h2> <p>With what we know now it looks like the values in the resulting dict are getting overwritten because they compare as equal. However, it turns out that this effect isn’t caused by the <code>__eq__</code> equality check alone, either.</p> <p>Let’s define the following class as our little detective tool:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">AlwaysEquals</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__eq__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span> <span class="sd">"""Make this object equal to any other object."""</span> <span class="k">return</span> <span class="bp">True</span> <span class="k">def</span> <span class="fm">__hash__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="sd">"""Give each object a unique hash code."""</span> <span class="k">return</span> <span class="nb">id</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> </pre></div> <p>All instances of this class will pretend they’re equal to any other object:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">AlwaysEquals</span><span class="p">()</span> <span class="o">==</span> <span class="n">AlwaysEquals</span><span class="p">()</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">AlwaysEquals</span><span class="p">()</span> <span class="o">==</span> <span class="mi">42</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">AlwaysEquals</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'waaat?'</span> <span class="bp">True</span> </pre></div> <p>However they will also each return a unique hash value:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="p">[</span><span class="nb">hash</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span> <span class="k">for</span> <span class="n">obj</span> <span class="ow">in</span> <span class="p">[</span><span class="n">AlwaysEquals</span><span class="p">(),</span> <span class="n">AlwaysEquals</span><span class="p">(),</span> <span class="n">AlwaysEquals</span><span class="p">()]]</span> <span class="p">[</span><span class="mi">4574298968</span><span class="p">,</span> <span class="mi">4574287912</span><span class="p">,</span> <span class="mi">4574287072</span><span class="p">]</span> </pre></div> <p>That’ll allow us to test if dictionary keys are overwritten based on their equality comparison result alone. And you’ll see that the keys are <em>not</em> getting overwritten even though they always compare as equal:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="p">{</span><span class="n">AlwaysEquals</span><span class="p">():</span> <span class="s1">'yes'</span><span class="p">,</span> <span class="n">AlwaysEquals</span><span class="p">():</span> <span class="s1">'no'</span><span class="p">}</span> <span class="p">{</span> <span class="o">&lt;</span><span class="n">AlwaysEquals</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x110a3c588</span><span class="o">&gt;</span><span class="p">:</span> <span class="s1">'yes'</span><span class="p">,</span> <span class="o">&lt;</span><span class="n">AlwaysEquals</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x110a3cf98</span><span class="o">&gt;</span><span class="p">:</span> <span class="s1">'no'</span> <span class="p">}</span> </pre></div> <p>We can also flip this idea around and check to see if returning the same hash value is enough to cause keys to get overwritten:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">class</span> <span class="nc">SameHash</span><span class="p">:</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="fm">__hash__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="o">&gt;&gt;&gt;</span> <span class="s2">"""Give each object the same hash code."""</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">return</span> <span class="mi">1</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="o">=</span> <span class="n">SameHash</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">b</span> <span class="o">=</span> <span class="n">SameHash</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="o">==</span> <span class="n">b</span> <span class="bp">False</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">hash</span><span class="p">(</span><span class="n">a</span><span class="p">),</span> <span class="nb">hash</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">({</span><span class="n">a</span><span class="p">:</span> <span class="s1">'a'</span><span class="p">,</span> <span class="n">b</span><span class="p">:</span> <span class="s1">'b'</span><span class="p">})</span> <span class="p">{</span> <span class="o">&lt;</span><span class="n">SameHash</span> <span class="n">instance</span> <span class="n">at</span> <span class="mh">0x7f7159020cb0</span><span class="o">&gt;</span><span class="p">:</span> <span class="s1">'a'</span><span class="p">,</span> <span class="o">&lt;</span><span class="n">SameHash</span> <span class="n">instance</span> <span class="n">at</span> <span class="mh">0x7f7159020cf8</span><span class="o">&gt;</span><span class="p">:</span> <span class="s1">'b'</span> <span class="p">}</span> </pre></div> <p>As this example shows, the “keys get overwritten” effect isn’t caused by the hash value alone either.</p> <p>A better explanation of what’s going on here is that dictionaries check for equality <em>and</em> compare the hash value to determine if two keys are the same:</p> <blockquote> <p>An object is hashable if it has a hash value which never changes during its lifetime (it needs a <code>__hash__()</code> method), and can be compared to other objects (it needs an <code>__eq__()</code> method). Hashable objects which compare equal must have the same hash value.</p> <p>Hashability makes an object usable as a dictionary key and a set member, because these data structures use the hash value internally.</p> <p>(Source: <a href="https://docs.python.org/3/glossary.html#term-hashable" target="_blank">https://docs.python.org/3/glossary.html#term-hashable</a>)</p> </blockquote> </section><section><h2>Umm okay, what’s the executive summary here?</h2> <p>Let’s try and summarize our findings:</p> <blockquote> <p><code>{True: 'yes', 1: 'no', 1.0: 'maybe'}</code> evaluates to <code>{True: 'maybe'}</code> because the keys <code>True</code>, <code>1</code>, and <code>1.0</code> all compare as equal <em>and</em> they all have the same hash value.</p> </blockquote> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="bp">True</span> <span class="o">==</span> <span class="mi">1</span> <span class="o">==</span> <span class="mf">1.0</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="p">(</span><span class="nb">hash</span><span class="p">(</span><span class="bp">True</span><span class="p">),</span> <span class="nb">hash</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> <span class="nb">hash</span><span class="p">(</span><span class="mf">1.0</span><span class="p">))</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> </pre></div> <p>That’s how we end up with this slightly surprising result as the dictionary’s final state:</p> <div class="codehilite"><pre><span></span><span class="p">{</span><span class="bp">True</span><span class="p">:</span> <span class="s1">'yes'</span><span class="p">,</span> <span class="mi">1</span><span class="p">:</span> <span class="s1">'no'</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">:</span> <span class="s1">'maybe'</span><span class="p">}</span> <span class="o">==</span> <span class="p">{</span><span class="bp">True</span><span class="p">:</span> <span class="s1">'maybe'</span><span class="p">}</span> </pre></div> </section><section><h2>It’s a Python Trick!</h2> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/CZ8bZPqtwU0?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p>I understand that all of this can be a bit mind-boggling at first. Try playing through the examples I gave one by one in a Python REPL. I’m sure it’ll help expand your knowledge of Python!</p> <p>I love these short one-line examples that teach you a ton about a language once you wrap your head around them. They’re almost like a <a href="https://en.wikipedia.org/wiki/K%C5%8Dan" target="_blank">Zen kōan</a> 😃</p> <p>There’s one more thing I want to tell you about:</p> <p>I’ve started a series of these Python “tricks” delivered over email. You can sign up at <a href="/python-tricks">dbader.org/python-tricks</a> and I’ll send you a new Python trick as a code screenshot every couple of days.</p> <p>This is still an experiment and a work in progress but I’ve heard some really positive feedback from the developers who’ve tried it out so far.</p> <p><em>Thanks to JayR, Murat, and kurashu89 for their feedback on this article.</em></p> <div class="footnote"> <hr> <ol> <li id="fn:bool-indexes"> <p>Yes, this also means you can do things like this and use bools as indexes: <code>('no', 'yes')[True] == 'yes'</code> But you probably <em>shouldn’t</em> do it for the sake of clarity 😉 <a class="footnote-backref" href="#fnref:bool-indexes" rev="footnote" title="Jump back to footnote 1 in the text">↩</a></p> </li> </ol> </div></section><footer></footer></article>https://dbader.org/blog/python-mystery-dict-expressionWed, 23 Nov 2016 00:00:00 GMTThe Ultimate List of Python Podcastshttps://dbader.org/blog/ultimate-list-of-python-podcasts<article><header><h1>The Ultimate List of Python Podcasts</h1> <p>I couldn’t find a good and updated list of Python developer or Python programming podcasts online. So I created my own list with the best Python podcasts.</p> </header><section><figure><img alt="The Ultimate List of Python Podcasts" src="/blog/figures/ultimate-list-of-python-podcasts.png" width="1280" height="720"></figure> <p>I enjoy listening to all kinds of podcasts when I’m at the gym or driving. There are some really good podcasts about Python development out there but I just couldn’t find a good (and updated) list.</p> <p>I initially created this list on forum posts and by searching the iTunes podcast directory and I will continue to grow it with user feedback. I plan to keep this list updated, so feel free to shoot me an email if you think anything is missing or if you’d like to see your own Python podcast added.</p> <p>My criteria for inclusion on this list are:</p> <ul> <li>episode download links must work; and</li> <li>the podcast must be active (new episodes are coming out) <em>OR</em> at least have an interesting archive with old episodes worth listening to.</li> </ul> <p>Enjoy the podcasts! 🎙🐍</p> </section><section><h2><a href="https://talkpython.fm/" target="_blank">Talk Python To Me</a></h2> <blockquote> <p>Talk Python to Me is a weekly podcast hosted by Michael Kennedy. The show covers a wide array of Python topics as well as many related topics (e.g. MongoDB, AngularJS, DevOps). The format is a casual 45 minute conversation with industry experts.</p> </blockquote> <ul> <li>Website: <a href="https://talkpython.fm/" target="_blank">talkpython.fm</a></li> <li>Twitter: <a href="https://twitter.com/TalkPython" target="_blank">@TalkPython</a></li> <li>Subscribe: <a href="https://talkpython.fm/episodes/rss" target="_blank">RSS</a> ⋅ <a href="https://itunes.apple.com/podcast/id979020229" target="_blank">iTunes</a> ⋅ <a href="https://overcast.fm/itunes979020229/" target="_blank">Overcast.fm</a></li> </ul> </section><section><h2><a href="https://www.podcastinit.com/" target="_blank">Podcast.__init__</a></h2> <blockquote> <p>A podcast about Python and the people who make it great. Hosted by Tobias Macey.</p> </blockquote> <ul> <li>Website: <a href="https://www.podcastinit.com/" target="_blank">www.podcastinit.com</a></li> <li>Twitter: <a href="https://twitter.com/Podcast__init__" target="_blank">@Podcast__init__</a></li> <li>Subscribe: <a href="https://www.podcastinit.com/feed/mp3/" target="_blank">RSS</a> ⋅ <a href="https://itunes.apple.com/podcast/id981834425" target="_blank">iTunes</a> ⋅ <a href="https://overcast.fm/itunes981834425/" target="_blank">Overcast.fm</a></li> </ul> </section><section><h2><a href="https://pythonbytes.fm/" target="_blank">Python Bytes</a></h2> <blockquote> <p>Python Bytes is a weekly podcast hosted by Michael Kennedy and Brian Okken. Python Bytes podcast delivers headlines directly to your earbuds. If you want to stay up on the Python developer news but don’t have time to scour reddit, twitter, and other news sources, just subscribe and you’ll get the best picks delivered weekly.</p> </blockquote> <ul> <li>Website: <a href="https://pythonbytes.fm/" target="_blank">pythonbytes.fm</a></li> <li>Twitter: <a href="https://twitter.com/PythonBytes" target="_blank">@PythonBytes</a></li> <li>Subscribe: <a href="https://pythonbytes.fm/episodes/rss" target="_blank">RSS</a> ⋅ <a href="https://itunes.apple.com/podcast/id1173690032" target="_blank">iTunes</a> ⋅ <a href="https://overcast.fm/itunes1173690032/" target="_blank">Overcast.fm</a></li> </ul> </section><section><h2><a href="http://python.madewithopinion.com/" target="_blank">The Python Experience</a></h2> <blockquote> <p>The personal ramblings of the perpetual computer programming newb, finally finding his way “home” to Python, and determined to do things more Pythonically.</p> </blockquote> <ul> <li>Website: <a href="http://python.madewithopinion.com/" target="_blank">python.madewithopinion.com</a></li> <li>Twitter: <a href="https://twitter.com/miklevin" target="_blank">@miklevin</a></li> <li>Subscribe: <a href="http://python.madewithopinion.com/feeds/python/rss/" target="_blank">RSS</a> ⋅ <a href="https://itunes.apple.com/podcast/id1165128787" target="_blank">iTunes</a> ⋅ <a href="https://overcast.fm/itunes1165128787/" target="_blank">Overcast.fm</a></li> </ul> </section><section><h2><a href="http://pythontesting.net/podcast" target="_blank">Test &amp; Code</a></h2> <blockquote> <p>A podcast about Software Development, Software Testing, and Python. How did you become a software developer/tester/engineer/lead, etc? Odds are we all are missing some important information to do our jobs most effectively. This podcast is an attempt to fill those education gaps. I focus on testing and process questions like “How do I know it works?”, “How do I effectively test?”, and the like. But really, anything in the software development realm is fair game.</p> </blockquote> <ul> <li>Website: <a href="http://pythontesting.net/podcast" target="_blank">pythontesting.net/podcast</a></li> <li>Twitter: <a href="https://twitter.com/TestPodcast" target="_blank">@TestPodcast</a></li> <li>Subscribe: <a href="http://pythontesting.net/feed/podcast/" target="_blank">RSS</a> ⋅ <a href="https://itunes.apple.com/podcast/id1029487211" target="_blank">iTunes</a> ⋅ <a href="https://overcast.fm/itunes1029487211/" target="_blank">Overcast.fm</a></li> </ul> </section><section><h2><a href="https://www.kennethreitz.org/import-this/" target="_blank">Import This: A Python Podcast for Humans</a></h2> <blockquote> <p>Featuring Kennneth Reitz and a random Python co-host.</p> </blockquote> <ul> <li>Website: <a href="https://www.kennethreitz.org/import-this/" target="_blank">www.kennethreitz.org/import-this/</a></li> <li>Twitter: <a href="https://twitter.com/kennethreitz" target="_blank">@kennethreitz</a></li> <li>Subscribe: <a href="http://feeds.soundcloud.com/users/soundcloud:users:82237854/sounds.rss" target="_blank">RSS</a> ⋅ <a href="https://itunes.apple.com/podcast/id829754832" target="_blank">iTunes</a> ⋅ <a href="https://overcast.fm/itunes829754832/" target="_blank">Overcast.fm</a></li> </ul> </section><section><h2><a href="http://radiofreepython.com/" target="_blank">Radio Free Python</a></h2> <blockquote> <p>A monthly podcast focused on the Python programming language and its community.</p> </blockquote> <p>(No updates since 2013 but an interesting archive of episodes.)</p> <ul> <li>Website: <a href="http://radiofreepython.com/" target="_blank">radiofreepython.com</a></li> <li>Twitter: <a href="https://twitter.com/RadioFreePython" target="_blank">@RadioFreePython</a></li> <li>Subscribe: <a href="http://radiofreepython.com/high_quality.rss" target="_blank">RSS</a></li> </ul> </section><section><h2><a href="http://frompythonimportpodcast.com/" target="_blank">from python import podcast</a></h2> <blockquote> <p>A small-batch artisanal podcast for irreverent pythonistas. Easy-going, conversational, often silly, and occasionally earning our iTunes “explicit” tag, From Python Import Podcast is news, analysis, discussion, and general shenanigans about the Python language and community. Put on your headphones and come hang out with us!</p> </blockquote> <p>(No updates since 2014 but an interesting archive of episodes.)</p> <ul> <li>Website: <a href="http://frompythonimportpodcast.com/" target="_blank">frompythonimportpodcast.com</a></li> <li>Twitter: <a href="https://twitter.com/__fpip__" target="_blank">@__fpip__</a></li> <li>Subscribe: <a href="http://feeds.feedburner.com/FromPythonImportPodcastmp3" target="_blank">RSS</a> ⋅ <a href="http://itunes.apple.com/podcast/id525611633" target="_blank">iTunes</a> ⋅ <a href="https://overcast.fm/itunes525611633/" target="_blank">Overcast.fm</a></li> </ul> </section><section><h2><a href="http://castalio.info/tag/python.html" target="_blank">Castálio Podcast (Portuguese &amp; English)</a></h2> <blockquote> <p>Um podcast inspirado prá castálio! Podcast com o objetivo de entrevistar e ao mesmo tempo apresentar pessoas e projetos que sejam fonte de inspiração para os ouvintes. Novos episódios toda semana.</p> </blockquote> <p>A primarily Portuguese podcast that often features Python topics and people in the Python community.</p> <ul> <li>Website: <a href="http://castalio.info/tag/python.html" target="_blank">castalio.info</a></li> <li>Twitter: <a href="https://twitter.com/castaliopod" target="_blank">@castaliopod</a></li> <li>Subscribe: <a href="http://feeds.feedburner.com/CastalioPodcastMP3" target="_blank">RSS</a> ⋅ <a href="https://itunes.apple.com/br/podcast/castalio-podcast/id446259197" target="_blank">iTunes</a> ⋅ <a href="https://overcast.fm/itunes446259197/" target="_blank">Overcast.fm</a></li> </ul> <p><em>If you think anything is missing from this list or if you’d like to see your own Python podcast added, then please email me at <a href="mailto:mail@dbader.org">mail@dbader.org</a>.</em></p></section><footer></footer></article>https://dbader.org/blog/ultimate-list-of-python-podcastsTue, 15 Nov 2016 00:00:00 GMTHow code linting will make you awesome at Pythonhttps://dbader.org/blog/python-code-linting<article><header><h1>How code linting will make you awesome at Python</h1> <p>In Python code reviews I’ve seen over and over that it can be tough for developers to format their Python code in a consistent way: extra whitespace, irregular indentation, and other “sloppiness” then often leads to actual bugs in the program.</p> </header><section><figure><img alt="" src="/blog/figures/python-code-linting-header.png" width="1280" height="719"></figure> <p>Luckily automated tools can help with this common problem. <em>Code linters</em> make sure your Python code is always formatted consistently—and their benefits go way beyond that.</p> </section><section><h2>What code linters can do for you</h2> <p>A <em>code linter</em> is a program that analyses your source code for potential errors. The kinds of errors a linter can detect include:</p> <ul> <li>syntax errors;</li> <li>structural problems like the use of undefined variables;</li> <li>best practice or code style guideline violations.</li> </ul> <p>I find code linting to be an indispensable productivity tool for writing Python. It’s possible to integrate linting into your editing environment. This gives you immediate feedback on your code right when you type it:</p> <figure><img alt="" src="/blog/figures/python-code-linting.jpg" width="958" height="220"></figure> <p>For some classes of errors, linting can shorten the usual <em>write code, run code, catch error, fix error</em> loop to <em>write code, see and fix error</em>. This difference might not seem much—but in the course of a day these time savings add up quickly and can have a huge impact on your productivity.</p> <p>In short, code linters are great!</p> </section><section><h2>Which Python linter should I use?</h2> <p>Python has several good options for code linters. The ones I’m listing here are available for free and are open-source software:</p> <ul> <li> <p><a href="https://pypi.python.org/pypi/flake8" target="_blank">Flake8</a> is my personal favorite these days. It’s fast and has a low rate of false positives. Flake8 is actually a combination of several other tools, mainly the <a href="https://pypi.python.org/pypi/pyflakes" target="_blank">Pyflakes</a> static analysis tool and the <a href="https://pypi.python.org/pypi/pycodestyle" target="_blank">Pycodestyle</a> (former pep8) code style checker.</p> </li> <li> <p><a href="https://pypi.python.org/pypi/pylint" target="_blank">Pylint</a> is another good choice. It takes a little more effort to set up than Flake8 and also triggers more false positives. On the other hand it provides a more comprehensive analysis. Definitely not a bad choice—but I’d stick with Flake8 if you’re just starting out.</p> </li> </ul> </section><section><h2>I’m sold—what’s the quickest way to get started?</h2> <p>If you’re not using a linter yet you’re missing out on some really great benefits. But don’t worry, I’ve got your back—I recorded a <a href="https://www.youtube.com/watch?v=TDUf93vqq3g" target="_blank">5 minute Python linting video tutorial</a> you can watch below.</p> <p>In the video I’ll give you the run down on how to set up the <a href="https://pypi.python.org/pypi/flake8" target="_blank">Flake8</a> Python linter from scratch. With a few simple steps you’ll be able run a code linter on your own Python programs. I’ll also demonstrate how linter feedback can be integrated with your code editor (I’m using <a href="/blog/tags/sublimetext">Sublime Text 3</a> in the video).</p> <p>I’ve seen great results from using linters. I believe they’re one of the quickest ways to improve your Python skills. Spend 5 minutes to try out Flake8—I’m sure it’ll be well worth your time 😊</p> <p>Enjoy the video:</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/TDUf93vqq3g?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div></section><footer></footer></article>https://dbader.org/blog/python-code-lintingThu, 10 Nov 2016 00:00:00 GMTA Python refactoring gone wronghttps://dbader.org/blog/python-refactoring-gone-wrong<article><header><h1>A Python refactoring gone wrong</h1> <p>Ever witnessed a colleague make a refactoring to “clean up” some Python code only to make it worse and harder to understand?</p> </header><section><figure><img alt="" src="/blog/figures/python-refactoring-gone-wrong.png" width="1280" height="720"></figure> <p>I know I did. And I’ve also <em>been that colleague</em> to others many times 😊</p> <p>There’s often a fine line between making code better by “cleaning it up” and just shuffling around or even making it slightly worse. Refactoring is hard!</p> <p>It’s challenging to come up with good examples for this – so I was delighted when I got this Python question from Bev:</p> <p><br></p> <hr> <p><em>I came across something in Python that I’m having difficulty understanding. As part of a code rewrite in a Python related Youtube video, an “if” statement was changed to:</em></p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="nb">any</span><span class="p">([</span><span class="bp">self</span><span class="o">.</span><span class="n">temperature</span> <span class="o">&gt;</span> <span class="n">MAX_TEMPERATURE</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">pressure</span> <span class="o">&gt;</span> <span class="n">MAX_PRESSURE</span><span class="p">]):</span> </pre></div> <p><em>Why was that used rather than the simpler:</em></p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">temperature</span> <span class="o">&gt;</span> <span class="n">MAX_TEMPERATURE</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">pressure</span> <span class="o">&gt;</span> <span class="n">MAX_PRESSURE</span><span class="p">):</span> </pre></div> <p><em>Why create a list and call a function in the if statement when there are only two (2) comparisons?</em></p> <hr> <p><br></p> <p>I agree with Bev, this is a surprising change and I don’t think it’s for the better!</p> <p>It complicates the code for no apparent gain.</p> <p>Let’s take a look at the definition for the <code>any()</code> function first:</p> <blockquote> <p>Return True if any element of the iterable is true. If the iterable is empty, return False (Source: <a href="https://docs.python.org/3/library/functions.html#any" target="_blank">Python docs</a>)</p> </blockquote> <p><code>any()</code> – and its colleague, <code>all()</code> – are handy if you need to check the elements of an iterable (like a list or a generator) for truthiness.</p> <p>In some cases using <code>any()</code> can help avoid having to write a loop to check the elements of the iterable individually<sup id="fnref:short-circuit"><a class="footnote-ref" href="#fn:short-circuit" rel="footnote">1</a></sup>. Imagine you needed to do something like this:</p> <div class="codehilite"><pre><span></span><span class="n">result</span> <span class="o">=</span> <span class="bp">False</span> <span class="k">for</span> <span class="n">elem</span> <span class="ow">in</span> <span class="n">my_iterable</span><span class="p">:</span> <span class="k">if</span> <span class="n">elem</span><span class="p">:</span> <span class="n">result</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">break</span> </pre></div> <p>You could replace these 5 lines with a simple assignment using the <code>any()</code> function:</p> <div class="codehilite"><pre><span></span><span class="n">result</span> <span class="o">=</span> <span class="nb">any</span><span class="p">(</span><span class="n">my_iterable</span><span class="p">)</span> </pre></div> <p>In this case it makes sense to go with the <code>any()</code> solution because it is shorter and easier to understand.</p> <p>Yet, too much of a good thing can make you sick… 😷</p> <p>In my mind it doesn’t make much sense to use <code>any()</code> if all you have is a fixed size list with 2-3 elements, like in the example Bev found:</p> <ul> <li> <p>Constructing a list so we have an iterable to pass to <code>any()</code> is confusing. It adds visual clutter and is a non-standard pattern.</p> </li> <li> <p>On top of that, using <code>any()</code> is also going to be slower than a nice and simple “or” expression: The Python interpreter first needs to construct that list and then call <code>any()</code> on it<sup id="fnref:memory"><a class="footnote-ref" href="#fn:memory" rel="footnote">2</a></sup>.</p> </li> </ul> <p>In summary, I think this refactoring was <em>not</em> helpful. I’d say it actually made the code worse by making it slightly harder to read and understand.</p> <p>As developers we need to be careful and deliberate about using the tools and patterns we know. “Fancy” often won’t equal “better”.</p> <p>Great question, Bev! I think you were spot-on in doubting this refactoring 😃</p> <div class="footnote"> <hr> <ol> <li id="fn:short-circuit"> <p>It’s good to note that <code>any()</code> has <a href="https://en.wikipedia.org/wiki/Short-circuit_evaluation" target="_blank">short-circuit evaluation semantics</a>, so using it is equivalent to a chain of <code>or</code> operations. (Thanks to <a href="https://axil.github.io/" target="_blank">Lev Maximov</a> for his feedback.) <a class="footnote-backref" href="#fnref:short-circuit" rev="footnote" title="Jump back to footnote 1 in the text">↩</a></p> </li> <li id="fn:memory"> <p>In all fairness, this won’t make a big real-world difference in performance or memory usage – but it’s always good to keep these things in mind. <a class="footnote-backref" href="#fnref:memory" rev="footnote" title="Jump back to footnote 2 in the text">↩</a></p> </li> </ol> </div></section><footer></footer></article>https://dbader.org/blog/python-refactoring-gone-wrongThu, 03 Nov 2016 00:00:00 GMTPython Code Review: Unplugged – Episode 3https://dbader.org/blog/python-code-review-unplugged-episode-3<article><header><h1>Python Code Review: Unplugged – Episode 3</h1> <p>In this third episode of my video code review series I take a look at a reader’s web scraping project and start adding some unit tests to it.</p> </header><section><div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/03Ki2WpVRBc?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p>This is a Python code review I did for Sunny’s web scraping project on GitHub. Sunny reached out to me after watching one of my previous code review videos, asking me if I could give him some feedback on his web scraping pet project.</p> <p>In this episode you’ll see Flake8 and Python code linting tools make a comeback. Also I’m doing an intro to adding Pytest unit tests to an existing Python code base in the second half of the video.</p> <p>By the way, I love how eager Sunny was to get feedback on his Python code:</p> <blockquote> <p>As a pet project I wanted to learn all the best practice as much as I could about python, trying to code as pythonic as I can, checking pep8 and autopep8 (didn’t knew about flake8!!) Eventually I re-factored using classes and methods. If you’ll see that it originally did everything in one monolithic function in the 2.x pull request on github. I still have a few things that I want to implement, so I welcome all kinds of feedback about it!</p> </blockquote> <p>This is exactly the right mindset that turns people into productive and successful software engineers. Even tiny – but constant – improvements add up and compound over time.</p> <p>I’ve seen this in friends and colleagues alike. Those developers who seek out constant small improvements eventually go on and do amazing things.</p> <p>Enjoy the video! And be sure to check out my other <a href="/screencasts">Python screencasts</a> if you liked this code review 😊</p> <p><strong>Links &amp; Resources</strong>:</p> <ul> <li><a href="https://github.com/sunnz" target="_blank">Sunny’s GitHub account</a></li> <li><a href="https://github.com/sunnz/auction-results.py/pull/2" target="_blank">The GitHub pull-request from the video</a></li> <li><a href="http://doc.pytest.org/en/latest/" target="_blank">Pytest Python testing framework</a></li> <li><a href="http://flake8.pycqa.org/en/latest/" target="_blank">Flake8 Python code linter</a></li> <li><a href="https://SublimeTextPython.com" target="_blank">How to integrate code linting with your editor setup</a></li> </ul> <p>» <a href="tags/code-review">Click here to watch my other Python Code Review: Unplugged videos</a></p></section><footer></footer></article>https://dbader.org/blog/python-code-review-unplugged-episode-3Wed, 26 Oct 2016 00:00:00 GMTClick & jump to any file or folder from the terminalhttps://dbader.org/blog/click-and-jump-to-any-file-or-folder-from-the-terminal<article><header><h1>Click &amp; jump to any file or folder from the terminal</h1> <p>iTerm2 for macOS has a little known feature that lets you <strong>open files and folders simply by Cmd+Clicking on them in the terminal</strong>. Among other things, this is super handy for debugging tests.</p> </header><section><figure><img alt="" src="/blog/figures/iterm-jump-to-file.gif" width="1280" height="720"></figure> <p>With this so called <em>Semantic History</em> feature you can configure <a href="https://www.iterm2.com/" target="_blank">iTerm2</a> to open folders and files in their default application when you press Cmd and then click on them.</p> <p>So if you click on a folder name it will open in the Finder, and if you click on a <code>.py</code> file, for example, it will open in your editor.</p> <p>The amazingly cool part is that this also works with line numbers, so if you click on something like <code>test_myapp.py:42</code> in the terminal your editor opens <code>test_myapp.py</code> and moves the cursor to line 42! 😀</p> <p>This unbelievably handy if you’re running your unit tests from the command line. I use it all the time to click and jump to failed test cases with the Pytest test runner, for example.</p> <p>Here’s how to set up <em>Semantic History</em> in iTerm2:</p> <ul> <li>Open the iTerm2 preferences by clicking on <strong>iTerm2 → Preferences</strong> in the menu bar (or press <strong>Cmd+,</strong>)</li> <li>Click on <strong>Profiles</strong> in the top row, then click <strong>Advanced</strong> all the way to the right. Find the section that says <strong>Semantic History</strong>.</li> <li>Under <strong>Semantic History</strong>, set the first option to <strong>Open with editor… </strong>and then pick your favorite editor (I use <strong>Sublime Text 3</strong>).</li> <li>Close the preferences window – that’s it!</li> </ul> <p>If you need some more help setting this up and a quick demo of what you can do with this feature, watch my video below:</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/Gw43Q0WZptI?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p>Like I said, I found this “click to jump to file” feature extremely helpful for working with tests.</p> <p>I usually run my Python tests with <a href="http://doc.pytest.org/en/latest/" target="_blank">Pytest</a> and it prints test failure messages in a format that iTerm2 understands. So I can simply Cmd+click on a failed test assertion and that’ll open up the test case Sublime Text, placing the cursor at the exact line that caused the test to fail.</p> <p>This feature should be completely language agnostic by the way. You’ll be able to use it with any test runner or programming language – and any editor.</p> <p><strong>P.S.</strong> Unfortunately iTerm2 is only available on macOS. I’d love to learn if there’s a way to get the same functionality on Windows or Linux, so far I haven’t been able to find anything. If you know how to do this on Linux or Windows please get in touch and tell me how to do it :) Thanks!</p></section><footer></footer></article>https://dbader.org/blog/click-and-jump-to-any-file-or-folder-from-the-terminalMon, 17 Oct 2016 00:00:00 GMTSublime Text plugin review: Djaneirohttps://dbader.org/blog/sublime-text-djaneiro-review<article><header><h1>Sublime Text plugin review: Djaneiro</h1> <p>A review of Djaneiro, a Sublime Text plugin for Django development.</p> </header><section><figure><img alt="" src="/blog/figures/djaneiro-snippets.gif" width="1280" height="720"></figure> <p>I’ll admit I was skeptical at first when a friend of mine recommended <a href="https://packagecontrol.io/packages/Djaneiro" target="_blank">Djaneiro</a> to enhance my Django development workflow in Sublime Text.</p> <p>I’d been happy with the <a href="/products/sublime-python-guide/">Python development setup</a> I built for myself over the years and I didn’t really understand what Djaneiro was going to add to that.</p> <p>But when I tried out Djaneiro I was impressed how helpful it turned out to be! I decided to write <a href="tags/plugin-review">another Sublime Text plugin review</a> to share my findings.</p> <p>Djaneiro’s main selling points are adding:</p> <ul> <li><strong>syntax highlighting for Django HTML templates</strong>; and</li> <li><strong>code completion snippets for Django HTML templates and Python files</strong>.</li> </ul> <p>In this review I’ll explain how Djaneiro can make your Django development workflow more productive and I’ll go over the pros and cons of the plugin as I experienced them. After that I’ll take a look at alternatives to Djaneiro in the Sublime Text plugin landscape. At the end I’ll share my final verdict and rating.</p> </section><section><h2>Pros</h2> <p><strong>Syntax highlighting for Django templates</strong>: In its default configuration Sublime Text doesn’t have syntax definitions for Django’s HTML templating syntax.</p> <p>This means that typos and syntax errors in templates are harder to catch visually. As you can see in the screenshot below (in the editing pane on the left), the standard HTML syntax highlighting in Sublime Text 3 uses a uniform white color for Django’s template tags.</p> <p>Djaneiro adds a <em>HTML (Django)</em> syntax that properly highlights Django’s template tags. As you can see in the right-hand editing pane in the screenshot, proper syntax highlighting makes these templates quite a bit easier to read. Also, syntax errors and typos stand out more due to the proper highlighting.</p> <p>This simple change adds a lot of value – I found that I was making fewer typos in my templates with Djaneiro’s syntax highlighting. Also, templates seemed easier to read and scan quickly with Djaneiro installed.</p> <figure><img alt="" src="/blog/figures/djaneiro-html-syntax.png" width="1516" height="588"></figure> <p><strong>Improved syntax highlighting for Django Python files</strong>: Djaneiro also makes some small tweaks to the default Python syntax highlighting. For example, it knows the standard Django settings constants like <code>INSTALLED_APPS</code> and highlights them differently so that they stand out more and typos are easier to find.</p> <p>This also happens for things like field definitions when writing Django model classes, which I found handy. In summary I found that the syntax highlighting changes introduced by Djaneiro make it easier to grasp the structure of the code I’m writing.</p> <figure><img alt="" src="/blog/figures/djaneiro-python-syntax.png" width="1680" height="669"></figure> <p><strong>Code completion snippets for Django templates and Python files</strong>: Another helpful feature provided by Djaneiro is a library of pre-made code completion snippets for common Django code and patterns.</p> <p>For example, you can insert an <code>{% if _____ %} {% endif %}</code> block by typing <code>if</code> as an abbreviation and hitting the auto-complete key (<em>Tab</em> by default). You can see a quick demo of that in the screenshot below. Generally, I found the <a href="https://packagecontrol.io/packages/Djaneiro" target="_blank">list of snippets included with Djaneiro</a> to be comprehensive and well-chosen.</p> <p>Besides snippets for Django HTML templates Djaneiro also includes a snippet library for Django Python code. These snippets let you quickly scaffold out whole view definitions or a barebones model classes, for example.</p> <p>Once you’ve gotten used to these snippets they can save you <em>a lot</em> of typing. Be sure to check out the full list of snippets in the <a href="https://github.com/squ1b3r/Djaneiro" target="_blank">Djaneiro README</a>.</p> <figure><img alt="" src="/blog/figures/djaneiro-snippets.gif" width="1280" height="720"></figure> </section><section><h2>Cons</h2> <p><strong>Snippets might get in the way</strong>: Because Djaneiro adds quite a substantial number of new code snippets I found myself triggering some of them accidentally, especially in the beginning. I really don’t want to hold this up against Djaneiro because the snippets do add a lot of value once I learned to use them well.</p> <p>It’s possible to <a href="http://stackoverflow.com/questions/27995926/how-do-i-disable-snippets-in-sublime-text-3" target="_blank">disable individual code snippets in Sublime Text</a> but unfortunately this process is a bit involved.</p> <p>If you find that the snippets get in your way occasionally you can temporarily switch them off by selecting a different syntax highlighting definition. Just open the Sublime Text <em>Command Palette</em>, type <em>Set syntax</em>, and select the default HTML or Python syntax.</p> </section><section><h2>Alternatives</h2> <p>There are a few more <a href="https://packagecontrol.io/search/django" target="_blank">Django-specific plugins available on Package Control</a> but Djaneiro seems to be the most popular and also the most powerful of the pack.</p> </section><section><h2>The verdict</h2> <p>I’ve grown quite fond of Djaneiro since I started using it. I immediately loved the improved syntax highlighting for Django templates and I’d say Djaneiro is worth installing for that feature alone.</p> <p>Once I’d gotten the hang of Djaneiro’s code snippets and their shortcodes I felt a noticeable improvement in my productivity. The snippets added by Djaneiro cover many things I encountered in day to day Django development. It’s great not having to go through the work of writing these snippets myself.</p> <p>I’d recommend any Django developer using Sublime Text to at least try out Djaneiro for a few days. There aren’t any substantial downsides to it and I’m sure it will make you more productive. You can &l