dbader.org - Pythonhttps://dbader.org/blog/tags/pythonenThu, 25 May 2017 00:00:00 GMTSun, 28 May 2017 05:00:30 GMTPyRSS2Gen-1.1.0http://blogs.law.harvard.edu/tech/rsshttps://dbader.org/logodbader.orghttps://dbader.org/In Love, War, and Open-Source: Never Give Uphttps://dbader.org/blog/in-love-war-and-open-source-never-give-up<article><header><h1>In Love, War, and Open-Source: Never Give Up</h1> <p>I’ll never forget launching my first open-source project and sharing it publicly on Reddit…</p> </header><section><figure><img alt="" src="/blog/figures/love-war-open-source.png" width="1280" height="720"></figure> <p>I had spent a couple of days at my parents’ place over Christmas that year and decided to use some of my spare time to work on a Python library I christened <a href="https://github.com/dbader/schedule" target="_blank"><code>schedule</code></a>.</p> <p>The idea behind <code>schedule</code> was very simple and had a narrow focus (I find that that that’s always a good idea for libraries by the way):</p> <p>Developers would use it like a timer to periodically call a function inside their Python programs.</p> <p>The kicker was that <code>schedule</code> used a funky “natural sounding” syntax to specify the timer interval. For example, if you wanted to run a function every 10 minutes you’d do this:</p> <div class="codehilite"><pre><span></span><span class="n">schedule</span><span class="o">.</span><span class="n">every</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span><span class="o">.</span><span class="n">minutes</span><span class="o">.</span><span class="n">do</span><span class="p">(</span><span class="n">myfunc</span><span class="p">)</span> </pre></div> <p>Or, if you wanted to run a particular task every day at 10:30 in the morning, you’d do this:</p> <div class="codehilite"><pre><span></span><span class="n">schedule</span><span class="o">.</span><span class="n">every</span><span class="p">()</span><span class="o">.</span><span class="n">day</span><span class="o">.</span><span class="n">at</span><span class="p">(</span><span class="s1">'10:30'</span><span class="p">)</span><span class="o">.</span><span class="n">do</span><span class="p">(</span><span class="n">mytask</span><span class="p">)</span> </pre></div> <p>Because I was so frustrated with Cron’s syntax I thought this approach was really cool. And so I decided this would be the first Python module I’d release as open-source.</p> <p>I cleaned up the code and spent some time coming up with <a href="write-a-great-readme-for-your-github-project">a nice README file</a>—because that’s really the first thing that your potential users will see when they check out your library.</p> <p>Once I had my module available on PyPI and the source code on GitHub I decided to call some attention to the project. The same night I posted a link to the repository to Reddit and a couple of other sites.</p> </section><section><h2 class="crosshead thin">I still remember that I had shaky hands when I clicked the “submit” button…</h2> <p>It’s scary to put your work out there for the whole world to judge! Also, I didn’t know what to expect.</p> <p>Would people call me stupid for writing a “simple” library like that?</p> <p>Would they think my code wasn’t good enough?</p> <p>Would they find all kinds of bugs and publicly shame me for them? I felt almost a physical sense of dread about pushing the “submit” button on Reddit that night!</p> <p>The next morning I woke up and immediately checked my email. Were there any comments? Yes, about twenty or so!</p> <p>I started reading through all of them, faster and faster—</p> <p>And of course my still frightful mind immediately zoomed in on the negative ones, like</p> <blockquote> <p>“Cool idea, but not particularly useful”,</p> </blockquote> <p>and</p> <blockquote> <p>“The documentation is not enough”,</p> </blockquote> <p>or</p> <blockquote> <p>“Not a big fan of the pseudo-english syntax. Way too clever and gimmicky.”</p> </blockquote> <p>At this point I was starting to feel a <em>little</em> discouraged… I’d never really shared my code publicly before and to be honest I my skin receiving criticism on it was paper thin. After all, this was just something I wrote in a couple of hours and gave away for free.</p> <p>The comment that really made my stomach churn was one from a well known member of the Python community:</p> <blockquote> <p>“And another library with global state :-( … Such an API should not even exist. It sets a bad example.”</p> </blockquote> <p>Ouch, that stung. I really looked up to that person and had used some of their libraries in other projects… It was almost like my worst fears we’re confirmed and we’re now playing out in front of me!</p> </section><section><h2 class="crosshead thin">I’d never be able to get another job as a Python developer after this…</h2> <p>At the time I didn’t see the positive and supportive comments in that discussion thread. I didn’t see the almost 70 upvotes. I didn’t see the valuable lessons hidden in the seemingly rude comments. I dwelled on the negative and felt terrible and depressed that whole day.</p> <p>So how do you think this story ends?</p> <p>Did I delete the <code>schedule</code> repo, switched careers and never looked at Reddit again?</p> <p>Wrong!</p> <p><code>schedule</code> now has almost 3,000 stars on GitHub and is among the top 70 Python repositories (out of more than 215,000). When PyPI’s download statistics we’re still working I saw that it got several thousand downloads per month. I get emails every week from people asking questions about it or thanking me for writing it…</p> </section><section><h2 class="crosshead thin">Isn’t that crazy!? How’s that possible after all of these disheartening comments?</h2> <p>My answer is “I don’t know”—and I also don’t think that <code>schedule</code> is a particularly great library that deserves all this attention, by the way.</p> <p>But, it seems to solve a problem for some people. It also seems to have a polarizing effect on developers who see it—some love it, some hate it.</p> <p>Today I’m glad I shipped <code>schedule</code> that night.</p> <p>Glad because it was helpful to so many people over the years and glad because it helped me develop a thicker skin when it comes to sharing and launching things publicly.</p> <p>I’m partly writing this meandering post because not very long ago I found this comment buried in my Reddit message history:</p> <blockquote> <p>As someone who has posted a number of projects and blog posts in r/Python, just wanted to drop you a line and encourage that you don’t let the comments in your thread get you down. You see all those upvotes?</p> <p>Those are people that like your library, but don’t really have a comment to make in the thread proper. My biggest issue with /r/Python is that it tends towards cynicism and sometimes cruelty rather than encouragement and constructive criticism.</p> <p>Keep up the great work,</p> <p>Rob</p> </blockquote> <p>Wow! What a positive and encouraging comment!</p> <p>Back when I felt discouraged by all of these negative comments I must’ve missed it. But reading it a few years later made me re-live that whole situation and it showed me how much I’d grown as a developer and as a person in the meantime.</p> <p>If you find yourself in a similar situation, maybe feeling bogged down by the developer community who can be unfiltered and pretty rude sometimes, don’t get discouraged.</p> <p>Even if some people don’t like what you did there can be thousands who love your work.</p> <p>It’s a big pond, and sometimes the best ideas are polarizing.</p> <p>The only way to find out is to ship, ship, ship.</p></section><footer></footer></article>https://dbader.org/blog/in-love-war-and-open-source-never-give-upThu, 25 May 2017 00:00:00 GMTThe Meaning of Underscores in Pythonhttps://dbader.org/blog/meaning-of-underscores-in-python<article><header><h1>The Meaning of Underscores in Python</h1> <p>The various meanings and naming conventions around single and double underscores (“dunder”) in Python, how name mangling works and how it affects your own Python classes.</p> </header><section><figure><img alt="The Meaning of _ and __ Underscores in Python" src="/blog/figures/python-underscores.png" width="1280" height="720"></figure> <p>Single and double underscores have a meaning in Python variable and method names. Some of that meaning is merely by convention and intended as a hint to the programmer—and some of it is enforced by the Python interpreter.</p> <p>If you’re wondering <em>“What’s the meaning of single and double underscores in Python variable and method names?”</em> I’ll do my best to get you the answer here.</p> <p>In this article I’ll discuss the following five underscore patterns and naming conventions and how they affect the behavior of your Python programs:</p> <ul> <li>Single Leading Underscore: <code>_var</code></li> <li>Single Trailing Underscore: <code>var_</code></li> <li>Double Leading Underscore: <code>__var</code></li> <li>Double Leading and Trailing Underscore: <code>__var__</code></li> <li>Single Underscore: <code>_</code></li> </ul> <p>At the end of the article you’ll also find a brief “cheat sheet” summary of the five different underscore naming conventions and their meaning, as well as a short video tutorial that gives you a hands-on demo of their behavior.</p> <p>Let’s dive right in!</p> </section><section><h2>1. Single Leading Underscore: <code>_var</code></h2> <p>When it comes to variable and method names, the single underscore prefix has a meaning by convention only. It’s a hint to the programmer—and it means what the Python community agrees it should mean, but it does not affect the behavior of your programs.</p> <p>The underscore prefix is meant as a <em>hint</em> to another programmer that a variable or method starting with a single underscore is intended for internal use. This convention is <a href="http://pep8.org/#descriptive-naming-styles" target="_blank">defined in PEP 8</a>.</p> <p>This isn’t enforced by Python. Python does not have strong distinctions between “private” and “public” variables like Java does. It’s like someone put up a tiny underscore warning sign that says:</p> <blockquote> <p>“Hey, this isn’t really meant to be a part of the public interface of this class. Best to leave it alone.”</p> </blockquote> <p>Take a look at the following example:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Test</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">foo</span> <span class="o">=</span> <span class="mi">11</span> <span class="bp">self</span><span class="o">.</span><span class="n">_bar</span> <span class="o">=</span> <span class="mi">23</span> </pre></div> <p>What’s going to happen if you instantiate this class and try to access the <code>foo</code> and <code>_bar</code> attributes defined in its <code>__init__</code> constructor? Let’s find out:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">t</span> <span class="o">=</span> <span class="n">Test</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">foo</span> <span class="mi">11</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">_bar</span> <span class="mi">23</span> </pre></div> <p>You just saw that the leading single underscore in <code>_bar</code> did not prevent us from “reaching into” the class and accessing the value of that variable.</p> <p>That’s because the single underscore prefix in Python is merely an agreed upon convention—at least when it comes to variable and method names.</p> <p><strong>However, leading underscores do impact how names get imported from modules.</strong> Imagine you had the following code in a module called <code>my_module</code>:</p> <div class="codehilite"><pre><span></span><span class="c1"># This is my_module.py:</span> <span class="k">def</span> <span class="nf">external_func</span><span class="p">():</span> <span class="k">return</span> <span class="mi">23</span> <span class="k">def</span> <span class="nf">_internal_func</span><span class="p">():</span> <span class="k">return</span> <span class="mi">42</span> </pre></div> <p>Now if you use a <em>wildcard import</em> to import all names from the module, Python will <em>not</em> import names with a leading underscore (unless the module defines an <a href="https://docs.python.org/3/tutorial/modules.html#importing-from-a-package" target="_blank"><code>__all__</code> list</a> that overrides this behavior):</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">my_module</span> <span class="kn">import</span> <span class="o">*</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">external_func</span><span class="p">()</span> <span class="mi">23</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">_internal_func</span><span class="p">()</span> <span class="ne">NameError</span><span class="p">:</span> <span class="s2">"name '_internal_func' is not defined"</span> </pre></div> <p>By the way, <a href="http://pep8.org/#imports" target="_blank">wildcard imports should be avoided</a> as they make it unclear which names are present in the namespace. It’s better to stick to regular imports for the sake of clarity.</p> <p>Unlike wildcard imports, regular imports are not affected by the leading single underscore naming convention:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">my_module</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_module</span><span class="o">.</span><span class="n">external_func</span><span class="p">()</span> <span class="mi">23</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_module</span><span class="o">.</span><span class="n">_internal_func</span><span class="p">()</span> <span class="mi">42</span> </pre></div> <p>I know this might be a little confusing at this point. If you stick to the PEP 8 recommendation that wildcard imports should be avoided, then really all you need to remember is this:</p> <blockquote> <p><em>Single underscores are a Python naming convention indicating a name is meant for internal use. It is generally not enforced by the Python interpreter and meant as a hint to the programmer only.</em></p> </blockquote> </section><section><h2>2. Single Trailing Underscore: <code>var_</code></h2> <p>Sometimes the most fitting name for a variable is already taken by a keyword. Therefore names like <code>class</code> or <code>def</code> cannot be used as variable names in Python. In this case you can append a single underscore to break the naming conflict:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">make_object</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">class</span><span class="p">):</span> <span class="ne">SyntaxError</span><span class="p">:</span> <span class="s2">"invalid syntax"</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">make_object</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">class_</span><span class="p">):</span> <span class="o">...</span> <span class="k">pass</span> </pre></div> <p>In summary, a single trailing underscore (postfix) is used by convention to avoid naming conflicts with Python keywords. This convention is <a href="http://pep8.org/#descriptive-naming-styles" target="_blank">explained in PEP 8</a>.</p> </section><section><h2>3. Double Leading Underscore: <code>__var</code></h2> <p>The naming patterns we covered so far received their meaning from agreed upon conventions only. With Python class attributes (variables and methods) that start with double underscores, things are a little different.</p> <p>A double underscore prefix causes the Python interpreter to rewrite the attribute name in order to avoid naming conflicts in subclasses.</p> <p>This is also called <em>name mangling</em>—the interpreter changes the name of the variable in a way that makes it harder to create collisions when the class is extended later.</p> <p>I know this sounds rather abstract. This is why I put together this little code example we can use for experimentation:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Test</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">foo</span> <span class="o">=</span> <span class="mi">11</span> <span class="bp">self</span><span class="o">.</span><span class="n">_bar</span> <span class="o">=</span> <span class="mi">23</span> <span class="bp">self</span><span class="o">.</span><span class="n">__baz</span> <span class="o">=</span> <span class="mi">23</span> </pre></div> <p>Let’s take a look at the attributes on this object using the <a href="https://docs.python.org/3/library/functions.html#dir" target="_blank">built-in <code>dir()</code> function</a>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">t</span> <span class="o">=</span> <span class="n">Test</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">dir</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="p">[</span><span class="s1">'_Test__baz'</span><span class="p">,</span> <span class="s1">'__class__'</span><span class="p">,</span> <span class="s1">'__delattr__'</span><span class="p">,</span> <span class="s1">'__dict__'</span><span class="p">,</span> <span class="s1">'__dir__'</span><span class="p">,</span> <span class="s1">'__doc__'</span><span class="p">,</span> <span class="s1">'__eq__'</span><span class="p">,</span> <span class="s1">'__format__'</span><span class="p">,</span> <span class="s1">'__ge__'</span><span class="p">,</span> <span class="s1">'__getattribute__'</span><span class="p">,</span> <span class="s1">'__gt__'</span><span class="p">,</span> <span class="s1">'__hash__'</span><span class="p">,</span> <span class="s1">'__init__'</span><span class="p">,</span> <span class="s1">'__le__'</span><span class="p">,</span> <span class="s1">'__lt__'</span><span class="p">,</span> <span class="s1">'__module__'</span><span class="p">,</span> <span class="s1">'__ne__'</span><span class="p">,</span> <span class="s1">'__new__'</span><span class="p">,</span> <span class="s1">'__reduce__'</span><span class="p">,</span> <span class="s1">'__reduce_ex__'</span><span class="p">,</span> <span class="s1">'__repr__'</span><span class="p">,</span> <span class="s1">'__setattr__'</span><span class="p">,</span> <span class="s1">'__sizeof__'</span><span class="p">,</span> <span class="s1">'__str__'</span><span class="p">,</span> <span class="s1">'__subclasshook__'</span><span class="p">,</span> <span class="s1">'__weakref__'</span><span class="p">,</span> <span class="s1">'_bar'</span><span class="p">,</span> <span class="s1">'foo'</span><span class="p">]</span> </pre></div> <p>This gives us a list with the object’s attributes. Let’s take this list and look for our original variable names <code>foo</code>, <code>_bar</code>, and <code>__baz</code>—I promise you’ll notice some interesting changes.</p> <ul> <li>The <code>self.foo</code> variable appears unmodified as <code>foo</code> in the attribute list.</li> <li><code>self._bar</code> behaves the same way—it shows up on the class as <code>_bar</code>. Like I said before, the leading underscore is just a <em>convention</em> in this case. A hint for the programmer.</li> <li>However with <code>self.__baz</code>, things look a little different. When you search for <code>__baz</code> in that list you’ll see that there is no variable with that name.</li> </ul> <p>So what happened to <code>__baz</code>?</p> <p>If you look closely you’ll see there’s an attribute called <code>_Test__baz</code> on this object. This is the <em>name mangling</em> that the Python interpreter applies. It does this to protect the variable from getting overridden in subclasses.</p> <p>Let’s create another class that extends the <code>Test</code> class and attempts to override its existing attributes added in the constructor:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">ExtendedTest</span><span class="p">(</span><span class="n">Test</span><span class="p">):</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">foo</span> <span class="o">=</span> <span class="s1">'overridden'</span> <span class="bp">self</span><span class="o">.</span><span class="n">_bar</span> <span class="o">=</span> <span class="s1">'overridden'</span> <span class="bp">self</span><span class="o">.</span><span class="n">__baz</span> <span class="o">=</span> <span class="s1">'overridden'</span> </pre></div> <p>Now what do you think the values of <code>foo</code>, <code>_bar</code>, and <code>__baz</code> will be on instances of this <code>ExtendedTest</code> class? Let’s take a look:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">t2</span> <span class="o">=</span> <span class="n">ExtendedTest</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">t2</span><span class="o">.</span><span class="n">foo</span> <span class="s1">'overridden'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">t2</span><span class="o">.</span><span class="n">_bar</span> <span class="s1">'overridden'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">t2</span><span class="o">.</span><span class="n">__baz</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="s2">"'ExtendedTest' object has no attribute '__baz'"</span> </pre></div> <p>Wait, why did we get that <code>AttributeError</code> when we tried to inspect the value of <code>t2.__baz</code>? Name mangling strikes again! It turns out this object doesn’t even have a <code>__baz</code> attribute:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">dir</span><span class="p">(</span><span class="n">t2</span><span class="p">)</span> <span class="p">[</span><span class="s1">'_ExtendedTest__baz'</span><span class="p">,</span> <span class="s1">'_Test__baz'</span><span class="p">,</span> <span class="s1">'__class__'</span><span class="p">,</span> <span class="s1">'__delattr__'</span><span class="p">,</span> <span class="s1">'__dict__'</span><span class="p">,</span> <span class="s1">'__dir__'</span><span class="p">,</span> <span class="s1">'__doc__'</span><span class="p">,</span> <span class="s1">'__eq__'</span><span class="p">,</span> <span class="s1">'__format__'</span><span class="p">,</span> <span class="s1">'__ge__'</span><span class="p">,</span> <span class="s1">'__getattribute__'</span><span class="p">,</span> <span class="s1">'__gt__'</span><span class="p">,</span> <span class="s1">'__hash__'</span><span class="p">,</span> <span class="s1">'__init__'</span><span class="p">,</span> <span class="s1">'__le__'</span><span class="p">,</span> <span class="s1">'__lt__'</span><span class="p">,</span> <span class="s1">'__module__'</span><span class="p">,</span> <span class="s1">'__ne__'</span><span class="p">,</span> <span class="s1">'__new__'</span><span class="p">,</span> <span class="s1">'__reduce__'</span><span class="p">,</span> <span class="s1">'__reduce_ex__'</span><span class="p">,</span> <span class="s1">'__repr__'</span><span class="p">,</span> <span class="s1">'__setattr__'</span><span class="p">,</span> <span class="s1">'__sizeof__'</span><span class="p">,</span> <span class="s1">'__str__'</span><span class="p">,</span> <span class="s1">'__subclasshook__'</span><span class="p">,</span> <span class="s1">'__weakref__'</span><span class="p">,</span> <span class="s1">'_bar'</span><span class="p">,</span> <span class="s1">'foo'</span><span class="p">,</span> <span class="s1">'get_vars'</span><span class="p">]</span> </pre></div> <p>As you can see <code>__baz</code> got turned into <code>_ExtendedTest__baz</code> to prevent accidental modification:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">t2</span><span class="o">.</span><span class="n">_ExtendedTest__baz</span> <span class="s1">'overridden'</span> </pre></div> <p>But the original <code>_Test__baz</code> is also still around:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">t2</span><span class="o">.</span><span class="n">_Test__baz</span> <span class="mi">42</span> </pre></div> <p>Double underscore name mangling is fully transparent to the programmer. Take a look at the following example that will confirm this:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">ManglingTest</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">__mangled</span> <span class="o">=</span> <span class="s1">'hello'</span> <span class="k">def</span> <span class="nf">get_mangled</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">__mangled</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ManglingTest</span><span class="p">()</span><span class="o">.</span><span class="n">get_mangled</span><span class="p">()</span> <span class="s1">'hello'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ManglingTest</span><span class="p">()</span><span class="o">.</span><span class="n">__mangled</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="s2">"'ManglingTest' object has no attribute '__mangled'"</span> </pre></div> <p>Does name mangling also apply to method names? It sure does—name mangling affects <em>all</em> names that start with two underscore characters (“dunders”) in a class context:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">MangledMethod</span><span class="p">:</span> <span class="k">def</span> <span class="nf">__method</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="mi">42</span> <span class="k">def</span> <span class="nf">call_it</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">__method</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">MangledMethod</span><span class="p">()</span><span class="o">.</span><span class="n">__method</span><span class="p">()</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="s2">"'MangledMethod' object has no attribute '__method'"</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">MangledMethod</span><span class="p">()</span><span class="o">.</span><span class="n">call_it</span><span class="p">()</span> <span class="mi">42</span> </pre></div> <p>Here’s another, perhaps surprising, example of name mangling in action:</p> <div class="codehilite"><pre><span></span><span class="n">_MangledGlobal__mangled</span> <span class="o">=</span> <span class="mi">23</span> <span class="k">class</span> <span class="nc">MangledGlobal</span><span class="p">:</span> <span class="k">def</span> <span class="nf">test</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="n">__mangled</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">MangledGlobal</span><span class="p">()</span><span class="o">.</span><span class="n">test</span><span class="p">()</span> <span class="mi">23</span> </pre></div> <p>In this example I declared a global variable called <code>_MangledGlobal__mangled</code>. Then I accessed the variable inside the context of a class named <code>MangledGlobal</code>. Because of name mangling I was able to reference the <code>_MangledGlobal__mangled</code> global variable as just <code>__mangled</code> inside the <code>test()</code> method on the class.</p> <p>The Python interpreter automatically expanded the name <code>__mangled</code> to <code>_MangledGlobal__mangled</code> because it begins with two underscore characters. This demonstrated that name mangling isn’t tied to class attributes specifically. It applies to any name starting with two underscore characters used in a class context.</p> <p><strong>Now this was a lot of stuff to absorb.</strong></p> <p>To be honest with you I didn’t write these examples and explanations down off the top of my head. It took me some research and editing to do it. I’ve been using Python for years but rules and special cases like that aren’t constantly on my mind.</p> <p>Sometimes the most important skills for a programmer are “pattern recognition” and knowing where to look things up. If you feel a little overwhelmed at this point, don’t worry. Take your time and play with some of the examples in this article.</p> <p>Make these concepts sink in enough so that you’ll recognize the general idea of name mangling and some of the other behaviors I showed you. If you encounter them “in the wild” one day, you’ll know what to look for in the documentation.</p> <div class="update-box"> <h2>⏰ Sidebar: What’s a “dunder” in Python?</h2> <p>I’ve you’ve heard some experienced Pythonistas talk about Python or watched a few conference talks you may have heard the term <em>dunder</em>. If you’re wondering what that is, here’s your answer:</p> <p>Double underscores are often referred to as <a href="https://nedbatchelder.com/blog/200605/dunder.html" target="_blank">“dunders”</a> in the Python community. The reason is that double underscores appear quite often in Python code and to avoid fatiguing their jaw muscles Pythonistas often shorten “double underscore” to “dunder.”</p> <p>For example, you’d pronounce <code>__baz</code> as “dunder baz”. Likewise <code>__init__</code> would be pronounced as “dunder init”, even though one might think it should be “dunder init dunder.” But that’s just yet another quirk in the naming convention.</p> <p>It’s like a <em>secret handshake</em> for Python developers 🙂</p> </div> </section><section><h2>4. Double Leading and Trailing Underscore: <code>__var__</code></h2> <p>Perhaps surprisingly, name mangling is <em>not</em> applied if a name <em>starts and ends</em> with double underscores. Variables surrounded by a double underscore prefix and postfix are left unscathed by the Python interpeter:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">PrefixPostfixTest</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">__bam__</span> <span class="o">=</span> <span class="mi">42</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">PrefixPostfixTest</span><span class="p">()</span><span class="o">.</span><span class="n">__bam__</span> <span class="mi">42</span> </pre></div> <p>However, names that have both leading and trailing double underscores are reserved for special use in the language. This rule covers things like <code>__init__</code> for object constructors, or <code>__call__</code> to make an object callable.</p> <p>These <em>dunder methods</em> are often referred to as <em>magic methods</em>—but many people in the Python community, including myself, <a href="http://www.pixelmonkey.org/2013/04/11/python-double-under-double-wonder" target="_blank">don’t like that</a>.</p> <p>It’s best to stay away from using names that start and end with double underscores (“dunders”) in your own programs to avoid collisions with future changes to the Python language.</p> </section><section><h2>5. Single Underscore: <code>_</code></h2> <p>Per convention, a single standalone underscore is sometimes used as a name to indicate that a variable is temporary or insignificant.</p> <p>For example, in the following loop we don’t need access to the running index and we can use “<code>_</code>” to indicate that it is just a temporary value:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">32</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="s1">'Hello, World.'</span><span class="p">)</span> </pre></div> <p>You can also use single underscores in unpacking expressions as a “don’t care” variable to ignore particular values. Again, this meaning is “per convention” only and there’s no special behavior triggered in the Python interpreter. The single underscore is simply a valid variable name that’s sometimes used for this purpose.</p> <p>In the following code example I’m unpacking a <code>car</code> tuple into separate variables but I’m only interested in the values for <code>color</code> and <code>mileage</code>. However, in order for the unpacking expression to succeed I need to assign all values contained in the tuple to variables. That’s where “<code>_</code>” is useful as a placeholder variable:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">car</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'red'</span><span class="p">,</span> <span class="s1">'auto'</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mf">3812.4</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">color</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">mileage</span> <span class="o">=</span> <span class="n">car</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">color</span> <span class="s1">'red'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">mileage</span> <span class="mf">3812.4</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">_</span> <span class="mi">12</span> </pre></div> <p>Besides its use as a temporary variable, “<code>_</code>” is a special variable in most Python REPLs that represents the result of the last expression evaluated by the interpreter.</p> <p>This is handy if you’re working in an interpreter session and you’d like to access the result of a previous calculation. Or if you’re constructing objects on the fly and want to interact with them without assigning them a name first:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="mi">20</span> <span class="o">+</span> <span class="mi">3</span> <span class="mi">23</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">_</span> <span class="mi">23</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="n">_</span><span class="p">)</span> <span class="mi">23</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">list</span><span class="p">()</span> <span class="p">[]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">_</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">_</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">_</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">_</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> </pre></div> </section><section><h2>📓 Python Underscore Naming Patterns – Summary</h2> <p>Here’s a quick summary or “cheat sheet” of what the five underscore patterns I covered in this article mean in Python:</p> <table> <thead> <tr> <th>Pattern</th> <th>Example</th> <th>Meaning</th> </tr> </thead> <tbody> <tr> <td><strong>Single Leading Underscore</strong></td> <td><code>_var</code></td> <td>Naming convention indicating a name is meant for internal use. Generally not enforced by the Python interpreter (except in wildcard imports) and meant as a hint to the programmer only.</td> </tr> <tr> <td><strong>Single Trailing Underscore</strong></td> <td><code>var_</code></td> <td>Used by convention to avoid naming conflicts with Python keywords.</td> </tr> <tr> <td><strong>Double Leading Underscore</strong></td> <td><code>__var</code></td> <td>Triggers name mangling when used in a class context. Enforced by the Python interpreter.</td> </tr> <tr> <td><strong>Double Leading and Trailing Underscore</strong></td> <td><code>__var__</code></td> <td>Indicates special methods defined by the Python language. Avoid this naming scheme for your own attributes.</td> </tr> <tr> <td><strong>Single Underscore</strong></td> <td><code>_</code></td> <td>Sometimes used as a name for temporary or insignificant variables (“don’t care”). Also: The result of the last expression in a Python REPL.</td> </tr> </tbody> </table> </section><section><h2>📺 Underscore Patterns – Video Tutorial</h2> <p>Watch a short video tutorial to see first-hand how things like double underscore name mangling work in Python and how they affect your own classes and modules:</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/ALZmCy2u0jQ?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p>Did I miss anything in this explanation? Want to add your own thoughts on the matter? Leave a comment below, I’d appreciate it.</p></section><footer></footer></article>https://dbader.org/blog/meaning-of-underscores-in-pythonTue, 23 May 2017 00:00:00 GMTWhen to Use Pythonhttps://dbader.org/blog/when-to-use-python<article><header><h1>When to Use Python</h1> <p>What is the Python programming language used for in the real world, and when is using Python the right choice?</p> </header><section><figure><img alt="When To Use Python?" src="/blog/figures/when-to-use-python.png" width="1280" height="720"></figure> <p>When I grew up in Germany as a kid there was this craze about “desks that can grow with you.” The idea was you’d buy your kid an adjustable desk and then they’d be able to use it throughout their whole education career.</p> <p>As your kid grows taller, so does his or her desk. Just turn the little crank handle every few months… And voila, you’re right on track for raising the next Albert Einstein or Marie Curie.</p> </section><section><h2 class="crosshead thin">Python is a great<br>“adjustable desk” language.</h2> <p>With the small but important difference that Python is also a <em>much prettier</em> desk. One that you wouldn’t be embarrassed of using past elementary school. And one you’d be okay with showing to your girlfriend/boyfriend. (Okay, time to stop with that desk analogy.)</p> <p>My point is this:</p> <p>What I love about Python is how it <em>scales so well</em> (no pun intended)—from writing simple prototypes to validate an idea, all the way to building “production grade” systems.</p> <p>Sure, sometimes it would be nice to have a compiler and static type checks to lean on—but often I realized that I would’ve never come this far in so little time with Java or C++. And with <a href="https://www.python.org/dev/peps/pep-0484/" target="_blank">optional type hints</a> in Python 3 and <a href="https://www.youtube.com/watch?v=2xWhaALHTvU" target="_blank">type checking tools like <code>mypy</code></a> this gap is starting to close.</p> <p>But not only does Python scale and grow with the project at hand, it also scales and grows <em>with your skills</em> as a developer.</p> <p>It’s relatively easy to get started with Python—but it’s not going to prevent you from growing as a developer and getting impressive real-world work done with it. My friend and fellow Python wrangler <a href="https://talkpython.fm/" target="_blank">Michael Kennedy</a> refers to it as a “full spectrum” language. And I really like that as an analogy.</p> <p>Python spans the gamut from <code>print('hello, world')</code> all the way to running the back-end infrastructure for massive applications like <a href="why-learn-python">Reddit, Instagram, or YouTube</a>.</p> </section><section><h2 class="crosshead thin">Now, is using Python<br><u>always</u> the right choice?</h2> <p>No.</p> <p>No single programming language is.</p> <p>For example, it’s unlikely you’re going to write a real-time operating system kernel in Python. Neither will id Software use it to implement their next-generation rendering engine…</p> <p>But millions of developers around the world are using Python to build web applications, write data-crunching pipelines, generate reports, automate tests, conduct research, and do all kinds of other amazing work in a multitude of domains.</p> </section><section><h2 class="crosshead thin">By learning Python you’re not limiting yourself to a specific niche.</h2> <p>And that’s what I love about this adorable, “adjustable desk” of a language.</p> <p>Happy Pythoning!</p></section><footer></footer></article>https://dbader.org/blog/when-to-use-pythonThu, 18 May 2017 00:00:00 GMTStacks in Pythonhttps://dbader.org/blog/stacks-in-python<article><header><h1>Stacks in Python</h1> <p>How to implement a stack data structure (LIFO) in Python using built-in types and classes from the standard library.</p> </header><section><figure><img alt="" src="/blog/figures/stacks-in-python.png" width="1280" height="720"></figure> <p>A stack is a collection of objects that supports fast <em>last-in, first-out (LIFO)</em> semantics for inserts and deletes. Unlike lists or arrays, stacks typically don’t allow for random access to the objects they contain. The insert and delete operations are also often called <em>push</em> and <em>pop</em>.</p> <p>A useful real-world analogy for a stack data structure is a <em>stack of plates</em>:</p> <blockquote> <p>New plates are added to the top of the stack. And because the plates are precious and heavy, only the topmost plate can be moved (last-in, first-out). To reach the plates lower down in the stack the topmost plates must be removed one by one.</p> </blockquote> <p>Stacks and queues are similar. They’re both linear collections of items and the difference lies in the order that items are accessed in:</p> <p>With a <strong>queue</strong> you remove the item <strong>least recently added</strong> (<em>first-in, first-out</em> or <em>FIFO</em>); and with a <strong>stack</strong> you remove the item <strong>most recently added</strong> (<em>last-in, first-out</em> or <em>LIFO</em>).</p> <p>Performance-wise, a proper stack implementation is expected to take <em>O(1)</em> time for insert and delete operations.</p> <p>Stacks have a wide range of uses in algorithms, for example in language parsing and <a href="https://en.wikipedia.org/wiki/Stack-based_memory_allocation" target="_blank">runtime memory management (“call stack”)</a>. A short and beautiful algorithm using a stack is <a href="https://en.wikipedia.org/wiki/Depth-first_search" target="_blank">depth-first search (DFS)</a> on a tree or graph data structure.</p> <p>Python ships with several stack implementations that each have slightly different characteristics. Let’s take a look at them:</p> </section><section><h2>✅ The <a href="https://docs.python.org/3/tutorial/datastructures.html#using-lists-as-stacks" target="_blank">list</a> Built-in</h2> <p>Python’s built-in <code>list</code> type makes a decent stack data structure as it supports push and pop operations in amortized <em>O(1)</em> time.</p> <p>Python’s lists are implemented as dynamic arrays internally which means they occasional need to resize the storage space for elements stored in them when elements are added or removed. The <a href="http://www.laurentluce.com/posts/python-list-implementation/" target="_blank">list over-allocates its backing storage</a> so that not every push or pop requires resizing and you get an amortized <em>O(1)</em> time complexity for these operations.</p> <p>The downside is that this makes their performance less consistent than the stable <em>O(1)</em> inserts and deletes provided by a linked list based implementation (like <code>collections.deque</code>, see below). On the other hand lists do provide fast <em>O(1)</em> time random access to elements on the stack which can be an added benefit.</p> <p>Here’s an <strong>important performance caveat</strong> when using lists as stacks:</p> <p>To get the amortized <em>O(1)</em> performance for inserts and deletes new items must be added to the end of the list with the <code>append()</code> method and removed again from the end using <code>pop()</code>. Stacks based on Python lists grow to the right and shrink to the left.</p> <p>Adding and removing from the front is much slower and takes <em>O(n)</em> time, as the existing elements must be shifted around to make room for the new element.</p> <div class="codehilite"><pre><span></span><span class="c1"># How to use a Python list as a stack (LIFO):</span> <span class="n">s</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">s</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'eat'</span><span class="p">)</span> <span class="n">s</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'sleep'</span><span class="p">)</span> <span class="n">s</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'code'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span> <span class="p">[</span><span class="s1">'eat'</span><span class="p">,</span> <span class="s1">'sleep'</span><span class="p">,</span> <span class="s1">'code'</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="s1">'code'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="s1">'sleep'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="s1">'eat'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="ne">IndexError</span><span class="p">:</span> <span class="s2">"pop from empty list"</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/collections.html#collections.deque" target="_blank">collections.deque</a> Class</h2> <p>The <code>deque</code> class implements a double-ended queue that supports adding and removing elements from either end in <em>O(1)</em> time (non-amortized).</p> <p>Because deques support adding and removing elements from either end equally well, they can serve both as queues and as stacks.</p> <p>Python’s <a href="https://github.com/python/cpython/blob/947629916a5ecb1f6f6792e9b9234e084c5bf274/Modules/_collectionsmodule.c#L24-L26" target="_blank">deque objects are implemented as doubly-linked lists</a> which gives them excellent and consistent performance for inserting and deleting elements, but poor <em>O(n)</em> performance for randomly accessing elements in the middle of the stack.</p> <p><code>collections.deque</code> is a great choice if you’re looking for a stack data structure in Python’s standard library with the performance characteristics of a linked-list implementation.</p> <div class="codehilite"><pre><span></span><span class="c1"># How to use collections.deque as a stack (LIFO):</span> <span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">deque</span> <span class="n">q</span> <span class="o">=</span> <span class="n">deque</span><span class="p">()</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'eat'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'sleep'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'code'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span> <span class="n">deque</span><span class="p">([</span><span class="s1">'eat'</span><span class="p">,</span> <span class="s1">'sleep'</span><span class="p">,</span> <span class="s1">'code'</span><span class="p">])</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="s1">'code'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="s1">'sleep'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="s1">'eat'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="ne">IndexError</span><span class="p">:</span> <span class="s2">"pop from an empty deque"</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/queue.html" target="_blank">queue.LifoQueue</a> Class</h2> <p>This stack implementation in the Python standard library is synchronized and provides locking semantics to support multiple concurrent producers and consumers.</p> <p>The <a href="https://docs.python.org/3/library/queue.html" target="_blank"><code>queue</code> module</a> contains several other classes implementing multi-producer, multi-consumer queues that are useful for parallel computing.</p> <p>Depending on your use case the locking semantics might be helpful, or just incur unneeded overhead. In this case you’d be better off with using a <code>list</code> or a <code>deque</code> as a general purpose stack.</p> <div class="codehilite"><pre><span></span><span class="c1"># How to use queue.LifoQueue as a stack:</span> <span class="kn">from</span> <span class="nn">queue</span> <span class="kn">import</span> <span class="n">LifoQueue</span> <span class="n">s</span> <span class="o">=</span> <span class="n">LifoQueue</span><span class="p">()</span> <span class="n">s</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'eat'</span><span class="p">)</span> <span class="n">s</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'sleep'</span><span class="p">)</span> <span class="n">s</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'code'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span> <span class="o">&lt;</span><span class="n">queue</span><span class="o">.</span><span class="n">LifoQueue</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x108298dd8</span><span class="o">&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'code'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'sleep'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'eat'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">get_nowait</span><span class="p">()</span> <span class="n">queue</span><span class="o">.</span><span class="n">Empty</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">s</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="c1"># Blocks / waits forever...</span> </pre></div> </section><section><h2>A good default choice: <code>collections.deque</code></h2> <p>If you’re not looking for parallel processing support (or don’t want to handle locking and unlocking manually) your choice comes down to the built-in <code>list</code> type or <code>collections.deque</code>.</p> <p>The difference lies in the data structure used behind the scenes and ease of use.</p> <ul> <li> <p><code>list</code> is backed by a dynamic array which makes it great for fast random access but requires occasional resizing when elements are added or removed. The list over-allocates its backing storage so that not every push or pop requires resizing and you get an amortized <em>O(1)</em> time complexity for these operations. But you do need to be careful to only insert and remove items from the right-hand side (<code>append</code> and <code>pop</code>) or otherwise performance slows down to <em>O(n)</em>.</p> </li> <li> <p><code>collections.deque</code> is backed by a doubly-linked list which optimizes appends and deletes at both ends and provides consistent <em>O(1)</em> performance for these operations. Not only is its performance more stable, the <code>deque</code> class is also easier to use because you don’t have to worry about adding or removing items from “the wrong end.”</p> </li> </ul> <p>For these reasons, <code>collections.deque</code> makes an excellent choice for implementing a stack (LIFO queue) data structure in Python.</p> <p><em><a href="fundamental-data-structures-in-python">Read the full “Fundamental Data Structures in Python” article series here</a>. This article is missing something or you found an error? Help a brother out and leave a comment below.</em></p></section><footer></footer></article>https://dbader.org/blog/stacks-in-pythonTue, 16 May 2017 00:00:00 GMTLet’s Program with Python: Reacting to User Input (Part 4)https://dbader.org/blog/python-intro-reacting-to-user-input<article><header><h1>Let’s Program with Python: Reacting to User Input (Part 4)</h1> <p>In the fourth (and final) class in this series you’ll learn how to make your Python programs interactive by letting them react to user input.</p> </header><section><figure><img alt="" src="/blog/figures/pywelcome-part4.png" width="1280" height="720"></figure> <p><em>In this guest post series by <a href="#author">Doug Farrell</a> you’ll learn the basics of programming with Python from scratch. If you’ve never programmed before or need a fun little class to work through with your kids, you’re welcome to follow along.</em></p> <p><strong>Looking for the rest of the “Let’s Program with Python” series?</strong> Here you go:</p> <ul> <li><a href="python-intro-statements-variables-and-loops">Part 1: Statements, Variables, and Loops</a></li> <li><a href="python-intro-functions-and-lists">Part 2: Functions and Lists</a></li> <li><a href="python-intro-conditionals-and-if-statements">Part 3: Conditionals and “if” Statements</a></li> <li>Part 4: Reacting to User Input (This article)</li> </ul> </section><section><h2>Table of Contents – Part 4</h2> <ol> <li><a href="#lets-write-a-program">Let’s Write a Program Together</a></li> <li><a href="#getting-info-from-the-player">Getting Information From the Player</a></li> <li><a href="#converting-a-string-to-a-number">Converting a String to a Number</a></li> <li><a href="#another-kind-of-loop">Another Kind of Loop</a></li> <li><a href="#more-things-we-can-do-with-lists">More Things We Can Do With Lists</a></li> <li><a href="#how-many-items-are-in-a-list">How Many Items Are in a List?</a></li> <li><a href="#how-to-pick-random-things-from-a-list">How to Pick Random Things From a List?</a></li> <li><a href="#our-completed-guess-my-number-program">Our Completed “Guess My Number” Program</a></li> <li><a href="#congratulations">Congratulations!</a></li> <li><a href="#appendix">Appendix – Python Info That Doesn’t Fit in Class</a></li> </ol> </section><section><h2><a class="anchor" name="lets-write-a-program"></a> Let’s Write a Program Together</h2> <p>For this class we’re going to write a “Guess My Number” game program. In this game the program will pick a random number from 1 to 10 and the player will try to guess what the number is. The program will respond in different ways depending on whether the player guessed correctly or incorrectly. The player can also end the game whenever they want by telling the program to “quit”.</p> <p>The interesting part of this program is you’re going to tell me how to write it instead of the other way around. But before we get started, we need to learn a few more things about Python to help us build our game.</p> </section><section><h2><a class="anchor" name="getting-info-from-the-player"></a> Getting Information From the Player</h2> <p>In order to play our game the player has to interact with it. We need a way to get guesses from the player so the game can compare its secret number to the players guess. To do this we use the <code>input()</code> function.</p> <p>The <code>input()</code> function let’s us ask the user for some information, and then wait for them to enter something using the keyboard. In the Python interactive mode it looks like this:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">guess</span> <span class="o">=</span> <span class="nb">input</span><span class="p">(</span><span class="s2">"Please enter a number: "</span><span class="p">)</span> <span class="n">Please</span> <span class="n">enter</span> <span class="n">a</span> <span class="n">number</span><span class="p">:</span> </pre></div> <p>At the point where the <code>input()</code> function runs, the cursor is at the end of the <code>"Please enter a number: "</code> string, waiting for you to type something.</p> <p>You can type anything you want, when you hit the <code>&lt;ENTER&gt;</code> key whatever you typed will be assigned to the <code>guess</code> variable as a string. This is a very simple way to get input from the user using the keyboard.</p> </section><section><h2><a class="anchor" name="converting-a-string-to-a-number"></a> Converting a String to a Number</h2> <p>We haven’t talked about this yet, but there is a difference between a string like <code>"10"</code> and the number <code>10</code>. Try this in the interactive mode:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="mi">10</span> <span class="o">==</span> <span class="mi">10</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="s2">"10"</span> <span class="o">==</span> <span class="mi">10</span> <span class="bp">False</span> </pre></div> <p>On the first line we are comparing the two number 10’s to each other to see if they are equal. Python knows they are, so it responds by printing <code>True</code> to the screen.</p> <p>But the next comparison, <code>"10" == 10</code>, why does Python respond with <code>False</code>? The simple answer is Python doesn’t think they’re equal.</p> <p>But why aren’t they equal? This can be confusing, <code>"10"</code> looks like the number ten. And <code>10</code> definitely looks like the number ten as well. For Python however, this isn’t true.</p> <p>The number <code>10</code> is exactly that, the numerical value 10. The string <code>"10"</code> is just a string, it has no numerical value, even though <code>"10"</code> looks like ten to us.</p> <p>The difference is the representation. The <code>"10"</code> represents a string to Python, it doesn’t know that string represents ten to us. The <code>10</code> however does mean numerical ten to Python, ten things, ten cars, ten whatever.</p> <p>What does this have to do with our game? A lot actually. When the game starts the program will randomly pick a <em>number</em> from 1 to 10, not a string, a number. However when the player types something into our <code>guess = input("Please enter a number: ")</code> prompt, <code>guess</code> is a string variable.</p> <p>Even if the player enters a “1” and then a “0” and then hits enter, the <code>guess</code> variable will be a string. This is where a problem comes in. Let’s say we call the game’s variable for its number <code>secret_number</code>. If we write some Python code that compares them, like this:</p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="n">secret_number</span> <span class="o">==</span> <span class="n">guess</span><span class="p">:</span> </pre></div> <p>This code will fail because comparing a string to a number will always be <code>False</code>. We need to make Python compare two of the same kinds of things. For our game, both things need to be numbers. We need to convert the player’s <code>guess</code> variable to a number. Python can do this using the <code>int()</code> function. It looks like this:</p> <div class="codehilite"><pre><span></span><span class="n">guess_number</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">guess</span><span class="p">)</span> </pre></div> <p>With this code we’re taking the player’s input, <code>guess</code>, which could be something like “8”, and converting it to the numerical value 8 and assigning it to the new variable <code>guess_number</code>. Now when we compare <code>guess_number</code> with <code>secret_number</code>, they are the same kind of thing (numbers) and will compare correctly when we write Python code like this:</p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="n">guess_number</span> <span class="o">==</span> <span class="n">secret_number</span><span class="p">:</span> </pre></div> </section><section><h2><a class="anchor" name="another-kind-of-loop"></a> Another Kind of Loop</h2> <p>We’ve only used the <code>for</code> loop so far because it’s handy when you know ahead of time how many times you want to loop. For our game program we won’t know ahead of time how many guesses it will take our player to guess the <code>secret_number</code>. We also don’t know how many times they’ll want to play the game.</p> <p>This is a perfect use for the other loop Python supports, the <code>while</code> loop. The <code>while</code> loop is called a <em>conditional loop</em> because it will continue looping until some condition it is testing is True. Here’s an example of a <code>while</code> loop:</p> <div class="codehilite"><pre><span></span><span class="n">game_running</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">while</span> <span class="n">game_running</span><span class="p">:</span> <span class="c1"># Run some Python statements</span> </pre></div> <p>What these program lines mean is that while the variable <code>game_running</code> is <code>True</code>, the while loop will keep looping. This also means something in the <code>while</code> loop will have to change the value of <code>game_running</code> in order for the program to exit the loop.</p> <p>Forgetting to provide a way for the <code>while</code> loop to end creates what’s called an <em>infinite loop</em>. This is usually a bad thing and means in order to exit the program it has to be crashed or stopped in some other way.</p> </section><section><h2><a class="anchor" name="more-things-we-can-do-with-lists"></a> More Things We Can Do With Lists</h2> <p>We’ve used Python lists before to hold things we want to deal with as one thing, like lists of turtles. We’ve created lists and appended things to lists. So far we’ve used the things in the list one at a time using the <code>for</code> loop. But how do we get to the individual things inside a list? For example, suppose I have this list in Python:</p> <div class="codehilite"><pre><span></span><span class="n">names</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"Andy"</span><span class="p">,</span> <span class="s2">"George"</span><span class="p">,</span> <span class="s2">"Sally"</span><span class="p">,</span> <span class="s2">"Sharon"</span><span class="p">,</span> <span class="s2">"Sam"</span><span class="p">,</span> <span class="s2">"Chris"</span><span class="p">]</span> </pre></div> <p>How can I get just the <code>"Sally"</code> name from the <code>names</code> list variable? We use something called <em>list indexing</em> to do that. Everything in a list has a position in the list, and all lists in Python start at position 0. The position is called an index, so to get <code>"Sally"</code> from the list, remembering all lists start at index 0, we do this:</p> <div class="codehilite"><pre><span></span><span class="n">name</span> <span class="o">=</span> <span class="n">names</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> </pre></div> <p>When we do this the variable <code>name</code> will be equal to <code>"Sally"</code> from our list. The <code>[2]</code> above is called the index into the list. We’ve told Python we want the thing inside the <code>names</code> list at index 2.</p> </section><section><h2><a class="anchor" name="how-many-items-are-in-a-list"></a> How Many Items Are in a List?</h2> <p>It’s often useful to be able to find out how many things are in a list. For instance, our <code>names</code> list above has six strings in it. But how could we find this out using Python? We use the <code>len()</code> function. It looks like this:</p> <div class="codehilite"><pre><span></span><span class="n">number_of_names_in_list</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">names</span><span class="p">)</span> </pre></div> <p>This will set the variable <code>number_of_names_in_list</code> equal to six. Notice something about the number of items in the <code>names</code> list and the largest index, the name “Chris”. To get the name “Chris” from our <code>names</code> list we would do this:</p> <div class="codehilite"><pre><span></span><span class="n">name</span> <span class="o">=</span> <span class="n">names</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span> </pre></div> <p>The last thing in the list is at index 5, but the number of things in the list is 6. This is because all lists start with index 0, which is included in the number of things in the list. So for the names list we have indexes 0, 1, 2, 3, 4 and 5, totaling 6 things.</p> </section><section><h2><a class="anchor" name="how-to-pick-random-things-from-a-list"></a> How to Pick Random Things From a List?</h2> <p>Now we know how to pick individual things from a list, how to determine how long a list is and what the maximum index value in a list is. Can we use this information to choose a random thing from a list? For a minute let’s think about our turtle programs, we had a list something like this:</p> <div class="codehilite"><pre><span></span><span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"black"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">,</span> <span class="s2">"organge"</span><span class="p">,</span> <span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"green"</span><span class="p">,</span> <span class="s2">"blue"</span><span class="p">]</span> </pre></div> <p>How could we pick a random color from this list to use when we were creating a turtle? We know the smallest index is 0, which would be the color “black”. We also know by looking at the list that our largest index is 5, the color blue. This is one less than the number of colors in the list. So we could do something like this:</p> <div class="codehilite"><pre><span></span><span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"black"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">,</span> <span class="s2">"organge"</span><span class="p">,</span> <span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"green"</span><span class="p">,</span> <span class="s2">"blue"</span><span class="p">]</span> <span class="n">turtle_color</span> <span class="o">=</span> <span class="n">colors</span><span class="p">[</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">5</span><span class="p">)]</span> </pre></div> <p>This Python statement would set the <code>turtle_color</code> variable to a random color from our <code>colors</code> list. But what if we added more colors to our list? Something like this:</p> <div class="codehilite"><pre><span></span><span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"black"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">,</span> <span class="s2">"organge"</span><span class="p">,</span> <span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"green"</span><span class="p">,</span> <span class="s2">"blue"</span><span class="p">,</span> <span class="s2">"violet"</span><span class="p">,</span> <span class="s2">"pink"</span><span class="p">]</span> <span class="n">turtle_color</span> <span class="o">=</span> <span class="n">colors</span><span class="p">[</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">5</span><span class="p">)]</span> </pre></div> <p>Unless we change the <code>5</code> in the <code>random.randint(5)</code> function we’ll still be picking from the first six colors and ignoring the new ones we added. What if we’re picking random colors all over our program, we’d have to change all the lines that pick a color every time we changed the number of colors in our <code>colors</code> list. Can we get Python to handle this for us? Sure we can, we can use the <code>len()</code> function to help us out. We can change our code to look like this:</p> <div class="codehilite"><pre><span></span><span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"black"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">,</span> <span class="s2">"organge"</span><span class="p">,</span> <span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"green"</span><span class="p">,</span> <span class="s2">"blue"</span><span class="p">,</span> <span class="s2">"violet"</span><span class="p">,</span> <span class="s2">"pink"</span><span class="p">]</span> <span class="n">turtle_color</span> <span class="o">=</span> <span class="n">colors</span><span class="p">[</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">colors</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)]</span> </pre></div> <p>What’s going on here? We still have our <code>colors</code> list variable, but now we’re using the <code>len()</code> function inside our <code>random.randint()</code> function. This is okay, the <code>len()</code> function returns a number and <code>random.randint()</code> expects a number as its second parameter.</p> <p>But now we’re telling <code>random.randint()</code> the upper index limit of the numbers we want to choose from is one less than the number of things in the <code>colors</code> list variable. And as we’ve seen, one less than the number of things in a list will always be the highest index in the list. By using the code above we can add or subtract as many items from the <code>colors</code> list as we want and our random selection will still work, using all the things in the list.</p> </section><section><h2><a class="anchor" name="our-completed-guess-my-number-program"></a> Our Completed “Guess My Number” Program</h2> <p>Here’s our Guess My Number program, complete with comments:</p> <div class="codehilite"><pre><span></span><span class="c1">#</span> <span class="c1"># Guess My Number</span> <span class="c1">#</span> <span class="kn">import</span> <span class="nn">random</span> <span class="c1"># Set our game ending flag to False</span> <span class="n">game_running</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">while</span> <span class="n">game_running</span><span class="p">:</span> <span class="c1"># Greet the user to our game</span> <span class="k">print</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="s2">"I'm thinking of a number between 1 and 10, can you guess it?"</span><span class="p">)</span> <span class="c1"># Have the program pick a random number between 1 and 10</span> <span class="n">secret_number</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="c1"># Set the player's guess number to something outside the range</span> <span class="n">guess_number</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span> <span class="c1"># Loop until the player guesses our number</span> <span class="k">while</span> <span class="n">guess_number</span> <span class="o">!=</span> <span class="n">secret_number</span><span class="p">:</span> <span class="c1"># Get the player's guess from the player</span> <span class="k">print</span><span class="p">()</span> <span class="n">guess</span> <span class="o">=</span> <span class="nb">input</span><span class="p">(</span><span class="s2">"Please enter a number: "</span><span class="p">)</span> <span class="c1"># Does the user want to quit playing?</span> <span class="k">if</span> <span class="n">guess</span> <span class="o">==</span> <span class="s2">"quit"</span><span class="p">:</span> <span class="n">game_running</span> <span class="o">=</span> <span class="bp">False</span> <span class="k">break</span> <span class="c1"># Otherwise, nope, player wants to keep going</span> <span class="k">else</span><span class="p">:</span> <span class="c1"># Convert the players guess from a string to an integer</span> <span class="n">guess_number</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">guess</span><span class="p">)</span> <span class="c1"># Did the player guess the program's number?</span> <span class="k">if</span> <span class="n">guess_number</span> <span class="o">==</span> <span class="n">secret_number</span><span class="p">:</span> <span class="k">print</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="s2">"Congratulations, you guessed my number!"</span><span class="p">)</span> <span class="c1"># Otherwise, whoops, nope, go around again</span> <span class="k">else</span><span class="p">:</span> <span class="k">print</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="s2">"Oh, to bad, that's not my number..."</span><span class="p">)</span> <span class="c1"># Say goodbye to the player</span> <span class="k">print</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="s2">"Thanks for playing!"</span><span class="p">)</span> </pre></div> </section><section><h2><a class="anchor" name="congratulations"></a> Congratulations!</h2> <p>We’ve completed our course and I hope you’ve had as much fun as I had! We’ve written some pretty amazing programs together and learned quite a bit about programming and Python along the way. My wish is this interested you enough to keep learning about programming and to continue on to discover new things you can do with Python.</p> <hr> </section><section><h2><a class="anchor" name="appendix"></a>Appendix – Python Info That Doesn’t Fit in Class</h2> </section><section><h2>Differences Between Python and Other Languages</h2> <p>There are many programming languages out in the wild you can use to program a computer. Some have been around for a long time, like Fortran and C, and some are quite new, like Dart or Go. Python falls in the middle ground of being fairly new, but quite mature.</p> <p>Why would a programmer choose one language to learn over another? That’s a somewhat complicated question as most languages will allow you to do anything you want. However it can be difficult to express what you want to do with a particular language instead of something else.</p> <p>For instance, Fortran excels at computation and in fact it’s name comes from Fromula Translation (ForTran). However it’s not known as a great language if you need to do a lot of string/text manipulation. The C programming language is a great language if your goal is to maximize the performance of your program. If you program it well you can create extremely fast programs. Notice I said “if you program it well”, if you don’t you can completely crash not only your program, but perhaps even your computer. The C language doesn’t hold your hand to prevent you from doing things that could be bad for your program.</p> <p>In addition to how well a language fits the problem you’re trying to solve, it might not be able to be used with the tools you like, or might not provide the tools you need, a particular language just may not appeal to you visually and appear ugly to you.</p> <p>My choice of teaching Python fits a “sweet spot” for me. It’s fast enough to create the kinds of programs I want to create. It’s visually very appealing to me, and the grammar and syntax of the language fit the way I want to express the problems I’m trying to solve.</p> </section><section><h2>Python Vocabulary</h2> <p>Let’s talk about some of the vocabulary used in the class and what it means. Programming languages have their own “jargon”, or words, meaning specific things to programmers and that language. Here are some terms we’ve used in relation to Python.</p> <p><strong>IDLE – command prompt</strong>: IDLE is the programming environment that comes with Python. It’s what’s called an IDE, or Integrated Development Environment, and pulls together some useful things to help write Python programs. When you start IDLE it opens up a window that has the Python interactive prompt <code>&gt;&gt;&gt;</code> in it.</p> <p>This is a window running the Python interpreter in interactive mode. This is where you can play around with some simple Python program statements. It’s kind of a sandbox where you can try things out. However, there is no way to save or edit your work; once the Python interpreter runs your statements, they’re gone.</p> <p><strong>IDLE – editor window</strong>: The file window (File → New Window) opens up a simple text editor. This is like Notepad in Windows, except it knows about Python code, how to format it and colorizes the text. This is where you can write, edit and save your work and run it again later. When you run this code, behind the scenes IDLE is running the program in the Python interpreter, just like it is in the first IDLE window.</p> <p><strong>Syntax Highlighting</strong>: When we edit code in the file window of IDLE it knows about Python code. One of the things this means is the editor can “colorize”, or syntax highlight, various parts of the Python code you’re entering. It sets the keywords of Python, like for and if, to certain colors. Strings to other colors and comments to another. This is just the file window being helpful and providing syntax highlighting to make it easier for the programmer to read and understand what’s going on in the program.</p> <p><strong>Python Command Line</strong>: In Windows if you open up a command line window, what used to be called a DOS box, and run python, the system will respond with the Python command prompt <code>&gt;&gt;&gt;</code>. At this point you’re running Python in it’s interactive mode, just like when you’re inside of IDLE. In fact they are the same thing, IDLE is running it’s own Python command line inside the window, they are functionally identical.</p> <p>You might think “what use is that?”, and I agree, I’d rather work in IDLE if I’m going to using the interactive mode and play with the sandbox mode and the <code>&gt;&gt;&gt;</code> command prompt. The real use of the Python command line is when you enter something like this at the system command prompt:</p> <div class="codehilite"><pre><span></span>python myprogram.py </pre></div> <p>If I’ve written a program called <code>myprogram.py</code> and entered the line above, instead of going into interactive mode, Python will read <code>myprogram.py</code> and run the code. This is very useful if you’re written a program you want to use and not run inside of IDLE. As a programmer I run programs in this manner all day long, and in many cases these programs run essentially forever as servers.</p> <p><strong>Attribute and Property</strong>: We’ve thrown around the terms “attribute” and “property” kind of randomly, and this can lead to some confusion. The reason it’s confusing is these things mean essentially the same thing. When talking about programming there is always the goal to use specific words and terms to eliminate confusion about what you’re talking about.</p> <p>For instance let’s talk about you. You have many qualities that different people want to express. Your friends want to know your name and phone number. Your school wants to know that as well, and your age, the grade you’re in and our attendance record. In programming terms we can think of these as attributes or properties about you.</p> <p>The attributes and properties of a thing (you for example) help get more specific information about the thing. And the specific information wanted depends on the audience asking. For example when meeting someone new they are more likely to be interested in your name property. Whereas your school might be more interested in your attendance property.</p> <p>In Python we’ve been working with turtles, and those turtles have attributes and properties. For example a turtle as a property called forward. This property happens to be a function that moves the turtle forward, but it’s still a property of the turtle. In fact all the properties and attributes associated with a turtle are expressed as functions. These functions either make the turtle do something, or tell us something about the turtle.</p> <p>Attributes and properties lead into a concept of Object Oriented Programming (OOP) that adds the concept of “things” to programs rather than just data and statements. Object Oriented Programming is beyond the scope of this book, but is very interesting and useful.</p> </section><section><h2>Interpreter vs Compiler</h2> <p>In class you’ve heard me talk about the Python interpreter, what does this mean. As we’ve talked about, computer languages are a way for people to tell a computer what to do. But the truth is a computer only understands 0’s and 1’s, so how does a computer understand a language like Python? That’s where a translation layer comes into play, and that translation layer is the interpreter for Python (and other interpreted languages) and a compiler for compiled languages. Let’s talk about compilers first.</p> <p><strong>Compiler</strong>: A compiler is a translator that converts a computer language into machine code, the 0’s and 1’s a computer understands. A compiler usually produces an executable file, on Windows machines this is a file that ends in .exe. This file contains machine code information the computer can run directly. Languages like C, C++ and Fortran are compiled languages and have to be processed by a compiler before the program can run. One thing this means is you can’t run a compiled language directly, you have to compile it first. It also means there is nothing like the interactive mode (the <code>&gt;&gt;&gt;</code> prompt in Python) in a compiled language. The entire program has to be compiled, it can’t compile and run single statements.</p> <p><strong>Interpreter</strong>: Here’s where things get a little more confusing. Most interpreted languages also have a compiled step, but the output of that step isn’t machine code, no 0’s and 1’s. Instead the compilation step produces what is called ByteCode. The ByteCode is kind of an intermediate step between the near English computer language and the machine code understood by the computer.</p> <p>The ByteCode can’t be run directly, it is run by a thing called a virtual machine. When the program is run, the virtual machine reads the ByteCode and it generates the computer specific machine code that actually is run by the computer. When you run the program the virtual machine is constantly “interpreting” the ByteCode and generating computer specific machine code. Unlike a compiled language, languages like Python with virtual machines can provide an interactive mode (the <code>&gt;&gt;&gt;</code> prompt) as the interpreter and virtual machine can translate and run program statements on the fly.</p> <p><strong>Advantages And Disadvantages</strong>: So why would a programmer pick a compiled language over an interpreted language, and vice versa? Well, what we said before still applies, expressiveness of the language, style, etc, those are important things to think about when choosing a language for a project. But there are some differences beyond that. In general compiled languages produce programs that run faster than programs produced by an interpreter. Remember, compiled languages produce programs containing machine code that can be run directly, whereas interpreted languages usually have a virtual machine between the ByteCode and the machine code, so there’s a speed penalty there. However, also keep in mind modern computers are so fast this difference is less important. In addition, interpreted languages are being constantly improved so their performance gets better and better, so the performance difference between the two is shrinking.</p> <p>Most interpreted languages also offer safety features to prevent the programmer from crashing the program. Interpreted languages make it hard to corrupt memory. They make it difficult to get direct access to the hardware. They don’t force the programmer to manage memory explicitly. Compiled programs like C offer none of this, and therefore it’s easy to do all of those things, which can put your program at risk, unless you’re a skilled programmer. The safety features can be added to a C program, but this has to be done manually by the programmer and isn’t handled by the language natively.</p> </section><section><h2>Python Reference Materials</h2> <p>Included below is a list of reference materials to help you go further in your study of Python.</p> <ul> <li><a href="https://www.python.org/" target="_blank">Python Website</a> – Main Python website</li> <li><a href="https://docs.python.org/3/" target="_blank">Python Documentation</a> – Official Python 3 Documentation</li> <li><a href="https://docs.python.org/3/library/turtle.html" target="_blank">Python Turtle Documentation</a> – Official Python Documentation for Turtle</li> <li><a href="/python-basics">Python Tutorials for Beginners on dbader.org</a></li> <li><a href="http://www.learnpython.org/" target="_blank">Learn Python</a> – An interesting tutorial to help learn Python</li> <li><a href="http://interactivepython.org/runestone/static/thinkcspy/toc.html" target="_blank">How To Think Like A Computer Scientist</a> – Interesting and interactive way to learn Python</li> <li><a href="http://www.pygame.org/news.html" target="_blank">PyGame</a> – An add on module for writing games with Python</li> </ul></section><footer></footer></article>https://dbader.org/blog/python-intro-reacting-to-user-inputThu, 11 May 2017 00:00:00 GMTPythonistaCafe: A Peer-to-Peer Learning Community for Python Developershttps://dbader.org/blog/introducing-pythonistacafe<article><header><h1>PythonistaCafe: A Peer-to-Peer Learning Community for Python Developers</h1> <p>Introducing PythonistaCafe—an invite-only, online community of Python and software development enthusiasts helping each other succeed and grow.</p> </header><section><figure><img alt="Introducing Pythonista Cafe" src="/blog/figures/pythonistacafe-announcement.png" width="670" height="335"></figure> <p>Most programmers I know say they sometimes feel “stuck” in their learning progress. Whether you’re a beginner, an intermediate developer, or an experienced senior dev—you’ll eventually hit a point where you feel like you’re no longer making progress:</p> <ul> <li>Where you think you’re no longer learning new things.</li> <li>Where it feels like you’re making <em>backwards</em> progress, and your skills seemingly atrophy and get <em>worse</em> over time; or</li> <li>Where you recognize you’re good at what you do—but you don’t know what to learn next, or how.</li> </ul> <p>It happens to all of us. For example, I found that I’m the happiest when I can learn new things and then apply them in practice or teach them to others. Every time I hit a plateau like that this feeling of being “stuck” strikes at the core of my identity. If it lasts for too long I get uneasy. It’s almost like I’m losing my sense of purpose.</p> <p>Not a nice feeling at all.</p> <p>But I’ve also been around the block enough that I know I can overcome it, that I can shake it off eventually. What usually helps me get my bearings straight again is talking to my friends who also work in the tech industry. I found it’s important to talk with people who are in the same boat:</p> <p>The folks that I went to university with or former colleagues who can relate to my “geek angst.”</p> <p>I get it that many of the things us programmers are struggling with seem funny or weird to “outsiders”—</p> <p>I’m not expecting my friend who works as an insurance broker to understand the strange headspace I can get in when I think that “my learning progress is stuck” (after having worked in the industry for a decade.)</p> <p>Let’s be honest here, this sounds ridiculous to a “normal” person… 🙂</p> <p>But it’s a real feeling—and one that’s quite common in the programming community.</p> <p>The quickest way I found to overcome it is to talk to people that you feel comfortable pouring your geeky heart out to. Whether you’re learning Python on your own, working by yourself as a freelancer, or if you’re the only Pythonista at your company:</p> <p>Experiencing this sense of community and exchanging your thoughts with other techies will have a huge benefit on your quality of life. I know it did on mine. And I want every developer I know to be able to experience the same:</p> <p>That’s why I started <a href="https://www.pythonistacafe.com?utm_source=dbaderorg&amp;utm_medium=web&amp;utm_campaign=pc-announcement" target="_blank">PythonistaCafe, a peer-to-peer learning community for Python developers</a>.</p> <p>It’s an idea I’ve been kicking around and refining since July 2016. And over the last few months it finally became a reality.</p> <p>A good way to think of PythonistaCafe is to see it as a <em>club of mutual improvement for Python enthusiasts</em>.</p> <p>We have members located all over the world, and with a wide range of proficiency levels. I’m impressed by their diverse skill set and the depth and quality of the conversations we had. Every day we discuss a broad range of programming questions, career advice, and other topics:</p> <figure><img alt="PythonistaCafe Screenshot" src="/blog/figures/pythonistacafe-screenshot.jpg" width="1122" height="366"></figure> <p>We now even have some open-source projects and Kaggle data science competitions we’re collaborating on to help people build up their portfolio and gain experience. It’s been a ton of fun—and a great “Python support group.”</p> <p>You can learn more about PythonistaCafe, our community values, and what we’re all about at <a href="https://www.pythonistacafe.com?utm_source=dbaderorg&amp;utm_medium=web&amp;utm_campaign=pc-announcement" target="_blank"><strong>www.pythonistacafe.com</strong></a>.</p> <p>In part, I also started PythonistaCafe to “scratch my own itch”—it is my new home when it comes to Python. Each week I receive a ton of emails asking me for programming or career advice. And to be honest, it’s hard to keep up with them all.</p> <p>If you need access to me to help solve a Python problem or get advice in what direction to go, the <a href="https://www.pythonistacafe.com?utm_source=dbaderorg&amp;utm_medium=web&amp;utm_campaign=pc-announcement" target="_blank">PythonistaCafe forums</a> are where you can find me. I’m checking the forums and replying to topics and questions every single day. If you’re looking for an alternative to my <a href="https://dbader.org/products/mentorship/">1:1 Python coaching program</a> a membership in <a href="https://www.pythonistacafe.com?utm_source=dbaderorg&amp;utm_medium=web&amp;utm_campaign=pc-announcement" target="_blank">PythonistaCafe</a> might be a great fit for you.</p></section><footer></footer></article>https://dbader.org/blog/introducing-pythonistacafeTue, 09 May 2017 00:00:00 GMTLet’s Program with Python: Conditionals and “if” Statements (Part 3)https://dbader.org/blog/python-intro-conditionals-and-if-statements<article><header><h1>Let’s Program with Python: Conditionals and “if” Statements (Part 3)</h1> <p>In part three of this four-part Python introduction you’ll see how to teach your program how to make decisions with conditionals and if-statements.</p> </header><section><figure><img alt="" src="/blog/figures/pywelcome-part3.png" width="1280" height="720"></figure> <p><em>In this guest post series by <a href="#author">Doug Farrell</a> you’ll learn the basics of programming with Python from scratch. If you’ve never programmed before or need a fun little class to work through with your kids, you’re welcome to follow along.</em></p> <p><strong>Looking for the rest of the “Let’s Program with Python” series?</strong> Here you go:</p> <ul> <li><a href="python-intro-statements-variables-and-loops">Part 1: Statements, Variables, and Loops</a></li> <li><a href="python-intro-functions-and-lists">Part 2: Functions and Lists</a></li> <li>Part 3: Conditionals and “if” Statements (This article)</li> <li><a href="python-intro-reacting-to-user-input">Part 4: Reacting to User Input</a></li> </ul> </section><section><h2>Table of Contents – Part 3</h2> <ul> <li><a href="#lets-get-those-turtles-thinking">Let’s Get Those Turtles Thinking</a></li> <li><a href="#new-turtle-drawing-functions">New Turtle Drawing Functions</a></li> <li><a href="#new-modules-and-functions">New Modules and Functions</a></li> <li><a href="#lets-get-our-program-going">Let’s Get Our Program Going</a></li> <li><a href="#conditionals-and-if-statements">Conditionals and “if” Statements</a></li> <li><a href="#conclusion">Conclusion</a></li> </ul> </section><section><h2><a class="anchor" name="lets-get-those-turtles-thinking"></a> Let’s Get Those Turtles Thinking</h2> <p>In our last class we used a Python list to help us get multiple turtles drawing on the screen. We could keep adding turtles to our hearts content and the program would faithfully make each turtle draw our flower. This worked great for drawing the well controlled structure of the flower.</p> <p>But what if we want to draw something that’s randomly generated, something where the turtles draw something and we don’t know ahead of time what that will be? How can we use what we know already to help us do that?</p> <p>Let’s teach our program how to make decisions and do things on its own. Here is an image of one possible graphical outcome for our class:</p> <figure><img alt="image" src="/blog/figures/pywelcome-random-turtles.png" width="1280" height="1102"></figure> </section><section><h2><a class="anchor" name="new-turtle-drawing-functions"></a> New Turtle Drawing Functions</h2> <p>We’re going to create a new program where our turtles use some new drawing functions and new modules to create a randomly drawn image. Let’s learn the new turtle drawing functions first.</p> <p>Let’s start out by starting <strong>Idle</strong>, opening a new program editor window and creating a new Python program. In this new program let’s start as we’ve done before by entering this Python statement:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> </pre></div> <p>Save this program to a new file name, somewhere you can remember where to find it.</p> <p><strong>Get the Turtle Screen: turtle.Screen()</strong></p> <p>The first new turtle drawing function we’re going to learn isn’t really about the turtles at all, but about the screen they draw on. Up until now we haven’t cared to much about the screen the turtles are drawing on, we’ve just let the turtles create it as needed and away we go.</p> <p>But now we want to modify something about the screen. In order to do that we have to first get the screen in a manner that we can change it. As with everything we do with Python programming, any time we want to get something so we can modify it, we save it to a variable. To get the screen we enter the following into our new program:</p> <div class="codehilite"><pre><span></span><span class="n">screen</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Screen</span><span class="p">()</span> </pre></div> <p>This calls another function of our <code>turtle</code> module, <code>Screen()</code>, which gets the screen the module will use to draw turtles on, and saves it in the newly created variable <code>screen</code>.</p> <p>Notice how the <code>Screen()</code> function of the turtle module has it’s first letter capitalized, like when we create a turtle with <code>Turtle()</code>.</p> <p><strong>Set the Screen Size: turtle.setup()</strong></p> <p>Until now we’ve let the turtle module create our window to be whatever size it wants. We can control this using the <code>setup()</code> function of a turtle. I’m not sure why this is a turtle function instead of a screen function, but sometimes programming is like that. This function looks like this:</p> <div class="codehilite"><pre><span></span><span class="n">turtle</span><span class="o">.</span><span class="n">setup</span><span class="p">(</span><span class="mi">1024</span><span class="p">,</span> <span class="mi">768</span><span class="p">)</span> </pre></div> <p>This Python statement sets our turtle drawing window to be 1024 pixels wide by 768 pixels tall.</p> <p><strong>Set the Background Color of the Screen: screen.bgcolor()</strong></p> <p>Now that we have a variable that represents the screen, we can modify a feature of it. We’re going to change the background color from white to some other color. We do this using this Python statement:</p> <div class="codehilite"><pre><span></span><span class="n">screen</span><span class="o">.</span><span class="n">bgcolor</span><span class="p">(</span><span class="s2">"#FFFFE0"</span><span class="p">)</span> </pre></div> <p>This statement shows how to use the <code>screen</code> variable and call one of its functions, <code>bgcolor()</code> (short for <em>background color</em>) to set the background color on the screen.</p> <p>If we save and run this you’ll see an empty turtle window that has a light yellow color instead of white. The light yellow color is the <code>"#FFFFE0"</code> we passed as a parameter to the <code>bgcolor()</code> function.</p> <p>So what does <code>"#FFFFE0"</code> mean? We could have just passed <code>"yellow"</code> to the <code>bgcolor()</code> function, like we’ve done with our turtles, but that yellow is pretty intense and I wanted something lighter for a background color.</p> <p>So we’ve used a different way to define a color, this way comes right out of <strong>HTML</strong> (web page) coding. The <code>"#FFFFE0"</code> value represents setting the RGB (Red / Green / Blue) color value, each two character portion of the string <code>FFFFE0</code> represents a value from 0 - 255 in <a href="https://en.wikipedia.org/wiki/Hexadecimal" target="_blank">hexadecimal</a> (base 16, common in programming). This breaks down like this:</p> <div class="codehilite"><pre><span></span>FF FF E0 | | | | | +--- 224 Blue | +------ 255 Green +--------- 255 Red </pre></div> <p>This somewhat complex color code let’s us pick a color much more precisely than the limited pre-defined set of named colors (like <code>"red"</code> or <code>"yellow"</code>) that are inside the turtle module.</p> <p><strong>Turtles Are Rubber Stamps!</strong></p> <p>We can also use our turtles as rubber stamps! By this I mean we can tell the turtle to leave a permanent image of itself at any point the turtle exists on the screen. We do this by using the turtle <code>stamp()</code> function, which looks like this:</p> <div class="codehilite"><pre><span></span><span class="n">turtle</span><span class="o">.</span><span class="n">stamp</span><span class="p">()</span> </pre></div> <p>Running this Python statement makes a “stamp” of our turtle on the screen. When next we move the turtle you’ll see the stamp it left behind, kind of like bread crumbs of where its been. Let’s see how this works by entering the following into our program to make it look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="n">screen</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Screen</span><span class="p">()</span> <span class="n">turtle</span><span class="o">.</span><span class="n">setup</span><span class="p">(</span><span class="mi">1024</span><span class="p">,</span> <span class="mi">768</span><span class="p">)</span> <span class="n">screen</span><span class="o">.</span><span class="n">bgcolor</span><span class="p">(</span><span class="s2">"#FFFFE0"</span><span class="p">)</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"red"</span><span class="p">)</span> <span class="k">for</span> <span class="n">side</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">4</span><span class="p">):</span> <span class="n">t1</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">stamp</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> </pre></div> <p>When we save and run this program we should end up with a box outlined in red and a turtle “stamp” at each corner. The screen should look like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-stamping-1.png" width="1024" height="790"></figure> </section><section><h2><a class="anchor" name="new-modules-and-functions"></a> New Modules and Functions</h2> <p>In order to make our new program have random behavior we need to import a new module, logically enough called “random”. The <code>random</code> module, like the <code>turtle</code> module, brings additional functionality into our program so we can use it.</p> <p>Add this line at the top of our program right under the <code>import turtle</code> statement:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">random</span> </pre></div> <p>Just like the turtle module this doesn’t do anything immediately, but now our program has access to the functions in the <code>random</code> module.</p> <p><strong>Pick a Number, Any Number: random.randint()</strong></p> <p>The module <code>random</code>, as the name suggests, creates randomness. We’ll use the functions in the random module to make our turtle drawing less predictable, and maybe more interesting.</p> <p>One of those functions on the module is called <code>randint()</code>, and it generates random integers. If we jump over to our Idle interactive window we can try out the function.</p> <p>Enter this into our <strong>Idle</strong> interactive window to try the <code>randint()</code> function out:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="mi">4</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="mi">10</span> </pre></div> <p>You can see that just like the functions in the turtle module we have to use the module name random and a dot (<code>.</code>) character before the function we want.</p> <p>In the above lines we’ve used the <code>randint()</code> function twice and it returned a different number each time. This is what <code>randint()</code> does, it returns randomly generated integers. This also means that the numbers you’ll see in your Idle window when you run this example will (likely) be different.</p> <p>The two numbers we passed to it (0 and 10) are parameters telling <code>randint()</code> the beginning and ending limits of numbers we want it to generate. In our case we want integer numbers ranging from 0 to 10, including both 0 and 10. Random number generators are used a lot in game programming to create unexpected behavior and challenges for the player.</p> </section><section><h2><a class="anchor" name="lets-get-our-program-going"></a> Let’s Get Our Program Going</h2> <p>Let’s get our random turtle program going so we can add things to it. Make your program look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="kn">import</span> <span class="nn">random</span> <span class="n">screen</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Screen</span><span class="p">()</span> <span class="n">screen</span><span class="o">.</span><span class="n">bgcolor</span><span class="p">(</span><span class="s2">"#FFFFE0"</span><span class="p">)</span> <span class="k">for</span> <span class="n">move</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="k">for</span> <span class="n">a_turtle</span> <span class="ow">in</span> <span class="n">turtles</span><span class="p">:</span> <span class="n">move_turtle</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">)</span> </pre></div> <p>If we save and try to run the above program we’ll get errors. Why is this?</p> <p>Well for a couple of reasons, we have no variable named <code>turtles</code> and the function <code>move_turtle()</code> isn’t defined. Let’s fix that. Like our flower program we want to create a list of turtles, and we’ll need to define our <code>move_turtle()</code> function.</p> <p>So make your program look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="kn">import</span> <span class="nn">random</span> <span class="n">screen</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Screen</span><span class="p">()</span> <span class="n">screen</span><span class="o">.</span><span class="n">bgcolor</span><span class="p">(</span><span class="s2">"#FFFFE0"</span><span class="p">)</span> <span class="k">def</span> <span class="nf">move_turtle</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="k">pass</span> <span class="n">turtles</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">move</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="k">for</span> <span class="n">a_turtle</span> <span class="ow">in</span> <span class="n">turtles</span><span class="p">:</span> <span class="n">move_turtle</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">)</span> </pre></div> <p>Now when we save and run our program it doesn’t crash with an error, but it doesn’t do anything other than open a light yellow window.</p> <p>Why is that? Again, a couple of reasons. We haven’t defined any turtles in our <code>turtles</code> list variable. We’ve also defined our <code>move_turtle()</code> function, but it doesn’t do anything. The <code>pass</code> statement is just a placeholder that makes program work, but doesn’t provide any functionality. First things first, let’s create our turtles.</p> <p><strong>Getting a Variable From a Function</strong></p> <p>In our flower program when we wanted to create our turtles we did so by copying the turtle creation and setup code for every turtle we wanted. Then we put all those turtles into a list we called <code>turtles</code>.</p> <p>This works fine, but let’s do something clever and create a function to create our turtle for us. And let’s define it so we can set the color of the turtle by passing the color as a parameter to the function. Here’s a function that will do just that:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">create_turtle</span><span class="p">(</span><span class="n">color</span><span class="p">):</span> <span class="n">t</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="n">color</span><span class="p">)</span> <span class="k">return</span> <span class="n">t</span> </pre></div> <p>Notice at the end of the <code>create_turtle(color)</code> definition, the <code>return t</code> statement. What does this do?</p> <p>This is how to return the turtle we just created for use in the rest of the program. We’ve seen this before when we used the <code>t1 = turtle.Turtle()</code> statement. The <code>turtle.Turtle()</code> function <em>returns</em> a turtle, and that returned turtle is assigned to the variable <code>t1</code>. In our case we’re returning the turtle we created, what we called <code>t</code>, so it can be saved someplace in our program and used later.</p> <p>Now we have a function that will create a turtle for us that will draw with the color we’ve asked for. But we need to create multiple turtles to put into our <code>turtles</code> list variable.</p> <p>The <code>create_turtle()</code> function only creates one turtle, how can we create multiple turtles with it? An easy way to do this is to create another function using <code>create_turtles()</code> inside a loop to create our list of turtles. Here’s a function that does that:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">create_turtles</span><span class="p">(</span><span class="n">colors</span><span class="p">):</span> <span class="n">turtles</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">color</span> <span class="ow">in</span> <span class="n">colors</span><span class="p">:</span> <span class="n">t</span> <span class="o">=</span> <span class="n">create_turtle</span><span class="p">(</span><span class="n">color</span><span class="p">)</span> <span class="n">turtles</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">return</span> <span class="n">turtles</span> </pre></div> <p>Here we’ve created a function <code>create_turtles(colors)</code> (notice the plural on both the name of the function and the parameter, this just helps us be clear what our intent is) that creates a list of turtles. We use this function like this:</p> <div class="codehilite"><pre><span></span><span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"black"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">,</span> <span class="s2">"orange"</span><span class="p">,</span> <span class="s2">"green"</span><span class="p">]</span> <span class="n">turtles</span> <span class="o">=</span> <span class="n">create_turtles</span><span class="p">(</span><span class="n">colors</span><span class="p">)</span> </pre></div> <p>In the above code we created a variable <code>colors</code> containing a list of four valid turtle colors. We then passed the list to our <code>create_turtles()</code> function. Inside that function we create an empty turtles list with the <code>turtles = []</code> statement.</p> <p>Then we start a <code>for</code> loop taking one color at a time from the <code>colors</code> list parameter, passes that to our <code>create_turtle()</code> function, which creates a turtle that draws in that color.</p> <p>We then use the <code>turtles.append(t)</code> statement to add the turtle to our <code>turtles</code> variable. The <code>append()</code> function is part of the functionality associated with lists, and lets us add elements to the end of the list programmatically. At the end of the loop we return our <code>turtles</code> list variable so it can be used later.</p> <p>If we save and run this program it works, but doesn’t draw anything but the last green turtle on the screen. Remember turtles are all created in the center of the screen, so all four are there, just stacked on top of each other.</p> <p>Let’s put some code in our <code>move_turtle(t)</code> function to get those turtles moving.</p> <p><strong>Moving Turtles Randomly</strong></p> <p>We want our turtles to draw randomly around the screen, so inside the <strong>draw_turtle(t)</strong> function is where we’re going to use our <strong>random.randint()</strong> function we learned about earlier. We also want to <em>stamp</em> a turtle on the screen with every move, which is where we’ll use our <strong>stamp()</strong> function. Here’s a function that will turn a turtle a random angle and move it a random distance:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">move_turtle</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">stamp</span><span class="p">()</span> <span class="n">angle</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="o">-</span><span class="mi">90</span><span class="p">,</span> <span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="n">angle</span><span class="p">)</span> <span class="n">distance</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">50</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">distance</span><span class="p">)</span> </pre></div> <p>This function does a couple of things. First, it expects a turtle as a parameter variable, in the example above that parameter variable is <code>t</code>. The first thing the function does is use our turtle <code>t</code> to <code>stamp()</code> a turtle image on the screen.</p> <p>It then uses the <code>random.randint()</code> function to create an <code>angle</code> variable set to between -90 and 90 degrees. This allows our turtle to turn left or right some random amount. We pass this random <code>angle</code> variable to our <code>t.turn(angle)</code> function to turn our <code>t</code> turtle.</p> <p>We then do a similar thing to create a random <code>distanace</code> varible set to between 50 and 100. We use this variable in our <code>t.forward(distance)</code> function call to move our <code>t</code> turtle forward some random distance.</p> <p><strong>Our Program So Far</strong></p> <p>Let’s see what we’ve got for our program so far:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="kn">import</span> <span class="nn">random</span> <span class="n">screen</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Screen</span><span class="p">()</span> <span class="n">turtle</span><span class="o">.</span><span class="n">setup</span><span class="p">(</span><span class="mi">1024</span><span class="p">,</span> <span class="mi">768</span><span class="p">)</span> <span class="n">screen</span><span class="o">.</span><span class="n">bgcolor</span><span class="p">(</span><span class="s2">"#FFFFE0"</span><span class="p">)</span> <span class="c1"># The number of turtles to create and what color to create them with</span> <span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"black"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">,</span> <span class="s2">"orange"</span><span class="p">,</span> <span class="s2">"green"</span><span class="p">]</span> <span class="c1"># Create a new turtle with a certain color</span> <span class="k">def</span> <span class="nf">create_turtle</span><span class="p">(</span><span class="n">color</span><span class="p">):</span> <span class="n">t</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="n">color</span><span class="p">)</span> <span class="k">return</span> <span class="n">t</span> <span class="c1"># Create a list of turtles from a list of colors</span> <span class="k">def</span> <span class="nf">create_turtles</span><span class="p">(</span><span class="n">colors</span><span class="p">):</span> <span class="n">turtles</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">color</span> <span class="ow">in</span> <span class="n">colors</span><span class="p">:</span> <span class="n">t</span> <span class="o">=</span> <span class="n">create_turtle</span><span class="p">(</span><span class="n">color</span><span class="p">)</span> <span class="n">turtles</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">return</span> <span class="n">turtles</span> <span class="k">def</span> <span class="nf">move_turtle</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">stamp</span><span class="p">()</span> <span class="n">angle</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="o">-</span><span class="mi">90</span><span class="p">,</span> <span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="n">angle</span><span class="p">)</span> <span class="n">distance</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">50</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">distance</span><span class="p">)</span> <span class="n">turtles</span> <span class="o">=</span> <span class="n">create_turtles</span><span class="p">(</span><span class="n">colors</span><span class="p">)</span> <span class="k">for</span> <span class="n">move</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="k">for</span> <span class="n">a_turtle</span> <span class="ow">in</span> <span class="n">turtles</span><span class="p">:</span> <span class="n">move_turtle</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">)</span> </pre></div> <p>If you save and run our program it will generate a screen that looks something like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-stamping-2.png" width="1026" height="790"></figure> <p>You probably noticed that your turtles might have wandered off the screen, sometimes never to return. How can we keep our turtles on the screen so we can see what they’re drawing?</p> <p>We have them make decisions so they know how to turn around if they go off the screen. This is where we use something called <em>conditionals</em> in programming, a way of making a decision based on a condition that is happening in our program.</p> </section><section><h2><a class="anchor" name="conditionals-and-if-statements"></a> Conditionals and “if” Statements</h2> <p>As we briefly talked about in our first class, the way to make programs act smarter is to have them make decisions. To do this we use something called <em>conditionals</em>.</p> <p>Conditionals are just a way for a program to look at something (a condition) and make a decision to do something or something else. For instance, here’s some possible Python conditional program statements:</p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="n">x</span> <span class="o">&lt;</span> <span class="o">-</span><span class="mi">250</span> <span class="ow">or</span> <span class="n">x</span> <span class="o">&gt;</span> <span class="mi">250</span><span class="p">:</span> <span class="n">outside_box</span> <span class="o">=</span> <span class="bp">True</span> </pre></div> <p>Here’s what’s happening in these Python statements:</p> <ol> <li>Use the <code>if</code> statement to test whether the variable <code>x</code> is less than negative 250, or greater than positive 250</li> <li>If <code>x</code> is outside those two values, set the variable <code>outside_box</code> to <em>Boolean</em> <code>True</code></li> </ol> <p>How can we use conditionals to keep our turtles inside a viewable area? First off let’s make our viewable area a box that’s inside our screen so we can see what our turtles do when they go outside that box.</p> <p>In our program we’ll create a variable <code>box_size</code> equal to the size of the box we want to make our viewable area, let’s say 500. We’ll also use one of our turtles to draw this viewable box on the screen so we can see the box edges.</p> <p>Let’s make our program look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="kn">import</span> <span class="nn">random</span> <span class="n">screen</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Screen</span><span class="p">()</span> <span class="n">turtle</span><span class="o">.</span><span class="n">setup</span><span class="p">(</span><span class="mi">1024</span><span class="p">,</span> <span class="mi">768</span><span class="p">)</span> <span class="n">screen</span><span class="o">.</span><span class="n">bgcolor</span><span class="p">(</span><span class="s2">"#FFFFE0"</span><span class="p">)</span> <span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"black"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">,</span> <span class="s2">"orange"</span><span class="p">,</span> <span class="s2">"green"</span><span class="p">]</span> <span class="n">box_size</span> <span class="o">=</span> <span class="mi">500</span> <span class="k">def</span> <span class="nf">create_turtle</span><span class="p">(</span><span class="n">color</span><span class="p">):</span> <span class="n">t</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="n">color</span><span class="p">)</span> <span class="k">return</span> <span class="n">t</span> <span class="k">def</span> <span class="nf">create_turtles</span><span class="p">(</span><span class="n">colors</span><span class="p">):</span> <span class="n">turtles</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">color</span> <span class="ow">in</span> <span class="n">colors</span><span class="p">:</span> <span class="n">t</span> <span class="o">=</span> <span class="n">create_turtle</span><span class="p">(</span><span class="n">color</span><span class="p">)</span> <span class="n">turtles</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">return</span> <span class="n">turtles</span> <span class="k">def</span> <span class="nf">move_turtle</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">stamp</span><span class="p">()</span> <span class="n">angle</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="o">-</span><span class="mi">90</span><span class="p">,</span> <span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="n">angle</span><span class="p">)</span> <span class="n">distance</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">50</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">distance</span><span class="p">)</span> <span class="n">turtles</span> <span class="o">=</span> <span class="n">create_turtles</span><span class="p">(</span><span class="n">colors</span><span class="p">)</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">turtles</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="n">box_size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">,</span> <span class="n">box_size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="k">for</span> <span class="n">side</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">4</span><span class="p">):</span> <span class="n">t1</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">box_size</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="k">for</span> <span class="n">move</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="k">for</span> <span class="n">a_turtle</span> <span class="ow">in</span> <span class="n">turtles</span><span class="p">:</span> <span class="n">move_turtle</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">)</span> </pre></div> <p>Right under where we create our <code>colors</code> list we’ve created the <code>box_size</code> variable and set it equal to 500. Further down under where we created our <code>turtles</code> list variable, we’ve used the first turtle from the list, <code>t1 = turtles[0]</code>, to draw our viewable boundary box. After we’re done drawing the box the turtle is moved back to it’s starting position.</p> <p>So how do we use a conditional to keep our turtles inside the box we’ve just drawn? First things first, we need to know where the turtle is in order to figure out if it’s outside the boundary box. To do this we need another turtle function.</p> <p><strong>Where’s My Turtle: xcor() and ycor()</strong></p> <p>A turtle has two functions telling us where it is in relation to the home position, (0, 0). Those functions are called <code>xcor()</code> and <code>ycor()</code>, which are short for x coordinate and y coordinate. They are used like this:</p> <div class="codehilite"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="n">t</span><span class="o">.</span><span class="n">xcor</span><span class="p">()</span> <span class="n">y</span> <span class="o">=</span> <span class="n">t</span><span class="o">.</span><span class="n">ycor</span><span class="p">()</span> </pre></div> <p>As you might have guessed, the <code>t.xcor()</code> function returns the current x coordinate of the turtle <code>t</code>, and <code>t.ycor()</code> returns the current y coordinate of the turtle.</p> <p>Now we have enough information to decide if a turtle is inside or outside our boundary box. We know where the edges of the boundary box are in relation to where we started drawing it, plus and minus 250 pixels in relation to the starting position of the turtles, (0, 0). We also can figure out where our turtles are any time we want, which we can compare to the boundary box edges.</p> <p>Let’s create a function that returns <code>True</code> if the turtle is outside the box and False otherwise. The function will need the turtle to test and information about the box. That function looks like this:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">is_turtle_outside_box</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="n">size</span><span class="p">):</span> <span class="n">outside_box</span> <span class="o">=</span> <span class="bp">False</span> <span class="n">x</span> <span class="o">=</span> <span class="n">t</span><span class="o">.</span><span class="n">xcor</span><span class="p">()</span> <span class="n">y</span> <span class="o">=</span> <span class="n">t</span><span class="o">.</span><span class="n">ycor</span><span class="p">()</span> <span class="k">if</span> <span class="n">x</span> <span class="o">&lt;</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">)</span> <span class="ow">or</span> <span class="n">x</span> <span class="o">&gt;</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">):</span> <span class="n">outside_box</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">if</span> <span class="n">y</span> <span class="o">&lt;</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="o">-</span><span class="mi">2</span><span class="p">)</span> <span class="ow">or</span> <span class="n">y</span> <span class="o">&gt;</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">):</span> <span class="n">outside_box</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">return</span> <span class="n">outside_box</span> </pre></div> <p>This function expects a turtle to be passed as the first parameter and a number for the size of the boundary box as the second parameter. It then sets the return variable <code>outside_box</code> initially to False. It then creates the <code>x</code> and <code>y</code> variables, setting them to the x and y coordinates of the passed in turtle <code>t</code> respectively. Then using an <code>if</code> statement it compares the <code>x</code> and <code>y</code> variables to the <code>size</code> divided by 2.</p> <p>Why is the <code>size</code> divided by 2? Because my intention is to pass the <code>box_size</code> variable to this function, and the boundary box is centered on the screen, with half (250 pixels) on each side of that.</p> <p>Now that we have this function, how can we use it? Inside our inner most loop we move our turtle, at which point it might be outside the boundary box, so this seems like a good place to use our <code>is_turtle_outside_box()</code> function. Here’s just the looping portion of our current program showing the inclusion of the new function:</p> <div class="codehilite"><pre><span></span><span class="k">for</span> <span class="n">move</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="k">for</span> <span class="n">a_turtle</span> <span class="ow">in</span> <span class="n">turtles</span><span class="p">:</span> <span class="n">move_turtle</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">)</span> <span class="k">if</span> <span class="n">is_turtle_outside_box</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">,</span> <span class="n">box_size</span><span class="p">)</span> <span class="o">==</span> <span class="bp">True</span><span class="p">:</span> <span class="n">a_turtle</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">180</span><span class="p">)</span> <span class="n">a_turtle</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> </pre></div> <p>What we’ve done is after our <code>move_turtle()</code> function call, we added an <code>if</code> statement using our <code>is_turtle_outside_box()</code> function to figure out if our turtle <code>t</code> is outside the boundary box. If the return value of <code>is_turtle_outside_box()</code> is True, we turn our turtle <code>t</code> around 180 degrees from where it’s currently facing and move it 100 pixels back inside the boundary box. Then the loop moves onto the next turtle and the next move for all turtles.</p> <p>Here’s our completed program with comments:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="kn">import</span> <span class="nn">random</span> <span class="c1"># Change the color of the background</span> <span class="n">screen</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Screen</span><span class="p">()</span> <span class="n">screen</span><span class="o">.</span><span class="n">bgcolor</span><span class="p">(</span><span class="s2">"#FFFFE0"</span><span class="p">)</span> <span class="c1"># The number of turtles to create and what color to create them with</span> <span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"black"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">,</span> <span class="s2">"orange"</span><span class="p">,</span> <span class="s2">"green"</span><span class="p">]</span> <span class="c1"># Size of our box</span> <span class="n">box_size</span> <span class="o">=</span> <span class="mi">500</span> <span class="c1"># Create a new turtle with a certain color</span> <span class="k">def</span> <span class="nf">create_turtle</span><span class="p">(</span><span class="n">color</span><span class="p">):</span> <span class="n">t</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="n">color</span><span class="p">)</span> <span class="k">return</span> <span class="n">t</span> <span class="c1"># Create a list of turtles from a list of colors</span> <span class="k">def</span> <span class="nf">create_turtles</span><span class="p">(</span><span class="n">colors</span><span class="p">):</span> <span class="n">turtles</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">color</span> <span class="ow">in</span> <span class="n">colors</span><span class="p">:</span> <span class="n">t</span> <span class="o">=</span> <span class="n">create_turtle</span><span class="p">(</span><span class="n">color</span><span class="p">)</span> <span class="n">turtles</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">return</span> <span class="n">turtles</span> <span class="c1"># Stamp and move the turtle</span> <span class="k">def</span> <span class="nf">move_turtle</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">stamp</span><span class="p">()</span> <span class="n">angle</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="o">-</span><span class="mi">90</span><span class="p">,</span> <span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="n">angle</span><span class="p">)</span> <span class="n">distance</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">50</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">distance</span><span class="p">)</span> <span class="c1"># Is the turtle outside the box?</span> <span class="k">def</span> <span class="nf">is_turtle_outside_box</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="n">size</span><span class="p">):</span> <span class="n">outside_box</span> <span class="o">=</span> <span class="bp">False</span> <span class="n">x</span> <span class="o">=</span> <span class="n">t</span><span class="o">.</span><span class="n">xcor</span><span class="p">()</span> <span class="n">y</span> <span class="o">=</span> <span class="n">t</span><span class="o">.</span><span class="n">ycor</span><span class="p">()</span> <span class="k">if</span> <span class="n">x</span> <span class="o">&lt;</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="o">-</span><span class="mi">2</span><span class="p">)</span> <span class="ow">or</span> <span class="n">x</span> <span class="o">&gt;</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">):</span> <span class="n">outside_box</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">if</span> <span class="n">y</span> <span class="o">&lt;</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="o">-</span><span class="mi">2</span><span class="p">)</span> <span class="ow">or</span> <span class="n">y</span> <span class="o">&gt;</span> <span class="p">(</span><span class="n">size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">):</span> <span class="n">outside_box</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">return</span> <span class="n">outside_box</span> <span class="c1"># Create our list of turtles</span> <span class="n">turtles</span> <span class="o">=</span> <span class="n">create_turtles</span><span class="p">(</span><span class="n">colors</span><span class="p">)</span> <span class="c1"># Use the first turtle to draw our boundary box</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">turtles</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="n">box_size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">,</span> <span class="n">box_size</span> <span class="o">/</span> <span class="mi">2</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="k">for</span> <span class="n">side</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">4</span><span class="p">):</span> <span class="n">t1</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">box_size</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="c1"># Move all the turtles a hundred times</span> <span class="k">for</span> <span class="n">move</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="c1"># Move a particular turtle from our list of turtles</span> <span class="k">for</span> <span class="n">a_turtle</span> <span class="ow">in</span> <span class="n">turtles</span><span class="p">:</span> <span class="n">move_turtle</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">)</span> <span class="c1"># Is the turtle outside the boundary box?</span> <span class="k">if</span> <span class="n">is_turtle_outside_box</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">,</span> <span class="n">box_size</span><span class="p">)</span> <span class="o">==</span> <span class="bp">True</span><span class="p">:</span> <span class="c1"># Turn the turtle around and move it back</span> <span class="n">a_turtle</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">180</span><span class="p">)</span> <span class="n">a_turtle</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> </pre></div> <p>When we run our program the screen should look something like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-random-turtles.png" width="1280" height="1102"></figure> </section><section><h2><a class="anchor" name="conclusion"></a> Conclusion</h2> <p>You’re all getting to be real Python programmers now! You’ve created a program that draws with turtles, and makes decisions based on where those turtles are, very, very cool!</p> <p><a href="python-intro-reacting-to-user-input">In the fourth (and final) class in this series</a> you’ll learn how to make your Python programs interactive by letting them react to user input:</p> <ul> <li><a href="python-intro-reacting-to-user-input">Let’s Program with Python: Reacting to User Input (Part 4)</a></li> </ul></section><footer></footer></article>https://dbader.org/blog/python-intro-conditionals-and-if-statementsThu, 04 May 2017 00:00:00 GMTQueues in Pythonhttps://dbader.org/blog/queues-in-python<article><header><h1>Queues in Python</h1> <p>How to implement a FIFO queue data structure in Python using only built-in data types and classes from the standard library.</p> </header><section><figure><img alt="" src="/blog/figures/queues-in-python.png" width="1280" height="720"></figure> <p>A queue is a collection of objects that supports fast <em>first-in, first-out (FIFO)</em> semantics for inserts and deletes. The insert and delete operations sometimes called <em>enqueue</em> and <em>dequeue</em>. Unlike lists or arrays, queues typically don’t allow for random access to the objects they contain.</p> <p>Here’s a real-world analogy for a first-in, first-out queue:</p> <blockquote> <p>Imagine a line of Pythonistas waiting to pick up their conference badges on day one of PyCon registration. New additions to the line are made to the back of the queue as new people enter the conference venue and “queue up” to receive their badges. Removal (serving) happens in the front of the queue, as developers receive their badges and conference swag bags and leave the queue.</p> </blockquote> <p>Another way to memorize the characteristics of a queue data structure is to think of it as a <em>pipe</em>:</p> <blockquote> <p>New items (water molecules, ping-pong balls, …) are put in at one end and travel to the other where you or someone else removes them again. While the items are in the queue, a solid metal pipe, you can’t get at them. The only way to interact with the items in the queue is to add new items at the back (<em>enqueue</em>) or to remove items at the front (<em>dequeue</em>) of the pipe.</p> </blockquote> <p>Queues are similar to stacks and the difference between them is in removing items:</p> <p>With a <strong>queue</strong> you remove the item <strong>least recently added</strong> (<em>first-in, first-out</em> or <em>FIFO</em>); and with a <strong>stack</strong> you remove the item <strong>most recently added</strong> (<em>last-in, first-out</em> or <em>LIFO</em>).</p> <p>Performance-wise, a proper queue implementation is expected to take <em>O(1)</em> time for insert and delete operations. These are the two main operations performed on a queue and they should be fast in a correct implementation.</p> <p>Queues have a wide range of applications in algorithms and to solve scheduling, as well as parallel programming problems. A short and beautiful algorithm using a queue is <a href="https://en.wikipedia.org/wiki/Breadth-first_search" target="_blank">breadth-first search (BFS)</a> on a tree or graph data structure.</p> <p>Scheduling algorithms often use <a href="priority-queues-in-python">priority queues</a> internally. These are specialized queues: instead of retrieving the next element by insertion time, a priority queue retrieves the highest-priority element. The priority of individual elements is decided by the queue based on the ordering applied to their keys.</p> <p>A regular queue, however, won’t re-order the items it carries. You get what you put in, and in exactly that order (remember the pipe example?)</p> <p>Python ships with several queue implementations that each have slightly different characteristics. Let’s take a look at them:</p> </section><section><h2>⛔ The <a href="https://docs.python.org/3/tutorial/datastructures.html#using-lists-as-queues" target="_blank">list</a> Built-in</h2> <p>It’s possible to use a regular <code>list</code> as a queue but this is <strong>not ideal from a performance perspective</strong>. Lists are quite slow for this purpose because inserting or deleting an element at the beginning requires shifting all of the other elements by one, requiring <em>O(n)</em> time.</p> <p>Therefore I would <strong>not recommend</strong> you use a <code>list</code> as a makeshift queue in Python (unless you’re dealing with a small number of elements only).</p> <div class="codehilite"><pre><span></span><span class="c1"># How to use Python's list as a FIFO queue:</span> <span class="n">q</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'eat'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'sleep'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'code'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span> <span class="p">[</span><span class="s1">'eat'</span><span class="p">,</span> <span class="s1">'sleep'</span><span class="p">,</span> <span class="s1">'code'</span><span class="p">]</span> <span class="c1"># Careful: This is slow!</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="s1">'eat'</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/collections.html#collections.deque" target="_blank">collections.deque</a> Class</h2> <p>The <code>deque</code> class implements a double-ended queue that supports adding and removing elements from either end in <em>O(1)</em> time.</p> <p>Python’s <a href="https://github.com/python/cpython/blob/947629916a5ecb1f6f6792e9b9234e084c5bf274/Modules/_collectionsmodule.c#L24-L26" target="_blank">deque objects are implemented as doubly-linked lists</a> which gives them excellent performance for enqueuing and dequeuing elements, but poor <em>O(n)</em> performance for randomly accessing elements in the middle of the queue.</p> <p>Because deques support adding and removing elements from either end equally well, they can serve both as queues and as stacks.</p> <p><strong><code>collections.deque</code> is a great default choice if you’re looking for a queue data structure in Python’s standard library.</strong></p> <div class="codehilite"><pre><span></span><span class="c1"># How to use collections.deque as a FIFO queue:</span> <span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">deque</span> <span class="n">q</span> <span class="o">=</span> <span class="n">deque</span><span class="p">()</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'eat'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'sleep'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'code'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span> <span class="n">deque</span><span class="p">([</span><span class="s1">'eat'</span><span class="p">,</span> <span class="s1">'sleep'</span><span class="p">,</span> <span class="s1">'code'</span><span class="p">])</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">popleft</span><span class="p">()</span> <span class="s1">'eat'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">popleft</span><span class="p">()</span> <span class="s1">'sleep'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">popleft</span><span class="p">()</span> <span class="s1">'code'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">popleft</span><span class="p">()</span> <span class="ne">IndexError</span><span class="p">:</span> <span class="s2">"pop from an empty deque"</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/queue.html" target="_blank">queue.Queue</a> Class</h2> <p>This queue implementation in the Python standard library is synchronized and provides locking semantics to support multiple concurrent producers and consumers.</p> <p>The <a href="https://docs.python.org/3/library/queue.html" target="_blank"><code>queue</code> module</a> contains several other classes implementing multi-producer, multi-consumer queues that are useful for parallel computing.</p> <p>Depending on your use case the locking semantics might be helpful, or just incur unneeded overhead. In this case you’d be better off with using <code>collections.deque</code> as a general purpose queue.</p> <div class="codehilite"><pre><span></span><span class="c1"># How to use queue.Queue as a FIFO queue:</span> <span class="kn">from</span> <span class="nn">queue</span> <span class="kn">import</span> <span class="n">Queue</span> <span class="n">q</span> <span class="o">=</span> <span class="n">Queue</span><span class="p">()</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'eat'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'sleep'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'code'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span> <span class="o">&lt;</span><span class="n">queue</span><span class="o">.</span><span class="n">Queue</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x1070f5b38</span><span class="o">&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'eat'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'sleep'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'code'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get_nowait</span><span class="p">()</span> <span class="n">queue</span><span class="o">.</span><span class="n">Empty</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="c1"># Blocks / waits forever...</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue" target="_blank">multiprocessing.Queue</a> Class</h2> <p>This is a shared job queue implementation that allows queued items to be processed in parallel by multiple concurrent workers. Process-based parallelization is popular in Python due to the <a href="https://wiki.python.org/moin/GlobalInterpreterLock" target="_blank">global interpreter lock (GIL)</a>.</p> <p><code>multiprocessing.Queue</code> is meant for sharing data between processes and can store any pickle-able object.</p> <div class="codehilite"><pre><span></span><span class="c1"># How to use multiprocessing.Queue as a FIFO queue:</span> <span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="kn">import</span> <span class="n">Queue</span> <span class="n">q</span> <span class="o">=</span> <span class="n">Queue</span><span class="p">()</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'eat'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'sleep'</span><span class="p">)</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'code'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span> <span class="o">&lt;</span><span class="n">multiprocessing</span><span class="o">.</span><span class="n">queues</span><span class="o">.</span><span class="n">Queue</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x1081c12b0</span><span class="o">&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'eat'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'sleep'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="s1">'code'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="c1"># Blocks / waits forever...</span> </pre></div> </section><section><h2>A good default choice: <code>collections.deque</code></h2> <p>If you’re not looking for parallel processing support the implementation offered by <code>collections.deque</code> is an excellent default choice for implementing a FIFO queue data structure in Python.</p> <p>I’d provides the performance characteristics you’d expect from a good queue implementation and can also be used as a stack (LIFO Queue).</p> <p><em><a href="fundamental-data-structures-in-python">Read the full “Fundamental Data Structures in Python” article series here.</a> This article is missing something or you found an error? Help a brother out and leave a comment below.</em></p></section><footer></footer></article>https://dbader.org/blog/queues-in-pythonTue, 02 May 2017 00:00:00 GMTLet’s Program with Python: Functions and Lists (Part 2)https://dbader.org/blog/python-intro-functions-and-lists<article><header><h1>Let’s Program with Python: Functions and Lists (Part 2)</h1> <p>In part two of this four-part Python introduction you’ll see how to write reusable “code building blocks” in your Python programs with functions.</p> </header><section><figure><img alt="" src="/blog/figures/pywelcome-part2.png" width="1280" height="720"></figure> <p><em>In this guest post series by <a href="#author">Doug Farrell</a> you’ll learn the basics of programming with Python from scratch. If you’ve never programmed before or need a fun little class to work through with your kids, you’re welcome to follow along.</em></p> <ul> <li><a href="python-intro-statements-variables-and-loops">Part 1: Statements, Variables, and Loops</a></li> <li>Part 2: Functions and Lists (This article)</li> <li><a href="python-intro-conditionals-and-if-statements">Part 3: Conditionals and “if” Statements</a></li> <li><a href="python-intro-reacting-to-user-input">Part 4: Reacting to User Input</a></li> </ul> </section><section><h2>Table of Contents – Part 2</h2> <ul> <li><a href="#programmers-are-lazy">Programmers Are Lazy</a></li> <li><a href="#introduction-to-functions">Introduction to Functions</a></li> <li><a href="#new-turtle-drawing-functions">New Turtle Drawing Functions</a></li> <li><a href="#drawing-with-multiple-turtles">Drawing With Multiple Turtles</a></li> <li><a href="#grouping-things-with-lists">Grouping Things With Lists</a></li> <li><a href="#conclusion">Conclusion</a></li> </ul> </section><section><h2><a class="anchor" name="programmers-are-lazy"></a>Programmers Are Lazy</h2> <p>We mentioned this in the last class, but if you’re going to be a programmer, you have to embrace basic laziness. Programmers don’t like to repeat themselves and always look for ways to write less code rather than more to get the same things done.</p> <p><a href="python-intro-statements-variables-and-loops">In our last class</a> we saw how using a for loop could reduce the amount of code we had to write to draw a flower. We used a loop to repeat drawing the “petals” of our flower so we didn’t have to write code for every one.</p> <p>Let’s learn about another tool we can put in our programmers toolbelt called functions.</p> </section><section><h2><a class="anchor" name="introduction-to-functions"></a>Introduction to Functions</h2> <p>Functions allow us to use the same set of Python statements over and over again, and even change what the Python code does without having to change the code. We’ve already used functions in the previous session in our turtle program. We used the <code>range()</code> function as part of a <code>for</code> loop.</p> <p>The <code>range()</code> function is built into Python, but what does it do?</p> <p>It generates a range of numbers we can use inside a <code>for</code> loop, as simple as that. Let’s start <strong>Idle</strong>, get into interactive mode and enter this at the Python command prompt:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> </pre></div> <p>The <code>range(10)</code> function created something that will generate a count from <code>0</code> to <code>9</code> (that’s 10 numbers in total). Notice we told the <code>range()</code> function how big the range we wanted was by passing <code>10</code> as the parameter of the function.</p> <p>Using this in a <code>for</code> loop shows the values generated by <code>range(10)</code>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="mi">0</span> <span class="mi">1</span> <span class="mi">2</span> <span class="mi">3</span> <span class="mi">4</span> <span class="mi">5</span> <span class="mi">6</span> <span class="mi">7</span> <span class="mi">8</span> <span class="mi">9</span> </pre></div> <p>What we’ve done is:</p> <ul> <li>Create a for loop that’s going to assign the range of values generated one at a time to the variable <code>x</code>.</li> <li>Then inside the loop we’re just printing the latest value of <code>x</code>.</li> </ul> <p>You’ll notice that value of <code>x</code> goes from <code>0</code> to <code>9</code>, not <code>10</code> as you might expect. There are still ten values, but because Python is zero based (starts things at zero, unless told otherwise), the <code>range(10)</code> function goes from <code>0</code> → <code>9</code>.</p> <p>In our flower drawing turtle program we called <code>range()</code> like this:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">)</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">36</span><span class="p">)</span> </pre></div> <p>This generated a range of 36 values, from <code>0</code> to <code>35</code>. These two examples demonstrate we are changing what the <code>range()</code> function does based on the value we give to it.</p> <p>The value we give to the <code>range()</code> function is called a <strong>parameter</strong>, and the value of that parameter is used to change what the <code>range()</code> function does. In the examples above the parameter tells the <code>range()</code> function how many numbers to generate and gives back to our program a way to use them.</p> <p>We’ve also used functions when we were working with our turtle. For example when I changed the color of my turtle <code>t</code>, with the <code>color()</code> function, like this:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> </pre></div> <p>I was calling the <code>color()</code> function of the turtle variable <code>t</code>, and passed it two parameters, <code>"yellow"</code> and <code>"red"</code>:</p> <ul> <li>The <code>"yellow"</code> parameter changed the color of the <code>t</code> turtle and the color it draws with.</li> <li>The <code>"red"</code> parameter changed the color the turtle used when filling a shape.</li> </ul> <p><strong>Flower Drawing Using Functions</strong></p> <p>Okay, so it’s great Python provides a bunch of functions we can use to do different things, how do functions help me be lazy?</p> <p>Well, Python also lets us create our own functions and use them just like we would any built in function.</p> <p>In <strong>Idle</strong> let’s open our turtle program code from last class and try something out. Modify your program to look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="k">def</span> <span class="nf">draw_box</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="k">for</span> <span class="n">petal</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">):</span> <span class="n">draw_box</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>Save and run our program and it should create our flower exactly as it did before. You’re probably thinking “what’s the big deal, it did exactly the same thing”, and you’d be right!</p> <p>Notice I renamed our turtle variable from <code>t</code> to <code>t1</code>. Why did I do this?</p> <p>I’m getting ready to draw with two turtles at the same time (coming soon to a lesson near you!). Notice also the function I’ve defined, <code>draw_box</code>, has a <code>t</code> in between the parenthesis. Even though my turtle variable is defined as <code>t1</code>, I’m using a variable called <code>t</code> inside the <code>draw_box</code> function.</p> <p>The <code>draw_box</code> function is <strong>defined</strong> by beginning the program line with the Python keyword <code>def</code>, followed by any word we’d like, parenthesis and finally a colon character ‘<code>:</code>’.</p> <p>Just like the <code>range(36)</code> function, where I pass it a value of 36 so it generates 36 numbers, here I’m passing a parameter I’m calling <code>t</code>, and it’s using it to draw with.</p> <p>Inside my <code>for</code> loop notice I’m calling <code>draw_box</code> with my newly renamed <code>t1</code> variable. This is because the variable name passed to a function as a parameter has nothing to do with the variable name inside the function when it’s defined.</p> <p>Notice also that all the drawing code in the <code>draw_box</code> function is indented. Just like the <code>for</code> loop this indicates these Python statements are part of the function definition for <code>draw_box()</code>.</p> <p>When our program runs the <code>for</code> loop calls our <code>draw_box</code> function 36 times, and each time it turns our turtle (<code>t1</code>) 10 degrees to the right.</p> </section><section><h2><a class="anchor" name="new-turtle-drawing-functions"></a> New Turtle Drawing Functions</h2> <p>We’re getting ready to draw multiple flowers with multiple turtles. To do that and have them look good on the screen we’ll learn some more turtle drawing functions.</p> <p><strong>Turtle Pen Up: penup()</strong></p> <p>We can move our turtle without drawing a line by lifting our pen up. In this way we can move the turtle and no line will be drawn. To do this we use the turtle <code>penup()</code> function. It looks like this:</p> <div class="codehilite"><pre><span></span><span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> </pre></div> <p><strong>Turtle Pen Down: pendown()</strong></p> <p>Once we’ve moved our turtle where we want it to be without drawing a line, we need to put the pen down again, and the turtle system provides this. We use the <code>pendown()</code> function. It looks like this:</p> <div class="codehilite"><pre><span></span><span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> </pre></div> <p><strong>Turtle Goto: goto()</strong></p> <p>We can move our turtle to a specific position on the screen using the <code>goto()</code> funciton. We pass x and y coordinates to the <code>goto()</code> function to position our turtle. One thing to be aware of is the 0, 0 coordinates are where our turtle is created (center of the screen) when we did this <code>t1 = turtle.Turtle()</code>.</p> <p>So the coordinates we pass to <code>goto()</code> are relative to that starting position. The <code>goto()</code> function looks like this to move our turtle up and to the right:</p> <div class="codehilite"><pre><span></span><span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="mi">150</span><span class="p">,</span> <span class="mi">150</span><span class="p">)</span> </pre></div> <p>Let’s update our program and move our <code>t1</code> turtle up and to the right a bit just to see how these new drawing functions work. Make your flower program look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="mi">150</span><span class="p">,</span> <span class="mi">150</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="k">def</span> <span class="nf">draw_box</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="k">for</span> <span class="n">petal</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">):</span> <span class="n">draw_box</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>Save and run your program and you should see your flower, but its offset up and to the right side of the screen by 150 pixels. Those are the offsets we passed as the first and second parameter to the <code>t1.goto(150, 150)</code> function call.</p> </section><section><h2><a class="anchor" name="drawing-with-multiple-turtles"></a>Drawing With Multiple Turtles</h2> <p>We want to draw with multiple turtles, and our goal for this class is to create this image:</p> <figure><img alt="image" src="/blog/figures/pywelcome-three-turtles.png" width="988" height="952"></figure> <p>So far our flower drawing program is working pretty well, but can we change it even more to draw two, or perhaps more, flowers at once?</p> <p>Sure we can, we’re programmers! In order to use two turtles we’ll have to create a second turtle. I’m going to call the second turtle <code>t2</code> just to stay consistent. Add this to your program right below where we created our first turtle <code>t1</code>:</p> <div class="codehilite"><pre><span></span><span class="n">t2</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"blue"</span><span class="p">,</span> <span class="s2">"orange"</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> </pre></div> <p>This creates a second turtle with a different variable name, drawing color and fill color. When we create a turtle it’s starting position is right in the center of the screen, so our second turtle starts out right in the middle of the screen.</p> <p>Let’s move it left and down so <code>t1</code> and <code>t2</code> don’t draw on top of each other. Add these lines for turtle <code>t2</code> under the same lines for <code>t1</code>:</p> <div class="codehilite"><pre><span></span><span class="n">t2</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">penup</span><span class="p">(</span><span class="o">-</span><span class="mi">150</span><span class="p">,</span> <span class="o">-</span><span class="mi">150</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> </pre></div> <p><strong>Houston We Have a Problem</strong></p> <p>At this point our program should look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> <span class="n">t2</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"blue"</span><span class="p">,</span> <span class="s2">"orange"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="mi">150</span><span class="p">,</span> <span class="mi">150</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="o">-</span><span class="mi">150</span><span class="p">,</span> <span class="o">-</span><span class="mi">150</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="k">def</span> <span class="nf">draw_box</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="k">for</span> <span class="n">petal</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">):</span> <span class="n">draw_box</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>If save our program and run it our turtle screen looks like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-two-turtles-1.png" width="988" height="952"></figure> <p><strong>Where’s The Second Flower?</strong></p> <p>When you get your program running you’ll notice the second turtle didn’t draw a flower. Why not? Well, we didn’t tell it to draw anything, so it just waited around while the first turtle drew a flower.</p> <p>How do we get it to draw it’s own flower? We add it to the <code>for</code> loop. Our updated program now looks like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> <span class="n">t2</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"blue"</span><span class="p">,</span> <span class="s2">"orange"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="mi">150</span><span class="p">,</span> <span class="mi">150</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="o">-</span><span class="mi">150</span><span class="p">,</span> <span class="o">-</span><span class="mi">150</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="k">def</span> <span class="nf">draw_box</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="k">for</span> <span class="n">petal</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">):</span> <span class="n">draw_box</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="n">draw_box</span><span class="p">(</span><span class="n">t2</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>Just by adding two lines we got our second turtle <code>t2</code> to draw its own complete flower. This is a definite win for laziness. All we had to do was add a couple Python statements to draw a complete second flower!</p> <p>By setting things up and using a function we are able to build more and more interesting programs. In fact we could keep going and add more and more turtles to fill the screen with flowers and all we’d have to do is create more turtles and add statements to our <code>for</code> loop.</p> <p>But this is starting to look like when we were adding flower petals to start with. Can we be even lazier and organize things differently to handle multiple turtles better? Yes of course, we can use something Python calls <em>lists</em>.</p> </section><section><h2><a class="anchor" name="grouping-things-with-lists"></a> Grouping Things With Lists</h2> <p>Lists are a way of grouping things together so we can work with them all at once. They’re a handy way of keeping things grouped together and giving that group a name. There’s nothing magical about this, we can create lists easily with Python. If we enter these statements in the interactive window:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">my_list</span> <span class="o">=</span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">0</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="n">my_list</span><span class="p">)</span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">0</span><span class="p">]</span> </pre></div> <p>We created a variable we called <code>my_list</code> containing the list <code>[4, 2, 3, 0]</code>.</p> <p>You can see the things in the list don’t have to be in order. Lists are created by surrounding a set of things separated by commas with the <code>[</code> and <code>]</code> characters at either end.</p> <p>We can use a list to organize our turtles. Let’s create a list of turtles like this in our program:</p> <div class="codehilite"><pre><span></span><span class="n">turtles</span> <span class="o">=</span> <span class="p">[</span><span class="n">t1</span><span class="p">,</span> <span class="n">t2</span><span class="p">]</span> </pre></div> <p>This creates a variable called <code>turtles</code> that is a list containing our two turtles. Now we can create a new <code>for</code> loop that gets a turtle from our <code>turtles</code> list one at a time and draws with it. We do this with these Python statements:</p> <div class="codehilite"><pre><span></span><span class="k">for</span> <span class="n">a_turtle</span> <span class="ow">in</span> <span class="n">turtles</span><span class="p">:</span> <span class="n">draw_box</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">)</span> <span class="n">a_turtle</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>We’re using a <code>for</code> loop to get each turtle one at a time from our <code>turtles</code> list, assigning it to the variable <code>a_turtle</code> and calling <code>draw_box(a_turtle)</code> and <code>a_turtle.right(10)</code> with that variable.</p> <p>If we put this inside our main <code>for</code> loop, it will be called for each petal the main <code>for</code> loop wants to draw.</p> <p>We can now add a third turtle easily by creating a new turtle and adding it to the <code>turtles</code> list.</p> <p>Let’s do that in our updated, three turtle program. I’ve added comments to describe what’s going on:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="c1"># Create our t1 turtle</span> <span class="n">t1</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> <span class="c1"># Create our t2 turtle</span> <span class="n">t2</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"blue"</span><span class="p">,</span> <span class="s2">"orange"</span><span class="p">)</span> <span class="c1"># Create our t3 turtle</span> <span class="n">t3</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t3</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t3</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t3</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="n">t3</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"red"</span><span class="p">,</span> <span class="s2">"blue"</span><span class="p">)</span> <span class="c1"># Move t1 to its starting position</span> <span class="n">t1</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t1</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="mi">150</span><span class="p">,</span> <span class="mi">150</span><span class="p">)</span> <span class="n">t1</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="c1"># Move t2 to its starting position</span> <span class="n">t2</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t2</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="o">-</span><span class="mi">150</span><span class="p">,</span> <span class="o">-</span><span class="mi">150</span><span class="p">)</span> <span class="n">t2</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="c1"># Move t3 to its starting position</span> <span class="n">t3</span><span class="o">.</span><span class="n">penup</span><span class="p">()</span> <span class="n">t3</span><span class="o">.</span><span class="n">goto</span><span class="p">(</span><span class="o">-</span><span class="mi">150</span><span class="p">,</span> <span class="mi">150</span><span class="p">)</span> <span class="n">t3</span><span class="o">.</span><span class="n">pendown</span><span class="p">()</span> <span class="c1"># Define our draw_box function</span> <span class="k">def</span> <span class="nf">draw_box</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="c1"># Create our list of turtles</span> <span class="n">turtles</span> <span class="o">=</span> <span class="p">[</span><span class="n">t1</span><span class="p">,</span> <span class="n">t2</span><span class="p">,</span> <span class="n">t3</span><span class="p">]</span> <span class="c1"># Create our for loop for 36 petals of the flower</span> <span class="k">for</span> <span class="n">petal</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">):</span> <span class="c1"># Create our for loop to draw a flower petal with</span> <span class="c1"># each turtle in the turtles list</span> <span class="k">for</span> <span class="n">a_turtle</span> <span class="ow">in</span> <span class="n">turtles</span><span class="p">:</span> <span class="c1"># Draw and rotate each turtle</span> <span class="n">draw_box</span><span class="p">(</span><span class="n">a_turtle</span><span class="p">)</span> <span class="n">a_turtle</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>I created a third turtle called <code>t3</code> and just added <code>t3</code> to the <code>turtles</code> list. Notice that our main <code>for</code> loop didn’t change, as far as it’s concerned, it’s just looping 36 times.</p> <p>The inner <code>for</code> loop is responsible for calling the <code>draw_box()</code> function with each turtle variable, and then turning that turtle right 10 degrees. Here’s what the output of the program looks like:</p> <figure><img alt="image" src="/blog/figures/pywelcome-three-turtles.png" width="988" height="952"></figure> </section><section><h2><a class="anchor" name="conclusion"></a> Conclusion</h2> <p>Congratulations, you’re a multi-turtle genius now! You saw how to use Python lists to help us get multiple turtles drawing on the screen. We could keep adding turtles to our hearts content and the program would faithfully make each turtle draw our flower. This worked very well drawing the well controlled structure of the flower.</p> <p>But what if we want to draw something that’s randomly generated, something where the turtles draw something and we don’t know ahead of time what that will be? How can we use what we know already to help us do that?</p> <p><a href="python-intro-conditionals-and-if-statements">In the next class in this series</a> you’ll teach our program how to make decisions and do things on its own:</p> <ul> <li><a href="python-intro-conditionals-and-if-statements">Let’s Program with Python: Conditionals and “if” Statements (Part 3)</a></li> </ul></section><footer></footer></article>https://dbader.org/blog/python-intro-functions-and-listsThu, 27 Apr 2017 00:00:00 GMTSets and Multisets in Pythonhttps://dbader.org/blog/sets-and-multiset-in-python<article><header><h1>Sets and Multisets in Python</h1> <p>How to implement mutable and immutable set and multiset (bag) data structures in Python using built-in data types and classes from the standard library.</p> </header><section><figure><img alt="" src="/blog/figures/python-sets-and-multisets.png" width="1280" height="720"></figure> <p>A <em>set</em> is an unordered collection of objects that does not allow duplicate elements. Typically sets are used to quickly test a value for membership in the set, to insert or delete new values from a set, and to compute the union or intersection of two sets.</p> <p>In a “proper” set implementation, membership tests are expected to run in <em>O(1)</em> time. Union, intersection, difference, and subset operations should take <em>O(n)</em> time on average. The set implementations included in Python’s standard library <a href="https://wiki.python.org/moin/TimeComplexity" target="_blank">follow these performance characteristics</a>.</p> <p>Just like dictionaries, sets get special treatment in Python and have some syntactic sugar that makes it easier to create sets. For example, the curly-braces set expression syntax and set comprehensions allow you to conveniently define new set instances:</p> <div class="codehilite"><pre><span></span><span class="n">vowels</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'a'</span><span class="p">,</span> <span class="s1">'e'</span><span class="p">,</span> <span class="s1">'i'</span><span class="p">,</span> <span class="s1">'o'</span><span class="p">,</span> <span class="s1">'u'</span><span class="p">}</span> <span class="n">squares</span> <span class="o">=</span> <span class="p">{</span><span class="n">x</span> <span class="o">*</span> <span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)}</span> </pre></div> <p>Careful: To create <strong>an empty set</strong> you’ll need to call the <code>set()</code> constructor, as using empty curly-braces (<code>{}</code>) is ambiguous and will create a dictionary instead.</p> <p>Python and its standard library provide the following set implementations:</p> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/stdtypes.html#set-types-set-frozenset" target="_blank">set</a> Built-in</h2> <p>The built-in set implementation in Python. The <code>set</code> type in Python is mutable and allows the dynamic insertion and deletion of elements. Python’s sets are backed by the <code>dict</code> data type and share the same performance characteristics. Any <a href="https://docs.python.org/3/glossary.html#term-hashable" target="_blank">hashable</a> object can be stored in a set.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">vowels</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'a'</span><span class="p">,</span> <span class="s1">'e'</span><span class="p">,</span> <span class="s1">'i'</span><span class="p">,</span> <span class="s1">'o'</span><span class="p">,</span> <span class="s1">'u'</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="s1">'e'</span> <span class="ow">in</span> <span class="n">vowels</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">letters</span> <span class="o">=</span> <span class="nb">set</span><span class="p">(</span><span class="s1">'alice'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">letters</span><span class="o">.</span><span class="n">intersection</span><span class="p">(</span><span class="n">vowels</span><span class="p">)</span> <span class="p">{</span><span class="s1">'a'</span><span class="p">,</span> <span class="s1">'e'</span><span class="p">,</span> <span class="s1">'i'</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">vowels</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s1">'x'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">vowels</span> <span class="p">{</span><span class="s1">'i'</span><span class="p">,</span> <span class="s1">'a'</span><span class="p">,</span> <span class="s1">'u'</span><span class="p">,</span> <span class="s1">'o'</span><span class="p">,</span> <span class="s1">'x'</span><span class="p">,</span> <span class="s1">'e'</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">vowels</span><span class="p">)</span> <span class="mi">6</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/stdtypes.html#set-types-set-frozenset" target="_blank">frozenset</a> Built-in</h2> <p>An <em>immutable</em> version of <code>set</code> that cannot be changed after it was constructed. Frozensets are static and only allow query operations on their elements (no inserts or deletions.) Because frozensets are static and <a href="https://docs.python.org/3/glossary.html#term-hashable" target="_blank">hashable</a> they can be used as dictionary keys or as elements of another set.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">vowels</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">({</span><span class="s1">'a'</span><span class="p">,</span> <span class="s1">'e'</span><span class="p">,</span> <span class="s1">'i'</span><span class="p">,</span> <span class="s1">'o'</span><span class="p">,</span> <span class="s1">'u'</span><span class="p">})</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">vowels</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s1">'p'</span><span class="p">)</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="s2">"'frozenset' object has no attribute 'add'"</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/collections.html#collections.Counter" target="_blank">collections.Counter</a> Class</h2> <p>The <code>collections.Counter</code> class in the Python standard library implements a <a href="https://en.wikipedia.org/wiki/Multiset" target="_blank">multiset (or bag)</a> type that allows elements in the set to have more than one occurrence.</p> <p>This is useful if you need to keep track not only <em>if</em> an element is part of a set but also <em>how many times</em> it is included in the set.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">Counter</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">inventory</span> <span class="o">=</span> <span class="n">Counter</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">loot</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'sword'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">'bread'</span><span class="p">:</span> <span class="mi">3</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">inventory</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">loot</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">inventory</span> <span class="n">Counter</span><span class="p">({</span><span class="s1">'bread'</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span> <span class="s1">'sword'</span><span class="p">:</span> <span class="mi">1</span><span class="p">})</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">more_loot</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'sword'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">'apple'</span><span class="p">:</span> <span class="mi">1</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">inventory</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">more_loot</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">inventory</span> <span class="n">Counter</span><span class="p">({</span><span class="s1">'bread'</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span> <span class="s1">'sword'</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span> <span class="s1">'apple'</span><span class="p">:</span> <span class="mi">1</span><span class="p">})</span> </pre></div> <p>Careful with counting the number of elements in a <code>Counter</code> object. Calling <code>len()</code> returns the number of <em>unique</em> elements in the multiset, whereas the total number of elements must be retrieved slightly differently:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">inventory</span><span class="p">)</span> <span class="mi">3</span> <span class="c1"># Unique elements</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">sum</span><span class="p">(</span><span class="n">inventory</span><span class="o">.</span><span class="n">values</span><span class="p">())</span> <span class="mi">6</span> <span class="c1"># Total no. of elements</span> </pre></div> </section><section><h2>📺🐍 Learn More With This Video Tutorial</h2> <p>I recorded a step-by-step video tutorial to go along with the article. See how sets work in general and how to use them in Python. Watch the video embedded below or <a href="https://www.youtube.com/watch?v=b-K1ujf8u_k" target="_blank">on my YouTube channel</a>:</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/b-K1ujf8u_k?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p><em><a href="fundamental-data-structures-in-python">Read the full “Fundamental Data Structures in Python” article series here.</a> This article is missing something or you found an error? Help a brother out and leave a comment below.</em></p></section><footer></footer></article>https://dbader.org/blog/sets-and-multiset-in-pythonTue, 25 Apr 2017 00:00:00 GMTLet’s Program with Python: Statements, Variables, and Loops (Part 1)https://dbader.org/blog/python-intro-statements-variables-and-loops<article><header><h1>Let’s Program with Python: Statements, Variables, and Loops (Part 1)</h1> <p>In this four-part introduction for new programmers you’ll learn the basics of programming with Python using step-by-step descriptions and graphical examples.</p> </header><section><figure><img alt="" src="/blog/figures/pywelcome-part1.png" width="1280" height="720"></figure> <p><em>In this guest post series by <a href="#author">Doug Farrell</a> you’ll learn the basics of programming with Python from scratch. If you’ve never programmed before or need a fun little class to work through with your kids, you’re welcome to follow along.</em></p> <p><strong>Looking for the rest of the “Let’s Program with Python” series?</strong> Here you go:</p> <ul> <li>Part 1: Statements, Variables, and Loops (This article)</li> <li><a href="python-intro-functions-and-lists">Part 2: Functions and Lists</a></li> <li><a href="python-intro-conditionals-and-if-statements">Part 3: Conditionals and “if” Statements</a></li> <li><a href="python-intro-reacting-to-user-input">Part 4: Reacting to User Input</a></li> </ul> </section><section><h2>Table of Contents – Part 1</h2> <ul> <li><a href="#what-is-python">What is Python?</a></li> <li><a href="#natural-vs-formal-language">Natural Language vs. Formal Language</a></li> <li><a href="#elements-of-programming">Elements of Programming</a></li> <li><a href="#lets-write-some-python">Enough of That, Let’s Write Some Python!</a></li> <li><a href="#statements-in-python">Statements in Python</a></li> <li><a href="#creating-python-program-files">Creating Python Program Files</a></li> <li><a href="#saving-and-running-a-python-program">Saving and Running a Python Program</a></li> <li><a href="#variables-in-python">Variables in Python</a></li> <li><a href="#lets-get-back-to-drawing">Let’s Get Back to Drawing!</a></li> <li><a href="#loops-in-python">Loops in Python</a></li> <li><a href="#conclusion">Conclusion</a></li> </ul> </section><section><h2><a class="anchor" name="what-is-python"></a>What is Python?</h2> <p>Since you’re reading this I’m hoping you’re interested in learning how to program in <a href="https://www.python.org/" target="_blank">Python</a>.</p> <p>Python is a programming language, which means it’a a language both people and computers can understand. A computer language is a formal subset of an natural language, like English. A computer language lets people express what they want a computer to do, and tells a computer how to do it.</p> <p>A computer program is a set of instructions written in a particular computer language. There are lots of different computer languages in the world, most were created to solve certain kinds of problems in different ways, and most over lap in the kinds of things they can do.</p> <p>Python was developed by a Dutch software engineer named <a href="http://en.wikipedia.org/wiki/Guido_van_Rossum" target="_blank">Guido van Rossum</a>, who created the language to solve some problems he saw in computer languages of the time.</p> <p>Python draws from a lot of good ideas in other languages and pulls them together in one place. Python is a pretty easy computer language to learn, and yet is very powerful. The name Python comes from Guido’s favorite comedy group, <a href="http://www.montypython.com/" target="_blank">Monty Python’s Flying Circus</a>.</p> <p>This course uses Python 3.6.1, but the examples should work with any version of Python 3 and greater.</p> </section><section><h2><a class="anchor" name="natural-vs-formal-language"></a>Natural Language vs. Formal Language</h2> <p>English is a natural language that’s evolved over time to help us talk with each other. It has a big vocabulary, lots of multiple meanings and depends a lot on how it’s used to make the meaning clear.</p> <p>Natural languages work well for people because we fill in the gaps where needed. This kind of language fails completely for computers because they need exact instructions in order to run. Formal languages (all programming languages) have limited vocabularies and almost no multiple meanings.</p> <p>Let’s take an English example that’s something like a “program” for a person, how to make scrambled eggs:</p> <div class="codehilite"><pre><span></span>1. Place a frying pan on the stove burner 2. Turn the burner to medium 3. Melt butter in the pan 4. Crack two eggs into pan 5. Stir the eggs to cook and scramble them 6. When finished, serve the eggs on a plate </pre></div> <p>If the steps above are followed in order, someone should be able to make scrambled eggs. This simple set of steps describe how to perform a task. A computer program is very much the same, a set of steps telling a computer how to perform a task.</p> </section><section><h2><a class="anchor" name="elements-of-programming"></a>Elements of Programming</h2> <p>As you learn to program you’ll find you need to do certain things to make the program do what you want: how to make the computer do something, remember things, doing things over and over and make decisions. Almost all programming languages provide ways to do these four basic things, and they’re known as:</p> <ul> <li><strong>Statements</strong>: the things a program can do, like performing calculations, drawing on the screen, etc.</li> <li><strong>Variables</strong>: these are the “things” (information) you want your program to work on and remember</li> <li><strong>Loops</strong>: doing things over and over again very quickly</li> <li><strong>Conditionals</strong>: these are choices a program can make about what to do, this is what makes programs “appear” smart.</li> </ul> <p>We’ll make use of these four things as we go along.</p> </section><section><h2><a class="anchor" name="lets-write-some-python"></a>Enough of That, Let’s Write Some Python!</h2> <p>Our goal is to create a Python program that will draw an image on our computer screen. The image we’re going to create looks something like a flower, and we’re going to learn how to use Python to create it. The end results will look like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-flower.png" width="1074" height="1036"></figure> <p>So how do we create a Python program? There are two ways to work with Python; working with it directly, and creating Python program files.</p> <p>This is where we can use the tool called <strong>Idle</strong>. <strong>Idle</strong> is a program that lets you both work with Python directly and create Python program files.</p> <p>So let’s start <strong>Idle</strong>. When you installed Python you should also have gotten the <strong>Idle</strong> program installed, if so, let’s start it up!</p> <p>Starting <strong>Idle</strong> should give you a window that looks something like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-idle-window.png" width="730" height="459"></figure> <p>This window provides a Python command prompt (hit return a couple of times if you don’t see it) that allows you to run Python statements line by line.</p> <p>This is called interactive mode as it allows us to ‘interact’ with Python. The command prompt in interactive mode looks like this:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> </pre></div> <p>Its at this prompt where you enter Python statements to try things out.</p> </section><section><h2><a class="anchor" name="statements-in-python"></a> Statements in Python</h2> <p>Statements are the program commands that make the computer do something in a Python program. Statements can be as simple or as complicated as we’d like to make them.</p> <p>Here are some examples:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="s2">"Hello there"</span><span class="p">)</span> <span class="n">Hello</span> <span class="n">there</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="mi">12</span><span class="p">)</span> <span class="mi">12</span> <span class="o">&gt;&gt;&gt;</span> <span class="mi">12</span> <span class="o">*</span> <span class="mi">3</span> <span class="mi">36</span> <span class="o">&gt;&gt;&gt;</span> <span class="mi">12</span> <span class="o">/</span> <span class="mi">3</span> <span class="mf">4.0</span> </pre></div> <p>The above statements print out a welcome string, perform some basic math and Python is responding. What we’ve done above is enter some Python statements at our command prompt, and Python ran them.</p> </section><section><h2><a class="anchor" name="creating-python-program-files"></a> Creating Python Program Files</h2> <p>Working with interactive mode, is great for trying things out with Python. However, we want to create a Python program we can run over and over again without having to re-type it every time.</p> <p>This is where creating a Python program and saving it as a file is very handy. Python program files are just like any other text file, but usually have the extension “.py”.</p> <p>We can create a Python program file in <strong>Idle</strong> by clicking the <strong>File → New Window</strong> menu item. This opens up a new, empty window that is a simple text editor.</p> <p>You’ll notice there is no <code>&gt;&gt;&gt;</code> Python command prompt in the window. This is because in the file window we’re not interacting with Python directly, we’re creating a Python program file.</p> <p>Let’s use our new Python program file to create our first Python program.</p> <p><strong>The “Turtle” Graphics Module</strong></p> <p>Python comes with a large library of modules that let us do some interesting things, and one of those modules is called <code>turtle</code>.</p> <p>The turtle module is a nice tool for drawing graphics on the screen. You can find the turtle graphics documentation <a href="https://docs.python.org/3/library/turtle.html#module-turtle" target="_blank">here</a>.</p> <p>The turtle module is based on the idea of a “turtle” on the screen that draws a line as it moves around, as if it had a marker taped to it’s shell.</p> <p>In order to use the turtle module we have to “import” it into our Python program. Importing a module adds the features and capabilities of that module to our Python program.</p> <p>To import the turtle module add this line to our Python program:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> </pre></div> <p><strong>Drawing With Turtle</strong></p> <p>Once the turtle module is available to us we can use it to draw things with a turtle. Enter the following lines into our program:</p> <div class="codehilite"><pre><span></span><span class="n">t</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> </pre></div> </section><section><h2><a class="anchor" name="saving-and-running-a-python-program"></a> Saving and Running a Python Program</h2> <p>Once you’ve got this entered, let’s run the program. To do that we have to save the file first, which we can do from the <strong>File → Save</strong> menu selection.</p> <p>Give our program a name and save it to a directory on the hard disk where you can find it again.</p> <p>To run the program select <strong>Run → Run Module</strong>. If your program runs without any errors (which usually means you have a typo in your program), a window will open up with a turtle shape at the end of a short line.</p> <p>That window should look something like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-turtle-1.png" width="988" height="952"></figure> <p>This is what our program told Python to do, use the <code>turtle</code> module to create a turtle we’re calling <code>t</code>, change it’s shape to look like a ‘turtle’ and move it forward 100 pixels.</p> <p>Our turtle, <code>t</code>, is the first <em>variable</em> we’ve created with Python in our program.</p> </section><section><h2><a class="anchor" name="variables-in-python"></a> Variables in Python</h2> <p>In Python things like our turtle <code>t</code> are represented by variables. Variables let us give a name to something so you and the program can remember it and use it later.</p> <p>For instance, here’s a variable assignment:</p> <div class="codehilite"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="mi">10</span> </pre></div> <p>This looks a lot like math, and that’s actually where the idea of assigning variables came from.</p> <p>This simple Python statement assigns the number 10 to a variable called <code>x</code>. The equal sign (<code>=</code>) in the line above creates the variable <code>x</code> and assigns it a value.</p> <p>In our program we’ve done this by using the turtle module to create a turtle (the Python statement <code>turtle.Turtle()</code>) and assigned the results, a turtle object, to a variable we called <code>t</code>.</p> </section><section><h2><a class="anchor" name="lets-get-back-to-drawing"></a> Let’s Get Back to Drawing!</h2> <p>Let’s add some more statements to our program to make it draw some more. Let’s make our Python program look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="n">t</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="s2">"turtle"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"red"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> </pre></div> <p>When you save and run our program the screen your turtle is drawing on should look like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-turtle-2.png" width="1074" height="1036"></figure> <p>So what’s going on here? What we’ve done is given our turtle a set of commands (Python program statements), and it has run them. Here’s what the statements we entered were doing:</p> <ul> <li><strong>Line 1</strong>: import the turtle module so our program can use it</li> <li><strong>Line 3</strong>: use the turtle module to create our turtle, <code>t</code></li> <li><strong>Line 4</strong>: change the shape of our turtle to look like a turtle</li> <li><strong>Line 5</strong>: from where the turtle is, move forward 100 pixels</li> <li><strong>Line 6</strong>: from where the turtle is, turn right 90 degrees, a right angle</li> <li><strong>Line 7</strong>: from where the turtle is, move forward 100 pixels</li> <li><strong>Line 8</strong>: from where the turtle is, turn right 90 degrees, a right angle</li> <li><strong>Line 9</strong>: from where the turtle is, move forward 100 pixels</li> <li><strong>Line 10</strong>: from where the turtle is, turn right 90 degrees, a right angle</li> <li><strong>Line 11</strong>: change the color used by the turtle to red</li> <li><strong>Line 12</strong>: from where the turtle is, move forward 100 pixels</li> <li><strong>Line 13</strong>: from where the turtle is, turn right 90 degrees, a right angle. This brings our turtle back to its original starting position.</li> </ul> <p>These statements made the turtle draw a box with the last side of the box drawn in red. You can see something interesting about drawing with our turtle; what it draws is based on where it is on the screen and which way it’s headed.</p> <p>Let’s learn some more Python statements to draw with our turtles.</p> <p><strong>Turtle Speed</strong>: To make our turtle draw faster we use the turtle <code>speed()</code> method. To use this we’ll add this statement to our program:</p> <div class="codehilite"><pre><span></span><span class="n">t</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> </pre></div> <p>The number <code>0</code> between the parenthesises is called a parameter, which is being given to the turtle’s <code>speed()</code> method, making our turtle draw as fast as it can.</p> <p><strong>Turtle Line Width</strong>: We can make our turtle draw with a thicker line, making it easier to see on screen. We do this with the turtle <code>width()</code> method. We can pass a parameter to the width method, expressing a value in pixels. So for example adding this line to our program makes our turtle draw with a line 3 pixels wide:</p> <div class="codehilite"><pre><span></span><span class="n">t</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> </pre></div> <p><strong>Filling-in Shapes</strong>: We can also fill a shape (like our box) with color using two other turtle methods, <code>begin_fill()</code> and <code>end_fill()</code>, and by modifying our <code>t.color()</code> method. If we use these Python statements:</p> <div class="codehilite"><pre><span></span><span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="c1"># Draw shape</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> </pre></div> <p>We have told our turtle to draw with “yellow” and fill in any shapes with “red”.</p> <p>Then we use <code>begin_fill()</code> at the start of drawing a closed shape, draw our shape and then used <code>end_fill()</code> to fill that shape with “red”.</p> <p>The following line is a Python <em>comment</em>:</p> <div class="codehilite"><pre><span></span><span class="c1"># Draw shape</span> </pre></div> <p>I’m using the comment here to indicate where our shape drawing code should go. Comments in Python are used to tell us, the programmers, what’s going on, but are ignored by Python.</p> <p><strong>Putting It All Together</strong></p> <p>Taking the things we’ve learned from our box drawing, let’s draw something a little more complicated by drawing a sort of flower.</p> <p>We’ll do this by doing two things; draw multiple boxes, filling the box with color and turning the turtle slightly between each one using the new turtle methods we just learned about.</p> <p>To draw our flower we’re going to draw multiple boxes, turning each box slightly every time. One way to do that would be to just repeat our box code over and over like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="n">t</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="c1"># Draw our first filled in box</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="c1"># Draw our second filled in box</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="c1"># Keep going till you've drawn your flower</span> </pre></div> <p>This would work fine, but we would have to repeat these statements for as many petals as we want to give our flower.</p> <p>One thing you should know about being a programmer is we’re very lazy and don’t like to repeat ourselves if we don’t have to.</p> <p>Can Python help us not repeat ourselves? Yes, it can by letting us use a loop to repeat drawing the box multiple times.</p> </section><section><h2><a class="anchor" name="loops-in-python"></a> Loops in Python</h2> <p>One of the elements of programming mentioned earlier is being able to create loops. Loops are statements in a programming language that allows us to repeat a set of program statements over and over in a controlled way. In our program we’d like to repeat the statements:</p> <div class="codehilite"><pre><span></span><span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>This set of statements creates our outlined and filled in box. We’d like to repeat these statements, with a slight turn of 10 degrees each time, in order to create a flower. Creating a loop lets us do this. One of the looping statements in Python is called a “for loop”, and it’s used to create a loop that repeats a fixed number of times.</p> <p>Let’s do a little math to figure out how many times we need to repeat this if we want to make a full circle when our turtle is turning right by 10 degrees after every filled in box.</p> <p>In case you didn’t know already, there are 360 degrees in a full circle. That means dividing a circle of 360 degrees by 10 degrees gives us a value of 36. This means we want to repeat our box 36 times in order to create our flower.</p> <p>How can we do this with Python?</p> <p><strong>The “for” loop</strong>: We want to repeat our filled in box 36 times, so we know beforehand how many times we want to loop. This is where the Python <code>for</code> loop comes in handy.</p> <p>It is made for repeating things a known number of times. It looks like this as a Python definition statement:</p> <div class="codehilite"><pre><span></span><span class="k">for</span> &lt;thing&gt; in &lt;list of things&gt;: <span class="c1"># Everything we want to do on the loop</span> </pre></div> <p>What does that mean? That’s a kind of formal presentation of what a <code>for</code> loop should look like in Python. What it means is this:</p> <ul> <li>Take one <em>&lt;thing&gt;</em> from the <em>&lt;list of things&gt;</em></li> <li>End the for statement with a <code>:</code> character</li> <li>All the statements indented under the <code>for</code> loop should be run every time through the loop. Indentation is very important to Python</li> <li>Go back to the start of the <code>for</code> loop, get another <em>&lt;thing&gt;</em> from <em>&lt;list_of_things&gt;</em></li> <li>Keep doing this till there are no more things in &lt;list_of_things&gt;</li> </ul> <p>This sounds a lot harder than it actually is. For our program we’ll use a <code>for</code> loop that looks like this:</p> <div class="codehilite"><pre><span></span><span class="k">for</span> <span class="n">petal</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>Our box code is in there, under the <code>for</code> loop, but what’s that strange looking <code>range(36)</code> thing at a the end of our <code>for</code> loop where the &lt;list of things&gt; should go?</p> <p>The <code>range(36)</code> thing is what provides the <em>&lt;list_of_things&gt;</em> from our formal definition of a <code>for</code> loop. Let’s talk about that.</p> <p><strong>The “range” function</strong>: Let’s jump back over to our <strong>Idle</strong> interactive window for a minute and enter this statement:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">)</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">36</span><span class="p">)</span> </pre></div> <p>Python responds by running this code and prints out what looks like it just told us about itself. What does this mean?</p> <p>To Python the <code>range(36)</code> function is going to provide 36 things when used inside a for loop. Each time through the <code>for</code> loop Python will take a value from the range defined (0 to 35) and assign it to a variable, in our case that variable is called <code>petal</code>.</p> <p>It will continue this until there are no values left in the range. The way we’ve set things up it will loop 36 times, which is what we want. In this case we’re not using the <code>petal</code> variable, but it’s required by Python to create a correct <code>for</code> loop.</p> <p>To create our flower using a <code>for</code> loop make our program look like this:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">turtle</span> <span class="n">t</span> <span class="o">=</span> <span class="n">turtle</span><span class="o">.</span><span class="n">Turtle</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">speed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">color</span><span class="p">(</span><span class="s2">"yellow"</span><span class="p">,</span> <span class="s2">"red"</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">width</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="k">for</span> <span class="n">petal</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">36</span><span class="p">):</span> <span class="n">t</span><span class="o">.</span><span class="n">begin_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">90</span><span class="p">)</span> <span class="n">t</span><span class="o">.</span><span class="n">end_fill</span><span class="p">()</span> <span class="n">t</span><span class="o">.</span><span class="n">right</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> </pre></div> <p>Let’s go through this line by line.</p> <ul> <li><strong>Line 1</strong>: import our turtle module</li> <li><strong>Line 3</strong>: create our turtle object and use our variable <code>t</code> to keep track of it</li> <li><strong>Line 4</strong>: set our turtle drawing speed to fast</li> <li><strong>Line 5</strong>: tell our turtle to draw in “yellow” and fill in shapes with “red”</li> <li><strong>Line 6</strong>: set our turtle drawing width to 3 pixels</li> <li><strong>Line 8</strong>: begin our <code>for</code> loop and tell it to loop 36 times</li> <li><strong>Line 9-19</strong>: draw our box and then turn slightly right 10 degrees.</li> </ul> <p>Notice how lines 9 through 19 are indented under the <code>for</code> loop. This is important as it tells Python all these lines are part of the <code>for</code> loop. In Python the indentation is required to tell the program a set of statements are part of a block like this.</p> <p>Once you’ve got this entered, save our program and then run it. If your program runs without any syntax errors (which usually means you have a typo in your program), you should get a window like this:</p> <figure><img alt="image" src="/blog/figures/pywelcome-flower.png" width="1074" height="1036"></figure> </section><section><h2><a class="anchor" name="conclusion"></a> Conclusion</h2> <p>Congratulations, you’ve written your first colorful, interesting Python program! In the <a href="python-intro-functions-and-lists">next class in this series</a> you’ll learn how to write reusable “code building blocks” with functions:</p> <ul> <li><a href="python-intro-functions-and-lists">Let’s Program with Python: Functions and Lists (Part 2)</a></li> </ul></section><footer></footer></article>https://dbader.org/blog/python-intro-statements-variables-and-loopsThu, 20 Apr 2017 00:00:00 GMTDictionaries, Maps, and Hash Tables in Pythonhttps://dbader.org/blog/python-dictionaries-maps-and-hashtables<article><header><h1>Dictionaries, Maps, and Hash Tables in Python</h1> <p>Need a dictionary, map, or hash table to implement an algorithm in your Python program? Read on to see how the Python standard library can help you.</p> </header><section><figure><img alt="" src="/blog/figures/dictionaries-maps-and-hashtables-in-python.png" width="1280" height="720"></figure> <p>Dictionaries store an arbitrary number of objects, each identified by a unique dictionary <em>key</em>. Dictionaries are often also called <em>maps</em> or <em>associative arrays</em> and allow the efficient lookup, insertion, and deletion of any object associated with a given key.</p> <p>To give a more practical explanation—phonebooks are a decent real-world analog for dictionaries. They allow you to quickly retrieve the information (phone number) associated with a given key (a person’s name). Instead of having to read a phonebook front to back in order to find someone’s number you can jump more or less directly to a name and look up the associated number.</p> <p>This analogy breaks down somewhat when it comes to how the information is organized to allow for fast lookups. But the fundamental performance characteristics hold: Dictionaries allow you to quickly find the information associated with a given key.</p> <p>The dictionary abstract data type is one of the most frequently used and most important data structures in computer science. Because of this importance Python features a robust dictionary implementation as one of its built-in data types (<a href="https://docs.python.org/3/library/stdtypes.html#mapping-types-dict" target="_blank"><code>dict</code></a>).</p> <p>Python even provides some useful syntactic sugar for working with dictionaries in your programs. For example, the curly-braces dictionary expression syntax (<code>{}</code>) and dictionary comprehensions allow you to conveniently define new dictionaries:</p> <div class="codehilite"><pre><span></span><span class="n">phonebook</span> <span class="o">=</span> <span class="p">{</span> <span class="s1">'bob'</span><span class="p">:</span> <span class="mi">7387</span><span class="p">,</span> <span class="s1">'alice'</span><span class="p">:</span> <span class="mi">3719</span><span class="p">,</span> <span class="s1">'jack'</span><span class="p">:</span> <span class="mi">7052</span><span class="p">,</span> <span class="p">}</span> <span class="n">squares</span> <span class="o">=</span> <span class="p">{</span><span class="n">x</span><span class="p">:</span> <span class="n">x</span> <span class="o">*</span> <span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)}</span> </pre></div> <p>Python’s dictionaries are indexed by keys that can be of any <a href="https://docs.python.org/3/glossary.html#term-hashable" target="_blank">hashable</a> type. A hashable object has a hash value which never changes during its lifetime (see <code>__hash__</code>), and it can be compared to other objects (see <code>__eq__</code>).</p> <p>In addition, hashable objects which compare equal must have the same hash value. Immutable types like strings and numbers work well as dictionary keys. You can also use tuples as dictionary keys as long as they contain only hashable types themselves.</p> </section><section><h2>✅ Built-in <a href="https://docs.python.org/3/library/stdtypes.html#mapping-types-dict" target="_blank">dict</a> type</h2> <p>For most use cases you’ll face Python’s built-in dictionary implementation will do everything you need. Dictionaries are highly optimized and underlie many parts of the language, for example class attributes and variables in a stack frame are both stored internally in dictionaries.</p> <p>Python dictionaries are based on a well-tested and finely tuned hash table implementation that provides the performance characteristics you’d expect: <em>O(1)</em> time complexity for lookup, insert, update, and delete operations in the average case.</p> <p>There’s little reason to not use the standard <code>dict</code> implementation included with Python. However, specialized third-party dictionary data structures exist, for example skip lists or B-tree based dictionary implementations.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">phonebook</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'bob'</span><span class="p">:</span> <span class="mi">7387</span><span class="p">,</span> <span class="s1">'alice'</span><span class="p">:</span> <span class="mi">3719</span><span class="p">,</span> <span class="s1">'jack'</span><span class="p">:</span> <span class="mi">7052</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">phonebook</span><span class="p">[</span><span class="s1">'alice'</span><span class="p">]</span> <span class="mi">3719</span> </pre></div> <p>Interestingly, <strong>Python ships with a number of specialized dictionary implementations in its standard library</strong>. These specialized dictionaries are all based on the built-in dictionary implementation (and share its performance characteristics) but add some convenience features:</p> </section><section><h2>✅ <a href="https://docs.python.org/3/library/collections.html#collections.OrderedDict" target="_blank">collections.OrderedDict</a> – Remember the insertion order of keys</h2> <p>A dictionary subclass that remembers the insertion order of keys added to the collection.</p> <p>While standard <code>dict</code> instances <a href="https://mail.python.org/pipermail/python-dev/2016-September/146327.html" target="_blank">preserve the insertion order of keys in CPython 3.6+</a> this is just a side effect of the CPython implementation and not defined in the language spec. If key order is important for your algorithm to work it’s best to communicate this clearly by using the <code>OrderDict</code> class.</p> <p><code>OrderedDict</code> is not a built-in part of the core language and must be imported from the <code>collections</code> module in the standard library.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">collections</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">d</span> <span class="o">=</span> <span class="n">collections</span><span class="o">.</span><span class="n">OrderedDict</span><span class="p">(</span><span class="n">one</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">two</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">three</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">d</span> <span class="n">OrderedDict</span><span class="p">([(</span><span class="s1">'one'</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="s1">'two'</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="p">(</span><span class="s1">'three'</span><span class="p">,</span> <span class="mi">3</span><span class="p">)])</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">d</span><span class="p">[</span><span class="s1">'four'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">4</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">d</span> <span class="n">OrderedDict</span><span class="p">([(</span><span class="s1">'one'</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="s1">'two'</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="p">(</span><span class="s1">'three'</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span> <span class="p">(</span><span class="s1">'four'</span><span class="p">,</span> <span class="mi">4</span><span class="p">)])</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">d</span><span class="o">.</span><span class="n">keys</span><span class="p">()</span> <span class="n">odict_keys</span><span class="p">([</span><span class="s1">'one'</span><span class="p">,</span> <span class="s1">'two'</span><span class="p">,</span> <span class="s1">'three'</span><span class="p">,</span> <span class="s1">'four'</span><span class="p">])</span> </pre></div> </section><section><h2>✅ <a href="https://docs.python.org/3/library/collections.html#collections.defaultdict" target="_blank">collections.defaultdict</a> – Return default values for missing keys</h2> <p>Another dictionary subclass that accepts a default value in its constructor that will be returned if a requested key cannot be found in a <code>defaultdict</code> instance. This can save some typing and make the programmer’s intention more clear compared to using the <code>get()</code> methods or catching a <code>KeyError</code> exception in regular dictionaries.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">defaultdict</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dd</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">list</span><span class="p">)</span> <span class="c1"># Accessing a missing key creates it and initializes it</span> <span class="c1"># using the default factory, i.e. list() in this example:</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dd</span><span class="p">[</span><span class="s1">'dogs'</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'Rufus'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dd</span><span class="p">[</span><span class="s1">'dogs'</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'Kathrin'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dd</span><span class="p">[</span><span class="s1">'dogs'</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'Mr Sniffles'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dd</span><span class="p">[</span><span class="s1">'dogs'</span><span class="p">]</span> <span class="p">[</span><span class="s1">'Rufus'</span><span class="p">,</span> <span class="s1">'Kathrin'</span><span class="p">,</span> <span class="s1">'Mr Sniffles'</span><span class="p">]</span> </pre></div> </section><section><h2>✅ <a href="https://docs.python.org/3/library/collections.html#collections.ChainMap" target="_blank">collections.ChainMap</a> – Search multiple dictionaries as a single mapping</h2> <p>This data structure groups multiple dictionaries into a single mapping. Lookups search the underlying mappings one by one until a key is found. Insertions, updates, and deletions only affect the first mapping added to the chain.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">ChainMap</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dict1</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'one'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">'two'</span><span class="p">:</span> <span class="mi">2</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dict2</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'three'</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span> <span class="s1">'four'</span><span class="p">:</span> <span class="mi">4</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">chain</span> <span class="o">=</span> <span class="n">ChainMap</span><span class="p">(</span><span class="n">dict1</span><span class="p">,</span> <span class="n">dict2</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">chain</span> <span class="n">ChainMap</span><span class="p">({</span><span class="s1">'one'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">'two'</span><span class="p">:</span> <span class="mi">2</span><span class="p">},</span> <span class="p">{</span><span class="s1">'three'</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span> <span class="s1">'four'</span><span class="p">:</span> <span class="mi">4</span><span class="p">})</span> <span class="c1"># ChainMap searches each collection in the chain</span> <span class="c1"># from left to right until it finds the key (or fails):</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">chain</span><span class="p">[</span><span class="s1">'three'</span><span class="p">]</span> <span class="mi">3</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">chain</span><span class="p">[</span><span class="s1">'one'</span><span class="p">]</span> <span class="mi">1</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">chain</span><span class="p">[</span><span class="s1">'missing'</span><span class="p">]</span> <span class="ne">KeyError</span><span class="p">:</span> <span class="s1">'missing'</span> </pre></div> </section><section><h2>✅ <a href="https://docs.python.org/3/library/types.html#types.MappingProxyType" target="_blank">types.MappingProxyType</a> – A wrapper for making read-only dictionaries</h2> <p>A wrapper around a standard dictionary that provides a read-only view into the wrapped dictionary’s data. This class was added in Python 3.3 and it can be used to create immutable proxy versions of dictionaries.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">types</span> <span class="kn">import</span> <span class="n">MappingProxyType</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">read_only</span> <span class="o">=</span> <span class="n">MappingProxyType</span><span class="p">({</span><span class="s1">'one'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">'two'</span><span class="p">:</span> <span class="mi">2</span><span class="p">})</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">read_only</span><span class="p">[</span><span class="s1">'one'</span><span class="p">]</span> <span class="mi">1</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">read_only</span><span class="p">[</span><span class="s1">'one'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">23</span> <span class="ne">TypeError</span><span class="p">:</span> <span class="s2">"'mappingproxy' object does not support item assignment"</span> </pre></div> <p><em><a href="fundamental-data-structures-in-python">Read the full “Fundamental Data Structures in Python” article series here.</a> This article is missing something or you found an error? Help a brother out and leave a comment below.</em></p></section><footer></footer></article>https://dbader.org/blog/python-dictionaries-maps-and-hashtablesTue, 18 Apr 2017 00:00:00 GMTPriority Queues in Pythonhttps://dbader.org/blog/priority-queues-in-python<article><header><h1>Priority Queues in Python</h1> <p>What are the various ways you can implement a priority queue in Python? Read on and find out what the Python standard library has to offer.</p> </header><section><figure><img alt="" src="/blog/figures/priority-queues-in-python.png" width="1280" height="720"></figure> <p>A priority queue is a container data structure that manages a set of records with <a href="https://en.wikipedia.org/wiki/Total_order" target="_blank">totally-ordered</a> keys (for example, a numeric <em>weight</em> value) to provide quick access to the record with the <em>smallest</em> or <em>largest</em> key in the set.</p> <p>You can think of a priority queue as a modified queue: instead of retrieving the next element by insertion time, it retrieves the <em>highest-priority</em> element. The priority of individual elements is decided by the ordering applied to their keys.</p> <p>Priority queues are commonly used for dealing with scheduling problems. For example, to give precedence to tasks with higher urgency.</p> <p>For example, let’s take an operating system task scheduler—ideally, high-priority tasks on the system (e.g. playing a real-time game) should take precedence over lower-priority tasks (e.g. downloading updates in the background). By organizing pending tasks in a priority queue that uses the task urgency as the key, the task scheduler can allow the highest-priority tasks to run first.</p> <p>Let’s take a look at a few options for how you can implement Priority Queues in Python using built-in data structures or data structures that ship with Python’s standard library. They each have their up- and downsides, but in my mind there’s a clear winner for most common scenarios. But see for yourself:</p> </section><section><h2>⛔ Keeping a Manually Sorted List</h2> <p>You can use a sorted <code>list</code> to quickly identify and delete the smallest or largest element. The downside is that inserting new elements into a list is a slow <em>O(n)</em> operation.</p> <p>While the insertion point can be found in <em>O(log n)</em> time using <a href="https://docs.python.org/3/library/bisect.html#bisect.insort" target="_blank"><code>bisect.insort</code></a> in the standard library, this is always dominated by the slow insertion step.</p> <p>Maintaining the order by appending to the list and re-sorting also takes at least <em>O(n log n)</em> time.</p> <p>Therefore sorted lists are <strong>only suitable when there will be few insertions</strong> into the priority queue.</p> <div class="codehilite"><pre><span></span><span class="n">q</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'code'</span><span class="p">))</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'eat'</span><span class="p">))</span> <span class="n">q</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="mi">3</span><span class="p">,</span> <span class="s1">'sleep'</span><span class="p">))</span> <span class="c1"># NOTE: Remember to re-sort every time</span> <span class="c1"># a new element is inserted, or use</span> <span class="c1"># bisect.insort().</span> <span class="n">q</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">reverse</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="k">while</span> <span class="n">q</span><span class="p">:</span> <span class="n">next_item</span> <span class="o">=</span> <span class="n">q</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="n">next_item</span><span class="p">)</span> <span class="c1"># Result:</span> <span class="c1"># (1, 'eat')</span> <span class="c1"># (2, 'code')</span> <span class="c1"># (3, 'sleep')</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/heapq.html" target="_blank">heapq</a> Module</h2> <p>This is a binary heap implementation usually backed by a plain <code>list</code> and it supports insertion and extraction of the smallest element in <em>O(log n)</em> time.</p> <p>This module is a good choice for implementing priority queues in Python. Because <code>heapq</code> technically only provides a min-heap implementation, extra steps must be taken to ensure sort stability and other features <a href="https://docs.python.org/3/library/heapq.html#priority-queue-implementation-notes" target="_blank">typically expected from a “practical” priority queue</a>.</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">heapq</span> <span class="n">q</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">heapq</span><span class="o">.</span><span class="n">heappush</span><span class="p">(</span><span class="n">q</span><span class="p">,</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'code'</span><span class="p">))</span> <span class="n">heapq</span><span class="o">.</span><span class="n">heappush</span><span class="p">(</span><span class="n">q</span><span class="p">,</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'eat'</span><span class="p">))</span> <span class="n">heapq</span><span class="o">.</span><span class="n">heappush</span><span class="p">(</span><span class="n">q</span><span class="p">,</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="s1">'sleep'</span><span class="p">))</span> <span class="k">while</span> <span class="n">q</span><span class="p">:</span> <span class="n">next_item</span> <span class="o">=</span> <span class="n">heapq</span><span class="o">.</span><span class="n">heappop</span><span class="p">(</span><span class="n">q</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">next_item</span><span class="p">)</span> <span class="c1"># Result:</span> <span class="c1"># (1, 'eat')</span> <span class="c1"># (2, 'code')</span> <span class="c1"># (3, 'sleep')</span> </pre></div> </section><section><h2>✅ The <a href="https://docs.python.org/3/library/queue.html#queue.PriorityQueue" target="_blank">queue.PriorityQueue</a> Class</h2> <p>This priority queue implementation uses <code>heapq</code> internally and shares the same time and space complexities.</p> <p>The difference is that <code>PriorityQueue</code> is synchronized and provides locking semantics to support multiple concurrent producers and consumers.</p> <p>Depending on your use case this might be helpful, or just incur unneeded overhead. In any case you might prefer its class-based interface over using the function-based interface provided by <code>heapq</code>.</p> <div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">queue</span> <span class="kn">import</span> <span class="n">PriorityQueue</span> <span class="n">q</span> <span class="o">=</span> <span class="n">PriorityQueue</span><span class="p">()</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'code'</span><span class="p">))</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">((</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'eat'</span><span class="p">))</span> <span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">((</span><span class="mi">3</span><span class="p">,</span> <span class="s1">'sleep'</span><span class="p">))</span> <span class="k">while</span> <span class="ow">not</span> <span class="n">q</span><span class="o">.</span><span class="n">empty</span><span class="p">():</span> <span class="n">next_item</span> <span class="o">=</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="n">next_item</span><span class="p">)</span> <span class="c1"># Result:</span> <span class="c1"># (1, 'eat')</span> <span class="c1"># (2, 'code')</span> <span class="c1"># (3, 'sleep')</span> </pre></div> </section><section><h2>A good default choice: <code>queue.PriorityQueue</code></h2> <p>Now which priority queue implementation should you use in your Python programs? They each have slightly different use cases. But in my mind <code>queue.PriorityQueue</code> is a good default choice.</p> <p>Sure, it might incur some unnecessary locking overhead—but it has a nice object-oriented interface and a name that states its intent clearly.</p> <p><em><a href="fundamental-data-structures-in-python">Read the full “Fundamental Data Structures in Python” article series here.</a> This article is missing something or you found an error? Help a brother out and leave a comment below.</em></p></section><footer></footer></article>https://dbader.org/blog/priority-queues-in-pythonWed, 12 Apr 2017 00:00:00 GMTFundamental Data Structures in Pythonhttps://dbader.org/blog/fundamental-data-structures-in-python<article><header><h1>Fundamental Data Structures in Python</h1> <p>In this article series we’ll take a tour of some fundamental data structures and implementations of abstract data types (ADTs) available in Python’s standard library.</p> </header><section><figure><img alt="" src="/blog/figures/fundamental-data-structures-in-python.png" width="1280" height="720"></figure> <p>Data structures are the fundamental constructs around which you build your applications. Each data structure provides a particular way of organizing data so it can be accessed efficiently, depending on the use case at hand.</p> <p>Python ships with an extensive set of data structures in its standard library. However due to naming differences it’s often unclear how even well-known “abstract data types” correspond to a specific implementation in Python.</p> <p>Other languages like Java stick to a more “computer sciencey” and explicit naming scheme for their standard data structures. For example, a list isn’t just a “list” in Java—it’s either a <code>LinkedList</code> or an <code>ArrayList</code>. This makes it easier to recognize the computational complexity of these types.</p> <p>Python favors a simpler and more “human” naming scheme. The downside is that to a Python initiate it’s unclear whether the built-in <code>list</code> type is implemented as a linked list or a dynamic array.</p> <p>My goal with this article series is to clarify how the most common abstract data types map to Python’s naming scheme and to provide a brief description for each. This information will also help you in Python coding interviews.</p> <p>If you’re looking for a good book to brush up your data structures knowledge I highly recommend Steven S. Skiena’s <a href="http://amzn.to/1oeOte3" target="_blank">The Algorithm Design Manual</a>.</p> <p>It strikes a great balance between teaching you fundamental (and more advanced) data structures and then showing you how to put them to practical use in various algorithms. Steve’s book was a great help in the writing of this series.</p> <p>Alright, let’s get started. This article serves as the “hub” for the individual data structure tutorials that I’ll link in the list below:</p> </section><section><h2>Python Data Structures Tutorials</h2> <ul> <li><a href="priority-queues-in-python">Priority Queues in Python</a></li> <li><a href="python-dictionaries-maps-and-hashtables">Dictionaries, Maps, and Hash Tables in Python</a></li> <li><a href="sets-and-multiset-in-python">Sets and Multisets in Python</a></li> <li><a href="queues-in-python">Queues in Python</a></li> <li><a href="stacks-in-python">Stacks in Python</a></li> <li>Graphs and Trees in Python (<em>Coming Soon</em>)</li> <li>Linked Lists in Python (<em>Coming Soon</em>)</li> <li>Arrays in Python (<em>Coming Soon</em>)</li> <li>Records and Data Objects in Python (<em>Coming Soon</em>)</li> </ul> <p>By the way, I’m always looking to improve these tutorials so if you find an error or would like to suggest an addition—please leave a comment on the article or reach out to me via email or <a href="https://twitter.com/@dbader_org" target="_blank">Twitter</a>.</p></section><footer></footer></article>https://dbader.org/blog/fundamental-data-structures-in-pythonTue, 11 Apr 2017 00:00:00 GMTPython Decorators: A Step-By-Step Introductionhttps://dbader.org/blog/python-decorators<article><header><h1>Python Decorators: A Step-By-Step Introduction</h1> <p>Understanding decorators is a milestone for any serious Python programmer. Here’s your step-by-step guide to how decorators can help you become a more efficient and productive Python developer.</p> </header><section><figure><img alt="Understanding Python Decorators" src="/blog/figures/python-decorators.png" width="1280" height="720"></figure> <p>Python’s decorators allow you to extend and modify the behavior of a callable (functions, methods, and classes) <em>without</em> permanently modifying the callable itself.</p> <p>Any sufficiently generic functionality you can “tack on” to an existing class or function’s behavior makes a great use case for decoration. This includes:</p> <ul> <li>logging,</li> <li>enforcing access control and authentication,</li> <li>instrumentation and timing functions,</li> <li>rate-limiting,</li> <li>caching; and more.</li> </ul> </section><section><h2>Why Should I Master Decorators in Python?</h2> <p>That’s a fair question. After all, what I just mentioned sounded quite abstract and it might be difficult to see <strong>how decorators can benefit you in your day-to-day work as a Python developer</strong>. Here’s an example:</p> <p>Imagine you’ve got 30 functions with business logic in your report-generating program. One rainy Monday morning your boss walks up to your desk and says:</p> <blockquote> <p>“Happy Monday! Remember those TPS reports? I need you to add input/output logging to each step in the report generator. XYZ Corp needs it for auditing purposes. Oh, and I told them we can ship this by Wednesday.”</p> </blockquote> <p>Depending on whether or not you’ve got a solid grasp on Python’s decorators, this request request will either send your blood pressure spiking—or leave you relatively calm.</p> <p><strong>Without decorators</strong> you might be spending the next three days scrambling to modify each of those 30 functions and clutter them up with manual logging calls. Fun times.</p> <p><strong>If you do know your decorators</strong>, you’ll calmly smile at your boss and say:</p> <blockquote> <p>“Don’t worry Jim, I’ll get it done by 2pm today.”</p> </blockquote> <p>Right after that you’ll type the code for a generic <code>@audit_log</code> decorator (that’s only about 10 lines long) and quickly paste it in front of each function definition. Then you’ll commit your code and grab another cup of coffee.</p> <p>I’m dramatizing here. But only a little. Decorators <em>can be</em> that powerful 🙂</p> <p>I’d go as far as to say that understanding decorators is a milestone for any serious Python programmer. They require a solid grasp of several advanced concepts in the language—including the <a href="python-first-class-functions">properties of <em>first-class functions</em></a>.</p> <p>But:</p> </section><section><h2>Understanding Decorators <u>Is Worth It</u> 💡</h2> <p>The payoff for understanding how decorators work in Python is huge.</p> <p>Sure, decorators are relatively complicated to wrap your head around for the first time—but they’re a highly useful feature that you’ll often encounter in third-party frameworks and the Python standard library.</p> <p>Explaining decorators is also a <em>make or break</em> moment for any good Python tutorial. I’ll do my best here to introduce you to them step by step.</p> <p>Before you dive in, now would be an excellent moment to refresh your memory on the properties of <em>first-class functions</em> in Python. I wrote <a href="python-first-class-functions">a tutorial on them here on dbader.org</a> and I would encourage you to take a few minutes to review it. The most important “first-class functions” takeaways for understanding decorators are:</p> <ul> <li><strong>Functions are objects</strong>—they can be assigned to variables and passed to and returned from other functions; and</li> <li><strong>Functions can be defined inside other functions</strong>—and a child function can capture the parent function’s local state (lexical closures.)</li> </ul> <p>Alright, ready to do this? Let’s start with some:</p> </section><section><h2>Python Decorator Basics</h2> <p>Now, what are decorators really? They “decorate” or “wrap” another function and let you execute code before and after the wrapped function runs.</p> <p>Decorators allow you to define reusable building blocks that can change or extend the behavior of other functions. And they let you do that without permanently modifying the wrapped function itself. The function’s behavior changes only when it’s <em>decorated</em>.</p> <p>Now what does the implementation of a simple decorator look like? In basic terms, a decorator is <em>a callable that takes a callable as input and returns another callable</em>.</p> <p>The following function has that property and could be considered the simplest decorator one could possibly write:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">null_decorator</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="k">return</span> <span class="n">func</span> </pre></div> <p>As you can see, <code>null_decorator</code> is a callable (it’s a function), it takes another callable as its input, and it returns the same input callable without modifying it.</p> <p>Let’s use it to <em>decorate</em> (or <em>wrap</em>) another function:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">greet</span><span class="p">():</span> <span class="k">return</span> <span class="s1">'Hello!'</span> <span class="n">greet</span> <span class="o">=</span> <span class="n">null_decorator</span><span class="p">(</span><span class="n">greet</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="p">()</span> <span class="s1">'Hello!'</span> </pre></div> <p>In this example I’ve defined a <code>greet</code> function and then immediately decorated it by running it through the <code>null_decorator</code> function. I know this doesn’t look very useful yet (I mean we specifically designed the null decorator to be useless, right?) but in a moment it’ll clarify how Python’s decorator syntax works.</p> <p>Instead of explicitly calling <code>null_decorator</code> on <code>greet</code> and then reassigning the <code>greet</code> variable, you can use Python’s <code>@</code> syntax for decorating a function in one step:</p> <div class="codehilite"><pre><span></span><span class="nd">@null_decorator</span> <span class="k">def</span> <span class="nf">greet</span><span class="p">():</span> <span class="k">return</span> <span class="s1">'Hello!'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="p">()</span> <span class="s1">'Hello!'</span> </pre></div> <p>Putting an <code>@null_decorator</code> line in front of the function definition is the same as defining the function first and then running through the decorator. Using the <code>@</code> syntax is just <em>syntactic sugar</em>, and a shortcut for this commonly used pattern.</p> <p>Note that using the <code>@</code> syntax decorates the function immediately at definition time. This makes it difficult to access the undecorated original without brittle hacks. Therefore you might choose to decorate some functions manually in order to retain the ability to call the undecorated function as well.</p> <p>So far, so good. Let’s see how:</p> </section><section><h2>Decorators Can Modify Behavior</h2> <p>Now that you’re a little more familiar with the decorator syntax, let’s write another decorator that <em>actually does something</em> and modifies the behavior of the decorated function.</p> <p>Here’s a slightly more complex decorator which converts the result of the decorated function to uppercase letters:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">uppercase</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="k">def</span> <span class="nf">wrapper</span><span class="p">():</span> <span class="n">original_result</span> <span class="o">=</span> <span class="n">func</span><span class="p">()</span> <span class="n">modified_result</span> <span class="o">=</span> <span class="n">original_result</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span> <span class="k">return</span> <span class="n">modified_result</span> <span class="k">return</span> <span class="n">wrapper</span> </pre></div> <p>Instead of simply returning the input function like the null decorator did, this <code>uppercase</code> decorator defines a new function on the fly (a closure) and uses it to <em>wrap</em> the input function in order to modify its behavior at call time.</p> <p>The <code>wrapper</code> closure has access to the undecorated input function and it is free to execute additional code before and after calling the input function. (Technically, it doesn’t even need to call the input function at all.)</p> <p>Note how up until now the decorated function has never been executed. Actually calling the input function at this point wouldn’t make any sense—you’ll want the decorator to be able to modify the behavior of its input function when it gets called eventually.</p> <p>Time to see the <code>uppercase</code> decorator in action. What happens if you decorate the original <code>greet</code> function with it?</p> <div class="codehilite"><pre><span></span><span class="nd">@uppercase</span> <span class="k">def</span> <span class="nf">greet</span><span class="p">():</span> <span class="k">return</span> <span class="s1">'Hello!'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="p">()</span> <span class="s1">'HELLO!'</span> </pre></div> <p>I hope this was the result you expected. Let’s take a closer look at what just happened here. Unlike <code>null_decorator</code>, our <code>uppercase</code> decorator returns a <em>different function object</em> when it decorates a function:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span> <span class="o">&lt;</span><span class="n">function</span> <span class="n">greet</span> <span class="n">at</span> <span class="mh">0x10e9f0950</span><span class="o">&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">null_decorator</span><span class="p">(</span><span class="n">greet</span><span class="p">)</span> <span class="o">&lt;</span><span class="n">function</span> <span class="n">greet</span> <span class="n">at</span> <span class="mh">0x10e9f0950</span><span class="o">&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">uppercase</span><span class="p">(</span><span class="n">greet</span><span class="p">)</span> <span class="o">&lt;</span><span class="n">function</span> <span class="n">uppercase</span><span class="o">.&lt;</span><span class="nb">locals</span><span class="o">&gt;.</span><span class="n">wrapper</span> <span class="n">at</span> <span class="mh">0x10da02f28</span><span class="o">&gt;</span> </pre></div> <p>And as you saw earlier, it needs to do that in order to modify the behavior of the decorated function when it finally gets called. The <code>uppercase</code> decorator is a function itself. And the only way to influence the “future behavior” of an input function it decorates is to replace (or <em>wrap</em>) the input function with a closure.</p> <p>That’s why <code>uppercase</code> defines and returns another function (the closure) that can then be called at a later time, run the original input function, and modify its result.</p> <p>Decorators modify the behavior of a callable through a wrapper so you don’t have to permanently modify the original. The callable isn’t permanently modified—its behavior changes only when decorated.</p> <p>This let’s you “tack on” reusable building blocks, like logging and other instrumentation, to existing functions and classes. It’s what makes decorators such a powerful feature in Python that’s frequently used in the standard library and in third-party packages.</p> <div class="update-box"> <h3>⏰ A Quick Intermission</h3> <p>By the way, if you feel like you need a quick coffee break at this point—that’s totally normal. In my opinion closures and decorators are some of the most difficult concepts to understand in Python. Take your time and don’t worry about figuring this out immediately. Playing through the code examples in an interpreter session one by one often helps make things sink in.</p> <p>I know you can do it 🙂</p> </div> </section><section><h2>Applying Multiple Decorators to a Single Function</h2> <p>Perhaps not surprisingly, you can apply more than one decorator to a function. This accumulates their effects and it’s what makes decorators so helpful as reusable building blocks.</p> <p>Here’s an example. The following two decorators wrap the output string of the decorated function in HTML tags. By looking at how the tags are nested you can see which order Python uses to apply multiple decorators:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">strong</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="k">def</span> <span class="nf">wrapper</span><span class="p">():</span> <span class="k">return</span> <span class="s1">'&lt;strong&gt;'</span> <span class="o">+</span> <span class="n">func</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'&lt;/strong&gt;'</span> <span class="k">return</span> <span class="n">wrapper</span> <span class="k">def</span> <span class="nf">emphasis</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="k">def</span> <span class="nf">wrapper</span><span class="p">():</span> <span class="k">return</span> <span class="s1">'&lt;em&gt;'</span> <span class="o">+</span> <span class="n">func</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'&lt;/em&gt;'</span> <span class="k">return</span> <span class="n">wrapper</span> </pre></div> <p>Now let’s take these two decorators and apply them to our <code>greet</code> function at the same time. You can use the regular <code>@</code> syntax for that and just “stack” multiple decorators on top of a single function:</p> <div class="codehilite"><pre><span></span><span class="nd">@strong</span> <span class="nd">@emphasis</span> <span class="k">def</span> <span class="nf">greet</span><span class="p">():</span> <span class="k">return</span> <span class="s1">'Hello!'</span> </pre></div> <p>What output do you expect to see if you run the decorated function? Will the <code>@emphasis</code> decorator add its <code>&lt;em&gt;</code> tag first or does <code>@strong</code> have precedence? Here’s what happens when you call the decorated function:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="p">()</span> <span class="s1">'&lt;strong&gt;&lt;em&gt;Hello!&lt;/em&gt;&lt;/strong&gt;'</span> </pre></div> <p>This clearly shows in what order the decorators were applied: from <em>bottom to top</em>. First, the input function was wrapped by the <code>@emphasis</code> decorator, and then the resulting (decorated) function got wrapped again by the <code>@strong</code> decorator.</p> <p>To help me remember this bottom to top order I like to call this behavior <em>decorator stacking</em>. You start building the stack at the bottom and then keep adding new blocks on top to work your way upwards.</p> <p>If you break down the above example and avoid the <code>@</code> syntax to apply the decorators, the chain of decorator function calls looks like this:</p> <div class="codehilite"><pre><span></span><span class="n">decorated_greet</span> <span class="o">=</span> <span class="n">strong</span><span class="p">(</span><span class="n">emphasis</span><span class="p">(</span><span class="n">greet</span><span class="p">))</span> </pre></div> <p>Again you can see here that the <code>emphasis</code> decorator is applied first and then the resulting wrapped function is wrapped again by the <code>strong</code> decorator.</p> <p>This also means that deep levels of decorator stacking will have an effect on performance eventually because they keep adding nested function calls. Usually this won’t be a problem in practice, but it’s something to keep in mind if you’re working on performance intensive code.</p> </section><section><h2>Decorating Functions That Accept Arguments</h2> <p>All examples so far only decorated a simple <em>nullary</em> <code>greet</code> function that didn’t take any arguments whatsoever. So the decorators you saw here up until now didn’t have to deal with forwarding arguments to the input function.</p> <p>If you try to apply one of these decorators to a function that takes arguments it will not work correctly. How do you decorate a function that takes arbitrary arguments?</p> <p>This is where <a href="https://www.youtube.com/watch?v=WcTXxX3vYgY" target="_blank">Python’s <code>*args</code> and <code>**kwargs</code> feature</a> for dealing with variable numbers of arguments comes in handy. The following <code>proxy</code> decorator takes advantage of that:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">proxy</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="k">def</span> <span class="nf">wrapper</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="k">return</span> <span class="n">func</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="k">return</span> <span class="n">wrapper</span> </pre></div> <p>There are two notable things going on with this decorator:</p> <ul> <li> <p>It uses the <code>*</code> and <code>**</code> operators in the <code>wrapper</code> closure definition to collect all positional and keyword arguments and stores them in variables (<code>args</code> and <code>kwargs</code>).</p> </li> <li> <p>The <code>wrapper</code> closure then forwards the collected arguments to the original input function using the <code>*</code> and <code>**</code> “argument unpacking” operators.</p> </li> </ul> <p>(It’s a bit unfortunate that the meaning of the star and double-star operators is overloaded and changes depending on the context they’re used in. But I hope you get the idea.)</p> <p>Let’s expand the technique laid out by the <code>proxy</code> decorator into a more useful practical example. Here’s a <code>trace</code> decorator that logs function arguments and results during execution time:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">trace</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="k">def</span> <span class="nf">wrapper</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'TRACE: calling {func.__name__}() '</span> <span class="n">f</span><span class="s1">'with {args}, {kwargs}'</span><span class="p">)</span> <span class="n">original_result</span> <span class="o">=</span> <span class="n">func</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'TRACE: {func.__name__}() '</span> <span class="n">f</span><span class="s1">'returned {original_result!r}'</span><span class="p">)</span> <span class="k">return</span> <span class="n">original_result</span> <span class="k">return</span> <span class="n">wrapper</span> </pre></div> <p>Decorating a function with <code>trace</code> and then calling it will print the arguments passed to the decorated function and its return value. This is still somewhat of a toy example—but in a pinch it makes a great debugging aid:</p> <div class="codehilite"><pre><span></span><span class="nd">@trace</span> <span class="k">def</span> <span class="nf">say</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">line</span><span class="p">):</span> <span class="k">return</span> <span class="n">f</span><span class="s1">'{name}: {line}'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">say</span><span class="p">(</span><span class="s1">'Jane'</span><span class="p">,</span> <span class="s1">'Hello, World'</span><span class="p">)</span> <span class="s1">'TRACE: calling say() with ("Jane", "Hello, World"), {}'</span> <span class="s1">'TRACE: say() returned "Jane: Hello, World"'</span> <span class="s1">'Jane: Hello, World'</span> </pre></div> <p>Speaking of debugging—there are some things you should keep in mind when debugging decorators:</p> </section><section><h2>How to Write “Debuggable” Decorators</h2> <p>When you use a decorator, really what you’re doing is replacing one function with another. One downside of this process is that it “hides” some of the metadata attached to the original (undecorated) function.</p> <p>For example, the original function name, its docstring, and parameter list are hidden by the wrapper closure:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">greet</span><span class="p">():</span> <span class="sd">"""Return a friendly greeting."""</span> <span class="k">return</span> <span class="s1">'Hello!'</span> <span class="n">decorated_greet</span> <span class="o">=</span> <span class="n">uppercase</span><span class="p">(</span><span class="n">greet</span><span class="p">)</span> </pre></div> <p>If you try to access any of that function metadata you’ll see the wrapper closure’s metadata instead:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="o">.</span><span class="vm">__name__</span> <span class="s1">'greet'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="o">.</span><span class="vm">__doc__</span> <span class="s1">'Return a friendly greeting.'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">decorated_greet</span><span class="o">.</span><span class="vm">__name__</span> <span class="s1">'wrapper'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">decorated_greet</span><span class="o">.</span><span class="vm">__doc__</span> <span class="bp">None</span> </pre></div> <p>This makes debugging and working with the Python interpreter awkward and challenging. Thankfully there’s a quick fix for this: the <a href="https://docs.python.org/3/library/functools.html#functools.wraps" target="_blank"><code>functools.wraps</code> decorator</a> included in Python’s standard library.</p> <p>You can use <code>functools.wraps</code> in your own decorators to copy over the lost metadata from the undecorated function to the decorator closure. Here’s an example:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">functools</span> <span class="k">def</span> <span class="nf">uppercase</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="nd">@functools.wraps</span><span class="p">(</span><span class="n">func</span><span class="p">)</span> <span class="k">def</span> <span class="nf">wrapper</span><span class="p">():</span> <span class="k">return</span> <span class="n">func</span><span class="p">()</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span> <span class="k">return</span> <span class="n">wrapper</span> </pre></div> <p>Applying <code>functools.wraps</code> to the wrapper closure returned by the decorator carries over the docstring and other metadata of the input function:</p> <div class="codehilite"><pre><span></span><span class="nd">@uppercase</span> <span class="k">def</span> <span class="nf">greet</span><span class="p">():</span> <span class="sd">"""Return a friendly greeting."""</span> <span class="k">return</span> <span class="s1">'Hello!'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="o">.</span><span class="vm">__name__</span> <span class="s1">'greet'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="o">.</span><span class="vm">__doc__</span> <span class="s1">'Return a friendly greeting.'</span> </pre></div> <p>As a best practice I’d recommend that you use <code>functools.wraps</code> in all of the decorators you write yourself. It doesn’t take much time and it will save you (and others) debugging headaches down the road.</p> </section><section><h2>Python Decorators – Key Takeaways</h2> <ul> <li>Decorators define reusable building blocks you can apply to a callable to modify its behavior without permanently modifying the callable itself.</li> <li>The <code>@</code> syntax is just a shorthand for calling the decorator on an input function. Multiple decorators on a single function are applied bottom to top (<em>decorator stacking</em>).</li> <li>As a debugging best practice, use the <a href="https://docs.python.org/3/library/functools.html#functools.wraps" target="_blank"><code>functools.wraps</code></a> helper in your own decorators to carry over metadata from the undecorated callable to the decorated one.</li> </ul> <p>Was this tutorial helpful? Got any suggestions on how it could be improved that could help other learners? Leave a comment below and share your thoughts.</p></section><footer></footer></article>https://dbader.org/blog/python-decoratorsTue, 04 Apr 2017 00:00:00 GMTFinding and Choosing Quality Python Packageshttps://dbader.org/blog/finding-and-choosing-quality-python-packages<article><header><h1>Finding and Choosing Quality Python Packages</h1> <p>PyPI, the Python packaging repository, just crossed 100,000 third-party packages in total the other week. That’s an <em>overwhelming</em> number of packages to choose from.</p> </header><section><figure><img alt="Finding and Choosing Quality Python Packages" src="/blog/figures/finding-and-choosing-python-packages.png" width="1280" height="720"></figure> </section><section><h2>The Quest for the Perfect Python Package</h2> <p>Back when I got “serious” about building my Python skills, mastering the syntax of the language was <em>not</em> the hardest part. Python’s syntax seemed quite clear and intuitive by comparison, and there was a (relatively) obvious path to learning it from books and other resources.</p> <p>But when it came to Python’s <em>tens of thousands</em> of libraries and frameworks that was simply an overwhelming number to choose from. Memorizing them was (and still is) an impossible task.</p> <p>And this feeling of overwhelm and “choice paralysis” is exactly what held me back earlier on in my Python career.</p> </section><section><h2>Mastering Python ≠ Mastering the Syntax</h2> <p>What tripped me up as a fledgling Pythonista was this: I had the basics of Python under my belt, but I struggled when it came to adopting the right workflows and tools of the “ecosystem” surrounding the core language.</p> <p>Thus, I wasted time reinventing existing solutions left and right—sometimes I spent <em>days</em> writing my own (terrible) versions of common building blocks like config file parsers, data validators, or visualization tools.</p> <p>Now, sure I learned quite a bit from doing that…</p> </section><section><h2>Overcoming “Reinventing the Wheel Disease”</h2> <p>But I kept repeating the same mistake and was “reinventing the wheel” even when under a tight deadline. In hindsight, my ignorance caused me a ton of undue stress and sleep deprivation.</p> <p>Part of it was overconfidence in my abilities, and another part was a lack of experience using “bread and butter” tools like the <code>pip</code> package manager, virtual environments, and requirements files.</p> <p>Once I got the hang of Python’s dependency management tools and workflows I was able to overcome my “reinventing the wheel disease” quickly.</p> </section><section><h2>Dependency Management Skills Are <u>Key</u></h2> <p>Mastering those tools and coming up with strategies for identifying high-quality Python packages opened up a whole new world to me:</p> <p>By leveraging Python’s packaging ecosystem I was suddenly coding at a higher level of abstraction—and it had a <em>massive</em> impact on my productivity and efficiency. Saying it allowed me to “10X” my output would <em>not</em> be too far off.</p> <p>If you use Python and you’re wondering how to go from “writing scripts” to “building applications”—then there’s a good chance you could benefit from focusing on your dependency management skills.</p> <p>You might be ready for a similar “quantum leap” in your productivity.</p> <p>To discover the strategies and exact steps I used to break through this barrier, check out my new “Managing Python Dependencies” course:</p> <p><a href="/products/managing-python-dependencies/"><strong>Click to Learn More About “Managing Python Dependencies” →</strong></a></p></section><footer></footer></article>https://dbader.org/blog/finding-and-choosing-quality-python-packagesTue, 28 Mar 2017 00:00:00 GMTPython’s Functions Are First-Classhttps://dbader.org/blog/python-first-class-functions<article><header><h1>Python’s Functions Are First-Class</h1> <p>Python’s functions are first-class objects. You can assign them to variables, store them in data structures, pass them as arguments to other functions, and even return them as values from other functions.</p> </header><section><figure><img alt="" src="/blog/figures/python-first-class-functions.png" width="1280" height="720"></figure> <p>Grokking these concepts intuitively will make understanding advanced features in Python like <a href="/blog/python-lambda-functions">lambdas</a> and decorators much easier. It also puts you on a path towards functional programming techniques.</p> <p>In this tutorial I’ll guide you through a number of examples to help you develop this intuitive understanding. The examples will build on top of one another, so you might want to read them in sequence and even to try out some of them in a Python interpreter session as you go along.</p> <p>Wrapping your head around the concepts we’ll be discussing here might take a little longer than expected. Don’t worry—that’s completely normal. I’ve been there. You might feel like you’re banging your head against the wall, and then suddenly things will “click” and fall into place when you’re ready.</p> <p>Throughout this tutorial I’ll be using this <code>yell</code> function for demonstration purposes. It’s a simple toy example with easily recognizable output:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">yell</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="k">return</span> <span class="n">text</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'!'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">yell</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span> <span class="s1">'HELLO!'</span> </pre></div> </section><section><h2>Functions Are Objects</h2> <p>All data in a Python program is <a href="https://docs.python.org/3/reference/datamodel.html#objects-values-and-types" target="_blank">represented by objects or relations between objects</a>. Things like strings, lists, modules, and functions are all objects. There’s nothing particularly special about functions in Python.</p> <p>Because the <code>yell</code> function is an <em>object</em> in Python you can assign it to another variable, just like any other object:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">bark</span> <span class="o">=</span> <span class="n">yell</span> </pre></div> <p>This line doesn’t call the function. It takes the function object referenced by <code>yell</code> and creates a second name pointing to it, <code>bark</code>. You could now also execute the same underlying function object by calling <code>bark</code>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">bark</span><span class="p">(</span><span class="s1">'woof'</span><span class="p">)</span> <span class="s1">'WOOF!'</span> </pre></div> <p>Function objects and their names are two separate concerns. Here’s more proof: You can delete the function’s original name (<code>yell</code>). Because another name (<code>bark</code>) still points to the underlying function you can still call the function through it:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">del</span> <span class="n">yell</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">yell</span><span class="p">(</span><span class="s1">'hello?'</span><span class="p">)</span> <span class="ne">NameError</span><span class="p">:</span> <span class="s2">"name 'yell' is not defined"</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">bark</span><span class="p">(</span><span class="s1">'hey'</span><span class="p">)</span> <span class="s1">'HEY!'</span> </pre></div> <p>By the way, Python attaches a string identifier to every function at creation time for debugging purposes. You can access this internal identifier with the <code>__name__</code> attribute:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">bark</span><span class="o">.</span><span class="vm">__name__</span> <span class="s1">'yell'</span> </pre></div> <p>While the function’s <code>__name__</code> is still “yell” that won’t affect how you can access it from your code. This identifier is merely a debugging aid. A <em>variable pointing to a function</em> and the <em>function itself</em> are two separate concerns.</p> <p>(<a href="https://www.python.org/dev/peps/pep-3155/" target="_blank">Since Python 3.3</a> there’s also <code>__qualname__</code> which serves a similar purpose and provides a <em>qualified name</em> string to disambiguate function and class names.)</p> </section><section><h2>Functions Can Be Stored In Data Structures</h2> <p>As functions are first-class citizens you can store them in data structures, just like you can with other objects. For example, you can add functions to a list:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">funcs</span> <span class="o">=</span> <span class="p">[</span><span class="n">bark</span><span class="p">,</span> <span class="nb">str</span><span class="o">.</span><span class="n">lower</span><span class="p">,</span> <span class="nb">str</span><span class="o">.</span><span class="n">capitalize</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">funcs</span> <span class="p">[</span><span class="o">&lt;</span><span class="n">function</span> <span class="n">yell</span> <span class="n">at</span> <span class="mh">0x10ff96510</span><span class="o">&gt;</span><span class="p">,</span> <span class="o">&lt;</span><span class="n">method</span> <span class="s1">'lower'</span> <span class="n">of</span> <span class="s1">'str'</span> <span class="n">objects</span><span class="o">&gt;</span><span class="p">,</span> <span class="o">&lt;</span><span class="n">method</span> <span class="s1">'capitalize'</span> <span class="n">of</span> <span class="s1">'str'</span> <span class="n">objects</span><span class="o">&gt;</span><span class="p">]</span> </pre></div> <p>Accessing the function objects stored inside the list works like it would with any other type of object:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">funcs</span><span class="p">:</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">f</span><span class="p">(</span><span class="s1">'hey there'</span><span class="p">))</span> <span class="o">&lt;</span><span class="n">function</span> <span class="n">yell</span> <span class="n">at</span> <span class="mh">0x10ff96510</span><span class="o">&gt;</span> <span class="s1">'HEY THERE!'</span> <span class="o">&lt;</span><span class="n">method</span> <span class="s1">'lower'</span> <span class="n">of</span> <span class="s1">'str'</span> <span class="n">objects</span><span class="o">&gt;</span> <span class="s1">'hey there'</span> <span class="o">&lt;</span><span class="n">method</span> <span class="s1">'capitalize'</span> <span class="n">of</span> <span class="s1">'str'</span> <span class="n">objects</span><span class="o">&gt;</span> <span class="s1">'Hey there'</span> </pre></div> <p>You can even call a function object stored in the list without assigning it to a variable first. You can do the lookup and then immediately call the resulting “disembodied” function object within a single expression:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">funcs</span><span class="p">[</span><span class="mi">0</span><span class="p">](</span><span class="s1">'heyho'</span><span class="p">)</span> <span class="s1">'HEYHO!'</span> </pre></div> </section><section><h2>Functions Can Be Passed To Other Functions</h2> <p>Because functions are objects you can pass them as arguments to other functions. Here’s a <code>greet</code> function that formats a greeting string using the function object passed to it and then prints it:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">greet</span><span class="p">(</span><span class="n">func</span><span class="p">):</span> <span class="n">greeting</span> <span class="o">=</span> <span class="n">func</span><span class="p">(</span><span class="s1">'Hi, I am a Python program'</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">greeting</span><span class="p">)</span> </pre></div> <p>You can influence the resulting greeting by passing in different functions. Here’s what happens if you pass the <code>yell</code> function to <code>greet</code>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="p">(</span><span class="n">yell</span><span class="p">)</span> <span class="s1">'HI, I AM A PYTHON PROGRAM!'</span> </pre></div> <p>Of course you could also define a new function to generate a different flavor of greeting. For example, the following <code>whisper</code> function might work better if you don’t want your Python programs to sound like Optimus Prime:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">whisper</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="k">return</span> <span class="n">text</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'...'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="p">(</span><span class="n">whisper</span><span class="p">)</span> <span class="s1">'hi, i am a python program...'</span> </pre></div> <p>The ability to pass function objects as arguments to other functions is powerful. It allows you to abstract away and pass around <em>behavior</em> in your programs. In this example, the <code>greet</code> function stays the same but you can influence its output by passing in different <em>greeting behaviors</em>.</p> <p>Functions that can accept other functions as arguments are also called <em>higher-order functions</em>. They are a necessity for the functional programming style.</p> <p>The classical example for higher-order functions in Python is the built-in <code>map</code> function. It takes a function and an iterable and calls the function on each element in the iterable, yielding the results as it goes along.</p> <p>Here’s how you might format a sequence of greetings all at once by <em>mapping</em> the <code>yell</code> function to them:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">list</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="n">yell</span><span class="p">,</span> <span class="p">[</span><span class="s1">'hello'</span><span class="p">,</span> <span class="s1">'hey'</span><span class="p">,</span> <span class="s1">'hi'</span><span class="p">]))</span> <span class="p">[</span><span class="s1">'HELLO!'</span><span class="p">,</span> <span class="s1">'HEY!'</span><span class="p">,</span> <span class="s1">'HI!'</span><span class="p">]</span> </pre></div> <p><code>map</code> has gone through the entire list and applied the <code>yell</code> function to each element.</p> </section><section><h2>Functions Can Be Nested</h2> <p>Python allows functions to be defined inside other functions. These are often called <em>nested functions</em> or <em>inner functions</em>. Here’s an example:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">speak</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="k">def</span> <span class="nf">whisper</span><span class="p">(</span><span class="n">t</span><span class="p">):</span> <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'...'</span> <span class="k">return</span> <span class="n">whisper</span><span class="p">(</span><span class="n">text</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">speak</span><span class="p">(</span><span class="s1">'Hello, World'</span><span class="p">)</span> <span class="s1">'hello, world...'</span> </pre></div> <p>Now, what’s going on here? Every time you call <code>speak</code> it defines a new inner function <code>whisper</code> and then calls it.</p> <p>And here’s the kicker—<code>whisper</code> <em>does not exist</em> outside <code>speak</code>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">whisper</span><span class="p">(</span><span class="s1">'Yo'</span><span class="p">)</span> <span class="ne">NameError</span><span class="p">:</span> <span class="s2">"name 'whisper' is not defined"</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">speak</span><span class="o">.</span><span class="n">whisper</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="s2">"'function' object has no attribute 'whisper'"</span> </pre></div> <p>But what if you really wanted to access that nested <code>whisper</code> function from outside <code>speak</code>? Well, functions are objects—you can <em>return</em> the inner function to the caller of the parent function.</p> <p>For example, here’s a function defining two inner functions. Depending on the argument passed to top-level function it selects and returns one of the inner functions to the caller:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">get_speak_func</span><span class="p">(</span><span class="n">volume</span><span class="p">):</span> <span class="k">def</span> <span class="nf">whisper</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="k">return</span> <span class="n">text</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'...'</span> <span class="k">def</span> <span class="nf">yell</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="k">return</span> <span class="n">text</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'!'</span> <span class="k">if</span> <span class="n">volume</span> <span class="o">&gt;</span> <span class="mf">0.5</span><span class="p">:</span> <span class="k">return</span> <span class="n">yell</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="n">whisper</span> </pre></div> <p>Notice how <code>get_speak_func</code> doesn’t actually <em>call</em> one of its inner functions—it simply selects the appropriate function based on the <code>volume</code> argument and then returns the function object:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">get_speak_func</span><span class="p">(</span><span class="mf">0.3</span><span class="p">)</span> <span class="o">&lt;</span><span class="n">function</span> <span class="n">get_speak_func</span><span class="o">.&lt;</span><span class="nb">locals</span><span class="o">&gt;.</span><span class="n">whisper</span> <span class="n">at</span> <span class="mh">0x10ae18</span><span class="o">&gt;</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">get_speak_func</span><span class="p">(</span><span class="mf">0.7</span><span class="p">)</span> <span class="o">&lt;</span><span class="n">function</span> <span class="n">get_speak_func</span><span class="o">.&lt;</span><span class="nb">locals</span><span class="o">&gt;.</span><span class="n">yell</span> <span class="n">at</span> <span class="mh">0x1008c8</span><span class="o">&gt;</span> </pre></div> <p>Of course you could then go on and call the returned function, either directly or by assigning it to a variable name first:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">speak_func</span> <span class="o">=</span> <span class="n">get_speak_func</span><span class="p">(</span><span class="mf">0.7</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">speak_func</span><span class="p">(</span><span class="s1">'Hello'</span><span class="p">)</span> <span class="s1">'HELLO!'</span> </pre></div> <p>Let that sink in for a second here… This means not only can functions <em>accept behaviors</em> through arguments but they can also <em>return behaviors</em>. How cool is that?</p> <p>You know what, this is starting to get a little loopy here. I’m going to take a quick coffee break before I continue writing (and I suggest you do the same.)</p> </section><section><h2>Functions Can Capture Local State</h2> <p>You just saw how functions can contain inner functions and that it’s even possible to return these (otherwise hidden) inner functions from the parent function.</p> <p>Best put on your seat belts on now because it’s going to get a little crazier still—we’re about to enter even deeper functional programming territory. (You had that coffee break, right?)</p> <p>Not only can functions return other functions, these inner functions can also <em>capture and carry some of the parent function’s state</em> with them.</p> <p>I’m going to slightly rewrite the previous <code>get_speak_func</code> example to illustrate this. The new version takes a “volume” <em>and</em> a “text” argument right away to make the returned function immediately callable:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">get_speak_func</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">volume</span><span class="p">):</span> <span class="k">def</span> <span class="nf">whisper</span><span class="p">():</span> <span class="k">return</span> <span class="n">text</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'...'</span> <span class="k">def</span> <span class="nf">yell</span><span class="p">():</span> <span class="k">return</span> <span class="n">text</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'!'</span> <span class="k">if</span> <span class="n">volume</span> <span class="o">&gt;</span> <span class="mf">0.5</span><span class="p">:</span> <span class="k">return</span> <span class="n">yell</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="n">whisper</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">get_speak_func</span><span class="p">(</span><span class="s1">'Hello, World'</span><span class="p">,</span> <span class="mf">0.7</span><span class="p">)()</span> <span class="s1">'HELLO, WORLD!'</span> </pre></div> <p>Take a good look at the inner functions <code>whisper</code> and <code>yell</code> now. Notice how they no longer have a <code>text</code> parameter? But somehow they can still access the <code>text</code> parameter defined in the parent function. In fact, they seem to <em>capture</em> and “remember” the value of that argument.</p> <p>Functions that do this are called <a href="https://en.wikipedia.org/wiki/Closure_(computer_programming)" target="_blank"><em>lexical closures</em></a> (or just <em>closures</em>, for short). A closure remembers the values from its enclosing lexical scope even when the program flow is no longer in that scope.</p> <p>In practical terms this means not only can functions <em>return behaviors</em> but they can also <em>pre-configure those behaviors</em>. Here’s another bare-bones example to illustrate this idea:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">make_adder</span><span class="p">(</span><span class="n">n</span><span class="p">):</span> <span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="n">x</span><span class="p">):</span> <span class="k">return</span> <span class="n">x</span> <span class="o">+</span> <span class="n">n</span> <span class="k">return</span> <span class="n">add</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_3</span> <span class="o">=</span> <span class="n">make_adder</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_5</span> <span class="o">=</span> <span class="n">make_adder</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_3</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span> <span class="mi">7</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_5</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span> <span class="mi">9</span> </pre></div> <p>In this example <code>make_adder</code> serves as a <em>factory</em> to create and configure “adder” functions. Notice how the “adder” functions can still access the <code>n</code> argument of the <code>make_adder</code> function (the enclosing scope).</p> </section><section><h2>Objects Can Behave Like Functions</h2> <p>Object’s aren’t functions in Python. But they can be made <em>callable</em>, which allows you to <em>treat them like functions</em> in many cases.</p> <p>If an object is callable it means you can use round parentheses <code>()</code> on it and pass function call arguments to it. Here’s an example of a callable object:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Adder</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">n</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">n</span> <span class="o">=</span> <span class="n">n</span> <span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">n</span> <span class="o">+</span> <span class="n">x</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_3</span> <span class="o">=</span> <span class="n">Adder</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_3</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span> <span class="mi">7</span> </pre></div> <p>Behind the scenes, “calling” an object instance as a function attempts to execute the object’s <code>__call__</code> method.</p> <p>Of course not all objects will be callable. That’s why there’s a built-in <code>callable</code> function to check whether an object appears callable or not:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">callable</span><span class="p">(</span><span class="n">plus_3</span><span class="p">)</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">callable</span><span class="p">(</span><span class="n">yell</span><span class="p">)</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">callable</span><span class="p">(</span><span class="bp">False</span><span class="p">)</span> <span class="bp">False</span> </pre></div> </section><section><h2>Key Takeaways</h2> <ul> <li>Everything in Python is an object, including functions. You can assign them to variables, store them in data structures, and pass or return them to and from other functions (first-class functions.)</li> <li>First-class functions allow you to abstract away and pass around behavior in your programs.</li> <li>Functions can be nested and they can capture and carry some of the parent function’s state with them. Functions that do this are called <em>closures</em>.</li> <li>Objects can be made callable which allows you to treat them like functions in many cases.</li> </ul></section><footer></footer></article>https://dbader.org/blog/python-first-class-functionsTue, 21 Mar 2017 00:00:00 GMTHow to Make Your Python Loops More Pythonichttps://dbader.org/blog/pythonic-loops<article><header><h1>How to Make Your Python Loops More Pythonic</h1> <p>Pythonize your C-style “for” and “while” loops by refactoring them using generators and other techniques.</p> </header><section><figure><img alt="Writing Pythonic Loops" src="/blog/figures/writing-pythonic-loops.png" width="1280" height="720"></figure> <p>One of the easiest ways to spot a developer with a background in C-style languages who only recently picked up Python is to look at how they write loops.</p> <p>For example, whenever I see a code snippet like this, that’s <strong>an example of someone trying to write Python like it’s C or Java</strong>:</p> <div class="codehilite"><pre><span></span><span class="n">my_items</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'a'</span><span class="p">,</span> <span class="s1">'b'</span><span class="p">,</span> <span class="s1">'c'</span><span class="p">]</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">while</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">my_items</span><span class="p">):</span> <span class="k">print</span><span class="p">(</span><span class="n">my_items</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span> </pre></div> </section><section><h2>Now, what’s so “unpythonic” about this code?</h2> <p>Two things could be improved in this code example:</p> <ul> <li>First, it keeps track of the index <code>i</code> manually—initializing it to zero and then carefully incrementing it upon every loop iteration.</li> <li>And second, it uses <code>len()</code> to get the size of a container in order to determine how often to iterate.</li> </ul> <p>In Python you can write loops that handle both of these responsibilities automatically. It’s a great idea to take advantage of that.</p> <p>If your code doesn’t have to keep track of a running index it’s much harder to write accidental infinite loops, for example. It also makes the code more concise and therefore more readable.</p> </section><section><h2>How to track the loop index <em>automatically</em></h2> <p>To refactor the <code>while</code>-loop in the code example, I’ll start by removing the code that manually updates the index. A good way to do that is with a <code>for</code>-loop in Python.</p> <p>Using Python’s <a href="https://docs.python.org/3/library/functions.html#func-range" target="_blank"><code>range()</code> built-in</a> I can generate the indexes automatically (without having to increment a running counter variable):</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">my_items</span><span class="p">))</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">]</span> </pre></div> <p>The <code>range</code> type represents an immutable sequence of numbers. It’s advantage over a regular <code>list</code> is that it always takes the same small amount of memory.</p> <p>Range objects don’t actually store the individual values representing the number sequence—instead they function as iterators and calculate the sequence values on the fly.</p> <p>(This is true for Python 3. In <a href="https://docs.python.org/2/library/functions.html#xrange" target="_blank">Python 2 you’ll need to use the <code>xrange()</code> built-in</a> to get this memory-saving behavior, as <code>range()</code> will construct a list object containing all the values.)</p> <p>Now, instead of incrementing <code>i</code> on each loop iteration, I could write a refactored version of that loop like this:</p> <div class="codehilite"><pre><span></span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">my_items</span><span class="p">)):</span> <span class="k">print</span><span class="p">(</span><span class="n">my_items</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> </pre></div> <p>This is better. However it still isn’t <em>super</em> Pythonic—in most cases when you see code that uses <code>range(len(...))</code> to iterate over a container it can be improved and simplified even further.</p> <p>Let’s have a look at how you might do that in practice.</p> </section><section><h2>💡 Python’s “for” loops are “for-each” loops</h2> <p>As I mentioned, <code>for</code>-loops in Python are really “for-each” loops that can iterate over items from a container or sequence directly, without having to look them up by index. I can take advantage of that and simplify my loop even further:</p> <div class="codehilite"><pre><span></span><span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">my_items</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> </pre></div> <p>I would consider this solution to be quite Pythonic. It’s nice and clean and almost reads like pseudo code from a text book. I don’t have to keep track of the container’s size or a running index to access elements.</p> <p>The container itself takes care of handing out the elements so they can be processed. If the container is <em>ordered</em>, so will be the resulting sequence of elements. If the container is <em>not ordered</em> it will return its elements in an arbitrary order but the loop will still cover all of them.</p> </section><section><h2>What if I <em>need</em> the item index?</h2> <p>Now, of course you won’t always be able to rewrite your loops like that. What if you <em>need</em> the item index, for example? There’s a Pythonic way to keep a running index that avoids the <code>range(len(...))</code> construct I recommended against.</p> <p>The <a href="https://docs.python.org/3/library/functions.html#enumerate" target="_blank"><code>enumerate()</code> built-in</a> is helpful in this case:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">item</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">my_items</span><span class="p">):</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'{i}: {item}'</span><span class="p">)</span> <span class="mi">0</span><span class="p">:</span> <span class="n">a</span> <span class="mi">1</span><span class="p">:</span> <span class="n">b</span> <span class="mi">2</span><span class="p">:</span> <span class="n">c</span> </pre></div> <p>You see, iterators in Python can return more than just one value. They can return tuples with an arbitrary number of values that can then be unpacked right inside the <code>for</code>-statement.</p> <p>This is very powerful. For example, you can use the same technique to iterate over the keys and values of a dictionary at the same time:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">emails</span> <span class="o">=</span> <span class="p">{</span> <span class="o">...</span> <span class="s1">'Bob'</span><span class="p">:</span> <span class="s1">'bob@example.com'</span><span class="p">,</span> <span class="o">...</span> <span class="s1">'Alice'</span><span class="p">:</span> <span class="s1">'alice@example.com'</span><span class="p">,</span> <span class="o">...</span> <span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">email</span> <span class="ow">in</span> <span class="n">emails</span><span class="o">.</span><span class="n">items</span><span class="p">():</span> <span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s1">'{name} → {email}'</span><span class="p">)</span> <span class="s1">'Bob → bob@example.com'</span> <span class="s1">'Alice → alice@example.com'</span> </pre></div> </section><section><h2>Okay, what if I just <em>have</em> to write a C-style loop?</h2> <p>There’s one more example I’d like to show you. What if you absolutely, positively need to write a C-style loop. For example, what if you must control the step size for the index? Imagine you had the following original C or Java <code>for</code>-loop:</p> <div class="codehilite"><pre><span></span><span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="n">a</span><span class="o">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">n</span><span class="o">;</span> <span class="n">i</span><span class="o">+=</span><span class="n">s</span><span class="o">)</span> <span class="o">{</span> <span class="c1">// ...</span> <span class="o">}</span> </pre></div> <p>How would this pattern translate to Python?</p> <p>The <code>range()</code> function comes to our rescue again—it can accept extra parameters to control the start value for the loop (<code>a</code>), the stop value (<code>n</code>), and the step size (<code>s</code>).</p> <p>Therefore our example C-style loop could be implemented as follows:</p> <div class="codehilite"><pre><span></span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">s</span><span class="p">):</span> <span class="c1"># ...</span> </pre></div> </section><section><h2>Main Takeaways</h2> <ul> <li>Writing C-style loops in Python is considered unpythonic. Avoid managing loop indexes and stop conditions manually if possible.</li> <li>Python’s <code>for</code>-loops are really “for-each” loops that can iterate over items from a container or sequence directly.</li> </ul> </section><section><h2>📺 Watch a video tutorial based on this article</h2> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/URxQGA9f_AA?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div></section><footer></footer></article>https://dbader.org/blog/pythonic-loopsTue, 14 Mar 2017 00:00:00 GMT5 Python Development Setup Tips to Boost Your Productivityhttps://dbader.org/blog/python-development-setup-tips-to-boost-your-productivity<article><header><h1>5 Python Development Setup Tips to Boost Your Productivity</h1> <p>I struggled with setting up an effective development environment as a new Python developer. It was difficult to build the right habits and to find a set of tools I enjoyed to use.</p> </header><section><figure><img alt="" src="/blog/figures/python-dev-setup-tips.png" width="1280" height="720"></figure> <p>Back then I didn’t understand how much this impacted my productivity. I didn’t even know some of the most valuable practices and tools I’m using today existed!</p> <p>As my experience grew I understood this was a common pain among Python developers. No matter who I spoke with—colleagues, strangers at a conference, or developers on web forums and mailing lists—I saw similar struggles.</p> <p>Today I believe entry-level Python programmers can make leaps in their productivity by adopting a few key practices and tools into their workflow.</p> <p>This article helps you identify and fix 5 common issues in your Python development setup. I experienced them all myself and in some cases helped others through them as a colleague and a team lead. If you can avoid these issues you’ll become a happier and more effective Python developer.</p> </section><section><h2>#1 – Don’t waste time doing the compiler’s job</h2> <p>When developer brains do what computer brains can do much better then that’s usually a costly mistake. One example is programmers spending time hunting bugs that could be spotted just as well by automated tools.</p> <p>For some reason, maybe because of Python’s dynamic nature and earlier status as a “scripting” language, it’s still rare to see it used with static code analysis tools and linters.</p> <p>But these tools are fantastic. They can help detect and avoid certain bugs and classes of errors completely. For example, they can catch functional bugs like misspelled identifiers or reveal code quality issues like unused variables and imports.</p> <p>I won’t say code analysis tools are a miracle cure—but they can help reduce debugging and code review time with a small initial time investment.</p> <p>If you’re looking for just one tool that will improve the quality of your Python code without getting in the way with false positives and verbose messages, then I’d recommend the <a href="https://pypi.python.org/pypi/pyflakes" target="_blank">Pyflakes</a> code linter. Pyflakes is open-source, available for free, and easy to set up.</p> <p>To get immediate feedback and catch bugs early I recommend you integrate Pyflakes with your code editor and build server. Automatic linting for code changes as part of your continuous integration process makes your life easier. It ensures all developers on your team use the same settings and no uncaught warnings slip through the cracks.</p> <p><strong>Tip 1: Use static code analysis tools like Pyflakes.</strong></p> </section><section><h2>#2 – Avoid fruitless code style discussions</h2> <p>Your team does code reviews? Great! But be aware that a common mistake among inexperienced code reviewers is to spend too much time on feedback that automated tools could give for them. I’m talking about code style issues.</p> <p>It’s easy for development teams to get into a habit where they mostly talk about code style issues in code reviews: “We need an extra space character here.” or “Class names should use camel case.”</p> <p>This is a form of bikeshedding that prevents developers from looking at the real issues. The ones that cost money and cause maintenance problems later on.</p> <p>A quick fix here is to pick one of the Python style guides available on the internet, like <a href="https://www.python.org/dev/peps/pep-0008/" target="_blank">PEP 8</a> or <a href="https://google.github.io/styleguide/pyguide.html" target="_blank">Google’s Python Style Guide</a>, and to put automated tools in place that make sure committed code follows the style guide.</p> <p>I recommend using <a href="https://www.python.org/dev/peps/pep-0008/" target="_blank">PEP 8</a> as a style guide in combination with the <a href="https://github.com/PyCQA/pycodestyle" target="_blank">Pycodestyle</a> or <a href="https://pypi.python.org/pypi/flake8" target="_blank">flake8</a> code style checker. This will help avoid most code style discussions and allows your team to focus on the issues that matter.</p> <p><strong>Tip 2: Pick a code style (PEP 8) and enforce it with automated tools.</strong></p> </section><section><h2>#3 – Micro delays and death by a thousand cuts</h2> <p>Usability research shows the large effect website page load time has on <a href="https://www.nngroup.com/articles/website-response-times/" target="_blank">user abandonment</a>: If people get bored waiting for something to happen it increases the chances that they’ll abandon the original task they had in mind.</p> <p>As software developers, waiting on tools to complete their job is a normal part of our day to day workflow. We’re always waiting for a module to install, a test to run, or a commit to finish (<a href="https://xkcd.com/303/" target="_blank">“It’s compiling!”</a>). Of course we’re not “abandoning” our work every time we have to wait a few seconds for a tool to run—keeping focused on the task at hand is part of our job after all.</p> <p>Yet, keeping that focus costs mental energy that we might then lack in other areas of our work: We get tired a little quicker in the afternoon, or introduce a tiny little extra bug with our latest commit.</p> <p>In my experience even small forced pauses and delays add up. Switching files in a slow editor or jumping between apps on a slow computer is frustrating! We can even apply this at a microscopic level and look into <a href="https://pavelfatin.com/typing-with-pleasure/" target="_blank">editor typing latencies</a>. I believe these micro delays add up, too. They cost us productivity and cause frustration.</p> <p>Got time for a little thought experiment? Let’s say you’re waiting for a task to complete for about 1 out of every 10 seconds you spend on productive work. That adds up to half a day per week, or 2 days a month, or a whole month of productive work you might be losing over the course of a year.</p> <p>Maybe this estimate is too high—but what if you could get an additional week of productive time a year just by spending an afternoon on optimizing your tools? I’d say that’s worth a try!</p> <p><strong>Tip 3: Your development tools should be fast. Favor simplicity.</strong></p> </section><section><h2>#4 – Don’t work with an unpleasant editing environment</h2> <p>Working with tools that I don’t enjoy crushes my productivity. You might know the feeling. Some tools are so frustrating to work with they zap your energy levels and motivation.</p> <p>What’s the most important tool that you work with every day as a developer? For me it’s my code editor. For some developers it might be their email client or a team chat app—but let’s hope that a large part of your day is spent writing code.</p> <p>This means it pays off in terms of productivity (and happiness!) to invest into an enjoyable code editing environment.</p> <p>As Python developers we have many editors and IDEs to choose from: <a href="http://www.vim.org/" target="_blank">Vim</a>, <a href="https://www.gnu.org/software/emacs/" target="_blank">Emacs</a>, <a href="https://www.jetbrains.com/pycharm/" target="_blank">PyCharm</a>, <a href="https://wingware.com/" target="_blank">Wing IDE</a>, <a href="https://atom.io/" target="_blank">Atom</a>, <a href="http://www.pydev.org/" target="_blank">Eclipse PyDev</a>, <a href="https://www.sublimetext.com/" target="_blank">Sublime Text</a>—just to name a few.</p> <p>I spent much time fine-tuning my editing environment over the years. After trying other editors and IDEs I <a href="https://dbader.org/blog/setting-up-sublime-text-for-python-development">eventually settled on Sublime Text</a>. I like its speed, simplicity, and stability. It just feels right for my programming workflow. And I arrived at this choice by trying as many other options as I could.</p> <p>Your choice might be different. The point I’m trying to make is you need to find out which tool works best for yourself and your unique needs. Go and try out some editors and see which one you enjoy the most. Your productivity will thank you for it.</p> <p><strong>Tip 4: Find the right editor and tailor it to your needs.</strong></p> </section><section><h2>#5 – Invest in your setup</h2> <p>I once worked with someone who used a commercial editor to write code. But that developer didn’t want to spend the money to purchase a license for it. Instead they used the trial version of the editor for months on end.</p> <p>The trial version of this particular editor has a nag screen that pops up every few minutes when you save a file, asking you to buy the full version. This developer constantly saved files out of habit and therefore got to see that nag screen hundreds of times a day…</p> <p>A license that would’ve removed the nag screen cost about $70. I love a frugal mindset but this was ridiculous! Trying to save some money on a critical tool you use all day was the wrong choice—I’m sure the nag screens and the subtle frustrations they caused added up to more than $70 of lost productivity.</p> <p>If you’re working for yourself then these license costs will be a business expense you can deduct from your taxes. If you’re working for a company I’m sure they’ll gladly invest in your tools if you explain how they make you more productive and more valuable as an employee.</p> <p>License costs for software development tools are low compared to what graphic designers or architects have to put up with, for example. Some of the best tools and editors are even available for free. Invest money in the right tools where it makes sense and your life (and career) will be better for it.</p> <p><strong>Tip 5: Invest in tools that make you happy and more effective.</strong></p> </section><section><h2>Where to start?</h2> <p>I showed you five common development setup issues that can harm your productivity as a Python programmer. Luckily most of them are easy to fix with the right approach:</p> <ul> <li><strong>Tip 1: Use static code analysis tools like Pyflakes.</strong></li> <li><strong>Tip 2: Pick a code style (PEP 8) and enforce it with automated tools.</strong></li> <li><strong>Tip 3: Your development tools should be fast. Favor simplicity.</strong></li> <li><strong>Tip 4: Find the right editor and tailor it to your needs.</strong></li> <li><strong>Tip 5: Invest in tools that make you happy and more effective.</strong></li> </ul> <p>Here’s a good way to start: Find the one problem that irritates you the most. You’ll want to <em>divide and conquer</em> instead of trying to achieve perfection immediately. Fix one small thing at a time. Then iterate and keep making improvements from there.</p> <p>Think of it as an investment—even small changes will compound over time and give you a nice long-term productivity gain. In my experience, success is all about building the right habits and a mindset of continuous improvement.</p> <p>A great development environment makes you feel confident and productive. When you feel right at home in your setup programming Python becomes even more enjoyable and fun. Good luck!</p> <p><em>(This article was originally published on <a href="https://techbeacon.com/productive-python-development-how-tackle-5-common-issues" target="_blank">TechBeacon</a>.)</em></p></section><footer></footer></article>https://dbader.org/blog/python-development-setup-tips-to-boost-your-productivityTue, 07 Mar 2017 00:00:00 GMTContext Managers and the “with” Statement in Pythonhttps://dbader.org/blog/python-context-managers-and-with-statement<article><header><h1>Context Managers and the “with” Statement in Python</h1> <p>The “with” statement in Python is regarded as an obscure feature by some. But when you peek behind the scenes of the underlying Context Manager protocol you’ll see there’s little “magic” involved.</p> </header><section><div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/iba-I4CrmyA?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p>So what’s the <code>with</code> statement good for? It helps simplify some common resource management patterns by abstracting their functionality and allowing them to be factored out and reused.</p> <p>In turn this helps you write more expressive code and makes it easier to avoid resource leaks in your programs.</p> <p>A good way to see this feature used effectively is by looking at examples in the Python standard library. A well-known example involves the <code>open()</code> function:</p> <div class="codehilite"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'hello.txt'</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'hello, world!'</span><span class="p">)</span> </pre></div> <p>Opening files using the <code>with</code> statement is generally recommended because it ensures that open file descriptors are closed automatically after program execution leaves the context of the <code>with</code> statement. Internally, the above code sample translates to something like this:</p> <div class="codehilite"><pre><span></span><span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'hello.txt'</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">try</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'hello, world'</span><span class="p">)</span> <span class="k">finally</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> </pre></div> <p>You can already tell that this is quite a bit more verbose. Note that the <code>try...finally</code> statement is significant. It wouldn’t be enough to just write something like this:</p> <div class="codehilite"><pre><span></span><span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'hello.txt'</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'hello, world'</span><span class="p">)</span> <span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> </pre></div> <p>This implementation won’t guarantee the file is closed if there’s an exception during the <code>f.write()</code> call—and therefore our program might leak a file descriptor. That’s why the <code>with</code> statement is so useful. It makes acquiring and releasing resources <em>properly</em> a breeze.</p> <p>Another good example where the <code>with</code> statement is used effectively in the Python standard library is the <code>threading.Lock</code> class:</p> <div class="codehilite"><pre><span></span><span class="n">some_lock</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Lock</span><span class="p">()</span> <span class="c1"># Harmful:</span> <span class="n">some_lock</span><span class="o">.</span><span class="n">acquire</span><span class="p">()</span> <span class="k">try</span><span class="p">:</span> <span class="c1"># Do something...</span> <span class="k">finally</span><span class="p">:</span> <span class="n">some_lock</span><span class="o">.</span><span class="n">release</span><span class="p">()</span> <span class="c1"># Better:</span> <span class="k">with</span> <span class="n">some_lock</span><span class="p">:</span> <span class="c1"># Do something...</span> </pre></div> <p>In both cases using a <code>with</code> statement allows you to abstract away most of the resource handling logic. Instead of having to write an explicit <code>try...finally</code> statement each time, <code>with</code> takes care of that for us.</p> <p>The <code>with</code> statement can make code dealing with system resources more readable. It also helps avoid bugs or leaks by making it almost impossible to forget cleaning up or releasing a resource after we’re done with it.</p> </section><section><h2>Supporting <code>with</code> in Your Own Objects</h2> <p>Now, there’s nothing special or magical about the <code>open()</code> function or the <code>threading.Lock</code> class and the fact that they can be used with a <code>with</code> statement. You can provide the same functionality in your own classes and functions by implementing so-called <a href="https://docs.python.org/3/reference/datamodel.html#context-managers" target="_blank">context managers</a>.</p> <p>What’s a context manager? It’s a simple “protocol” (or interface) that your object needs to follow so it can be used with the <code>with</code> statement. Basically all you need to do is add <code>__enter__</code> and <code>__exit__</code> methods to an object if you want it to function as a context manager. Python will call these two methods at the appropriate times in the resource management cycle.</p> <p>Let’s take a look at what this would look like in practical terms. Here’s how a simple implementation of the <code>open()</code> context manager might look like:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">ManagedFile</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span> <span class="k">def</span> <span class="fm">__enter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">file</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">file</span> <span class="k">def</span> <span class="fm">__exit__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">exc_type</span><span class="p">,</span> <span class="n">exc_val</span><span class="p">,</span> <span class="n">exc_tb</span><span class="p">):</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">file</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">file</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> </pre></div> <p>Our <code>ManagedFile</code> class follows the context manager protocol and now supports the <code>with</code> statement, just like the original <code>open()</code> example did:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">with</span> <span class="n">ManagedFile</span><span class="p">(</span><span class="s1">'hello.txt'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="o">...</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'hello, world!'</span><span class="p">)</span> <span class="o">...</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'bye now'</span><span class="p">)</span> </pre></div> <p>Python calls <code>__enter__</code> when execution <em>enters</em> the context of the <code>with</code> statement and it’s time to acquire the resource. When execution <em>leaves</em> the context again, Python calls <code>__exit__</code> to free up the resource.</p> <p>Writing a class-based context manager isn’t the only way to support the <code>with</code> statement in Python. The <a href="https://docs.python.org/3/library/contextlib.html" target="_blank"><code>contextlib</code> utility module in the standard library</a> provides a few more abstractions built on top of the basic context manager protocol. This can make your life a little easier if your use cases matches what’s offered by <code>contextlib</code>.</p> <p>For example, you can use the <code>contextlib.contextmanager</code> decorator to define a generator-based <em>factory function</em> for a resource that will then automatically support the <code>with</code> statement. Here’s what rewriting our <code>ManagedFile</code> context manager with this technique looks like:</p> <div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">contextlib</span> <span class="kn">import</span> <span class="n">contextmanager</span> <span class="nd">@contextmanager</span> <span class="k">def</span> <span class="nf">managed_file</span><span class="p">(</span><span class="n">name</span><span class="p">):</span> <span class="k">try</span><span class="p">:</span> <span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">yield</span> <span class="n">f</span> <span class="k">finally</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">with</span> <span class="n">managed_file</span><span class="p">(</span><span class="s1">'hello.txt'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="o">...</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'hello, world!'</span><span class="p">)</span> <span class="o">...</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'bye now'</span><span class="p">)</span> </pre></div> <p>In this case, <code>managed_file()</code> is a generator that first acquires the resource. Then it temporarily suspends its own executing and <em>yields</em> the resource so it can be used by the caller. When the caller leaves the <code>with</code> context, the generator continues to execute so that any remaining clean up steps can happen and the resource gets released back to the system.</p> <p>Both the class-based implementations and the generator-based are practically equivalent. Depending on which one you find more readable you might prefer one over the other.</p> <p>A downside of the <code>@contextmanager</code>-based implementation might be that it requires understanding of advanced Python concepts, like decorators and generators.</p> <p>Once again, making the right choice here comes down to what you and your team are comfortable using and find the most readable.</p> </section><section><h2>Writing Pretty APIs With Context Managers</h2> <p>Context managers are quite flexible and if you use the <code>with</code> statement creatively you can define convenient APIs for your modules and classes.</p> <p>For example, what if the “resource” we wanted to manage was text indentation levels in some kind of report generator program? What if we could write code like this to do it:</p> <div class="codehilite"><pre><span></span><span class="k">with</span> <span class="n">Indenter</span><span class="p">()</span> <span class="k">as</span> <span class="n">indent</span><span class="p">:</span> <span class="n">indent</span><span class="o">.</span><span class="k">print</span><span class="p">(</span><span class="s1">'hi!'</span><span class="p">)</span> <span class="k">with</span> <span class="n">indent</span><span class="p">:</span> <span class="n">indent</span><span class="o">.</span><span class="k">print</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span> <span class="k">with</span> <span class="n">indent</span><span class="p">:</span> <span class="n">indent</span><span class="o">.</span><span class="k">print</span><span class="p">(</span><span class="s1">'bonjour'</span><span class="p">)</span> <span class="n">indent</span><span class="o">.</span><span class="k">print</span><span class="p">(</span><span class="s1">'hey'</span><span class="p">)</span> </pre></div> <p>This almost reads like a domain-specific language (DSL) for indenting text. Also, notice how this code enters and leaves the same context manager multiple times to change indentation levels. Running this code snippet should lead to the following output and print neatly formatted text:</p> <div class="codehilite"><pre><span></span>hi! hello bonjour hey </pre></div> <p>How would you implement a context manager to support this functionality?</p> <p>By the way, this could be a great exercise to wrap your head around how context managers work. So before you check out my implementation below you might take some time and try to implement this yourself as a learning exercise.</p> <p>Ready? Here’s how we might implement this functionality using a class-based context manager:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Indenter</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">level</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">def</span> <span class="fm">__enter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">level</span> <span class="o">+=</span> <span class="mi">1</span> <span class="k">return</span> <span class="bp">self</span> <span class="k">def</span> <span class="fm">__exit__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">exc_type</span><span class="p">,</span> <span class="n">exc_val</span><span class="p">,</span> <span class="n">exc_tb</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">level</span> <span class="o">-=</span> <span class="mi">1</span> <span class="k">def</span> <span class="nf">print</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">text</span><span class="p">):</span> <span class="k">print</span><span class="p">(</span><span class="s1">' '</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">level</span> <span class="o">+</span> <span class="n">text</span><span class="p">)</span> </pre></div> <p>Another good exercise would be trying to refactor this code to be generator-based.</p> </section><section><h2>Things to Remember</h2> <ul> <li>The <code>with</code> statement simplifies exception handling by encapsulating standard uses of <code>try/finally</code> statements in so-called Context Managers.</li> <li>Most commonly it is used to manage the safe acquisition and release of system resources. Resources are acquired by the <code>with</code> statement and released automatically when execution leaves the <code>with</code> context.</li> <li>Using <code>with</code> effectively can help you avoid resource leaks and make your code easier to read.</li> </ul></section><footer></footer></article>https://dbader.org/blog/python-context-managers-and-with-statementTue, 28 Feb 2017 00:00:00 GMTInstalling Python and Pip on Windowshttps://dbader.org/blog/installing-python-and-pip-on-windows-10<article><header><h1>Installing Python and Pip on Windows</h1> <p>In this tutorial you’ll learn how to set up Python and the Pip package manager on Windows 10, completely from scratch.</p> </header><section><figure><img alt="" src="/blog/figures/python-windows-setup.png" width="1280" height="720"></figure> </section><section><h2>Step 1: Download the Python Installer</h2> <p>The best way to install Python on Windows is by downloading the official Python installer from the Python website at <a href="https://python.org" target="_blank">python.org</a>.</p> <p>To do so, open a browser and navigate to <a href="https://python.org/" target="_blank">https://python.org/</a>. After the page has finished loading, click <strong>Downloads.</strong></p> <ul> <li>The website should detect that you’re on Windows and offer you to download the latest version of Python 3 or Python 2. If you don’t know which version of Python to use then I recommend <strong>Python 3</strong>. If you know you’ll need to work with legacy Python 2 code <em>only then</em> should you pick Python 2.</li> </ul> <p>Under <strong>Downloads → Download for Windows</strong>, click the <strong>“Python 3.X.X”</strong> (or “Python 2.X.X”) button to begin downloading the installer.</p> <figure><img alt="" src="/blog/figures/windows-setup-download-the-python-installer.jpg" width="1212" height="942"></figure> <div class="update-box"> <h2>Sidebar: 64-bit Python vs 32-bit Python</h2> <p>If you’re wondering whether you should use a 32-bit or a 64-bit version of Python then you might want to go with the 32-bit version.</p> <p>It’s sometimes still problematic to find binary extensions for 64-bit Python on Windows, which means that some third-party modules might not install correctly with a 64-bit version of Python.</p> <p>My thinking is that it’s best to go with the version currently recommended on python.org. If you click the Python 3 or Python 2 button under “Download for Windows” you’ll get just that.</p> <p>Remember that if you get this choice wrong and you’d like to switch to another version of Python you can just uninstall Python and then re-install it by downloading another installer from <a href="https://python.org" target="_blank">python.org</a>.</p> </div> </section><section><h2>Step 2: Run the Python Installer</h2> <p>Once the Python installer file has finished downloading, <strong>launch it</strong> by double-clicking on it in order to begin the installation.</p> <p>Be sure to select the <strong>Add Python X.Y to PATH</strong> checkbox in the setup wizard.</p> <ul> <li>Please make sure the “Add Python X.Y to PATH” checkbox was enabled in the installer because otherwise you will have problems accessing your Python installation from the command line. If you accidentally installed Python without checking the box, <a href="http://www.anthonydebarros.com/2015/08/16/setting-up-python-in-windows-10/" target="_blank">follow this tutorial</a> to add <code>python.exe</code> to your system PATH.</li> </ul> <p>Click <strong>Install Now</strong> to begin the installation process. The installation should finish quickly and then Python will be ready to go on your system. We’re going to make sure everything was set up correctly in the next step.</p> <figure><img alt="" src="/blog/figures/windows-setup-run-the-python-installer.jpg" width="684" height="427"></figure> </section><section><h2>Step 3: Verify Python Was Installed Correctly</h2> <p>After the Python installer finished its work Python should be installed on your system. Let’s make sure everything went correctly by testing if Python can be accessed from the Windows Command Prompt:</p> <ol> <li>Open the Windows Command Prompt by launching <strong>cmd.exe</strong></li> <li>Type <strong>pip</strong> and hit <strong>Return</strong></li> <li>You should see the help text from Python’s “pip” package manager. If you get an error message running <strong>pip</strong> go through the Python install steps again to make sure you have a working Python installation. Most issues you will encounter here will have something to do with the PATH not being set correctly. Re-installing and making sure that the “Add Python to PATH” option is enabled in the installer should resolve this.</li> </ol> <figure><img alt="" src="/blog/figures/windows-setup-verify-python-was-installed-correctly.jpg" width="1351" height="1028"></figure> </section><section><h2>What Now?</h2> <p>Assuming everything went well and you saw the output from Pip in your command prompt window—Congratulations, you just installed Python on your system!</p> <p>Wondering where to go from here? <a href="/python-basics">Click here to get some pointers</a> for Python beginners.</p></section><footer></footer></article>https://dbader.org/blog/installing-python-and-pip-on-windows-10Sat, 25 Feb 2017 00:00:00 GMTSublime Text Settings for Writing Clean Pythonhttps://dbader.org/blog/sublime-text-settings-for-writing-clean-python<article><header><h1>Sublime Text Settings for Writing Clean Python</h1> <p>How to write beautiful and clean Python by tweaking your Sublime Text settings so that they make it easier to adhere to the PEP 8 style guide recommendations.</p> </header><section><figure><img alt="" src="/blog/figures/sublime-settings-pep8-python.png" width="1280" height="720"></figure> <p>There are a few settings you can change to make it easier for you to write PEP 8 compliant Python with Sublime Text 3. <a href="https://www.python.org/dev/peps/pep-0008/" target="_blank">PEP 8 is the most common Python style guide</a> and widely used in the Python community.</p> <p>The tweaks I describe in this article mainly deal with getting the placement of whitespace correct so that you don’t have to manage this (boring) aspect yourself.</p> <p>I’ll also show you how to get visual indicators for the maximum allowed line-lengths in your editor window so that your lines can be concise and beautifully PEP 8 compliant—just like Guido wants them to be 🙂</p> </section><section><h2>Optional: Opening Sublime’s Syntax-Specific Settings for Python</h2> <p>The settings we’re changing now are specific to Python. Feel free to place them in your User settings, that will work just fine. However if you’d like to apply some or all of the settings in this chapter only to Python code then here’s how you can do that:</p> <ol> <li>Open a Python file in Sublime Text (or create a new file, open the Command Palette and execute the “Set Syntax: Python” command)</li> <li>Click on <strong>Sublime Text → Preferences → Settings – More → Syntax Specific – User</strong> to open your Python-specific user settings. Make sure this opens a new editor tab called <strong>Python.sublime-settings</strong>. That’s the one you want!</li> </ol> <figure><img alt="" src="/blog/figures/sublime-pep8-syntax-specific-settings.png" width="926" height="564"></figure> <p>If you’d like like to learn more about how Sublime Text’s preferences system works, then <a href="sublime-text-preferences-tutorial">check out this tutorial I wrote</a>.</p> </section><section><h2>Better Whitespace Handling</h2> <p>The following changes you can make to your (<strong>Syntax Specific</strong>) <strong>User Settings</strong> will help you keep the whitespace in your Python code clean and consistent:</p> <div class="codehilite"><pre><span></span><span class="s2">"tab_size"</span><span class="o">:</span> <span class="mi">4</span><span class="p">,</span> <span class="s2">"translate_tabs_to_spaces"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="s2">"trim_trailing_white_space_on_save"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="s2">"ensure_newline_at_eof_on_save"</span><span class="o">:</span> <span class="kc">true</span> </pre></div> <p>A <strong>tab_size</strong> of <strong>4</strong> is the general recommendation for writing Python. You’ll also want to enable <strong>translate_tabs_to_spaces</strong> to ensure that you don’t have a mixture of tabs and spaces in your Python files, which should be avoided.</p> <p>The <strong>trim_trailing_white_space_on_save</strong> option will remove superfluous whitespace at the end of lines or on empty lines. I highly recommend enabling this because it can save headaches and merge conflicts when working with Git and other forms of source control.</p> <p>PEP 8 recommends that Python files should end with a blank line to ensure that <a href="https://stackoverflow.com/questions/729692/why-should-text-files-end-with-a-newline" target="_blank">POSIX tools can process the file correctly</a>. If you want to never have to worry about this again then turn on the <strong>ensure_newline_at_eof_on_save</strong> setting as this will make sure that your Python files end with a newline automatically.</p> </section><section><h2>Enable PEP 8 Line-Length Indicators</h2> <p>Another setting that’s really handy for writing PEP 8 compliant code is the “rulers” feature. It enables visual indicators in the editor area that show you the preferred maximum line length.</p> <p>You can enable several rulers with different line lengths at the same time. This helps you follow the PEP 8 recommendations of limiting your docstrings to 72 characters and limiting all other lines to 79 characters.</p> <p>Here’s how to set up the rulers feature for Python development. Open your (<strong>Syntax Specific</strong>) <strong>User Settings</strong> and add the following setting:</p> <div class="codehilite"><pre><span></span><span class="s2">"rulers"</span><span class="o">:</span> <span class="p">[</span> <span class="mi">72</span><span class="p">,</span> <span class="mi">79</span> <span class="p">]</span> </pre></div> <p>This will add two line-length indicators—one at 72 characters for docstrings, and one at 79 characters for regular lines. You can see them in the screenshot as vertical lines on the right-hand side of the editor area.</p> <figure><img alt="" src="/blog/figures/sublime-pep8-line-length-indicators.png" width="992" height="488"></figure> </section><section><h2>Turn On Word Wrapping</h2> <p>I like enabling Sublime’s word-wrapping feature when I’m writing Python. Most of my projects follow the PEP 8 style guide and therefore use a maximum line length of 79 characters.</p> <p>I don’t want to get into an argument why that’s a good idea or not—but one benefit I found from limiting the lengths of my lines is that I can comfortably fit several files on my screen at once using Sublime’s “split layouts” feature.</p> <p>This is especially useful if you’re following a test-heavy development process because you can see and edit the test and the production code at the same time.</p> <p>Of course sometimes you’ll encounter a file that uses line-lengths above the 79 characters recommended by PEP 8. If I’m using split layouts with multiple editor panes at the same time it impacts my productivity if I have to scroll around horizontally.</p> <p>The idea is to see <em>all of the code</em> at once. So, how can we fix that?</p> <p>The best way I found to handle this is to enable Sublime’s word-wrap feature. This will visually break apart lines that are longer than the maximum line length. It might look a little odd sometimes but it’s still light years better than having to scroll around horizontally.</p> <p>Here’s how you enable word wrapping. Open your (<strong>Syntax Specific</strong>) <strong>User Settings</strong> and add (or modify) the following options:</p> <div class="codehilite"><pre><span></span><span class="s2">"word_wrap"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="s2">"wrap_width"</span><span class="o">:</span> <span class="mi">80</span> </pre></div> <p>I’m setting the <strong>wrap_width</strong> to 80 which is one character past the 79 characters recommended by PEP 8. Therefore any line that goes beyond the PEP 8 recommendations will get wrapped.</p> <figure><img alt="" src="/blog/figures/sublime-pep8-word-wrapping.png" width="964" height="308"></figure></section><footer></footer></article>https://dbader.org/blog/sublime-text-settings-for-writing-clean-pythonTue, 21 Feb 2017 00:00:00 GMTWriting Clean Python With Namedtupleshttps://dbader.org/blog/writing-clean-python-with-namedtuples<article><header><h1>Writing Clean Python With Namedtuples</h1> <p>Python comes with a specialized “namedtuple” container type that doesn’t seem to get the attention it deserves. It’s one of these amazing features in Python that’s hidden in plain sight.</p> </header><section><figure><img alt="" src="/blog/figures/namedtuples.png" width="1280" height="720"></figure> <p>Namedtuples can be a great alternative to defining a class manually and they have some other interesting features that I want to introduce you to in this article.</p> <p>Now, what’s a namedtuple and what makes it so special? A good way to think about namedtuples is to view them as an extension of the built-in tuple data type.</p> <p>Python’s tuples are a simple data structure for grouping arbitrary objects. Tuples are also immutable—they cannot be modified once they’ve been created.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">tup</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'hello'</span><span class="p">,</span> <span class="nb">object</span><span class="p">(),</span> <span class="mi">42</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">tup</span> <span class="p">(</span><span class="s1">'hello'</span><span class="p">,</span> <span class="o">&lt;</span><span class="nb">object</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x105e76b70</span><span class="o">&gt;</span><span class="p">,</span> <span class="mi">42</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">tup</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="mi">42</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">tup</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="mi">23</span> <span class="ne">TypeError</span><span class="p">:</span> <span class="s2">"'tuple' object does not support item assignment"</span> </pre></div> <p>A downside of plain tuples is that the data you store in them can only be pulled out by accessing it through integer indexes. You can’t give names to individual properties stored in a tuple. This can impact code readability.</p> <p>Also, a tuple is always an ad-hoc structure. It’s hard to ensure that two tuples have the same number of fields and the same properties stored on them. This makes it easy to introduce “slip-of-the-mind” bugs by mixing up the field order.</p> </section><footer><h3>Namedtuples to the Rescue</h3> <p>Namedtuples aim to solve these two problems.</p> <p>First of all, namedtuples are immutable just like regular tuples. Once you store something in them you can’t modify it.</p> <p>Besides that, namedtuples are, well…<em>named tuples</em>. Each object stored in them can be accessed through a unique (human-readable) identifier. This frees you from having to remember integer indexes, or resorting to workarounds like defining integer constants as mnemonics for your indexes.</p> <p>Here’s what a namedtuple looks like:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">namedtuple</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">Car</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span><span class="s1">'Car'</span> <span class="p">,</span> <span class="s1">'color mileage'</span><span class="p">)</span> </pre></div> <p>To use namedtuples you need to import the <code>collections</code> module. They were added to the standard library in Python 2.6. In the above example we defined a simple “Car” data type with two fields: “color” and “mileage”.</p> <p>You might find the syntax a little weird here—Why are we passing the fields as a string encoding them <code>"color mileage"</code>?</p> <p>The answer is that namedtuple’s factory function calls <code>split()</code> on the field names string, so this is really just a shorthand to say the following:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="s1">'color mileage'</span><span class="o">.</span><span class="n">split</span><span class="p">()</span> <span class="p">[</span><span class="s1">'color'</span><span class="p">,</span> <span class="s1">'mileage'</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">Car</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span><span class="s1">'Car'</span><span class="p">,</span> <span class="p">[</span><span class="s1">'color'</span><span class="p">,</span> <span class="s1">'mileage'</span><span class="p">])</span> </pre></div> <p>Of course you can also pass a list with string field names directly if you prefer how that looks. The advantage of using a proper list is that it’s easier to reformat this code if you need to split it across multiple lines:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">Car</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span><span class="s1">'Car'</span><span class="p">,</span> <span class="p">[</span> <span class="o">...</span> <span class="s1">'color'</span><span class="p">,</span> <span class="o">...</span> <span class="s1">'mileage'</span><span class="p">,</span> <span class="o">...</span> <span class="p">])</span> </pre></div> <p>However you decide, you can now create new “car” objects with the <code>Car</code> factory function. It behaves as if you had defined a <code>Car</code> class manually and given it a constructor accepting a “color” and a “mileage” value:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span> <span class="o">=</span> <span class="n">Car</span><span class="p">(</span><span class="s1">'red'</span><span class="p">,</span> <span class="mf">3812.4</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span><span class="o">.</span><span class="n">color</span> <span class="s1">'red'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span><span class="o">.</span><span class="n">mileage</span> <span class="mf">3812.4</span> </pre></div> <p>Tuple unpacking and the <a href="https://www.youtube.com/watch?v=YWY4BZi_o28" target="_blank"><code>*</code>-operator for function argument unpacking</a> also work as expected:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">color</span><span class="p">,</span> <span class="n">mileage</span> <span class="o">=</span> <span class="n">my_car</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="n">color</span><span class="p">,</span> <span class="n">mileage</span><span class="p">)</span> <span class="n">red</span> <span class="mf">3812.4</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="o">*</span><span class="n">my_car</span><span class="p">)</span> <span class="n">red</span> <span class="mf">3812.4</span> </pre></div> <p>Besides accessing the values stored in a namedtuple by their identifiers, you can still access them by their index. That way namedtuples can be used as a drop-in replacement for regular tuples:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="s1">'red'</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">my_car</span><span class="p">)</span> <span class="p">(</span><span class="s1">'red'</span><span class="p">,</span> <span class="mf">3812.4</span><span class="p">)</span> </pre></div> <p>You’ll even get a nice string representation for free, which saves some typing and redundancy:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span> <span class="n">Car</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="s1">'red'</span> <span class="p">,</span> <span class="n">mileage</span><span class="o">=</span><span class="mf">3812.4</span><span class="p">)</span> </pre></div> <p>Like tuples, namedtuples are immutable. When you try to overwrite one of their fields you’ll get an <code>AttributeError</code> exception:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span><span class="o">.</span><span class="n">color</span> <span class="o">=</span> <span class="s1">'blue'</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="s2">"can't set attribute"</span> </pre></div> <p>Namedtuple objects are implemented as regular Python classes internally. When it comes to memory usage they are also “better” than regular classes and just as memory efficient as regular tuples.</p> <p>A good way to view them is to think that <em>namedtuples are a memory-efficient shortcut to defining an immutable class in Python manually</em>.</p> <h3>Subclassing Namedtuples</h3> <p>Because they are built on top of regular classes you can even add methods to a namedtuple’s class. For example, you can extend the class like any other class and add methods and new properties to it that way. Here’s an example:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">Car</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span><span class="s1">'Car'</span><span class="p">,</span> <span class="s1">'color mileage'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">class</span> <span class="nc">MyCarWithMethods</span><span class="p">(</span><span class="n">Car</span><span class="p">):</span> <span class="o">...</span> <span class="k">def</span> <span class="nf">hexcolor</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="o">...</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">color</span> <span class="o">==</span> <span class="s1">'red'</span><span class="p">:</span> <span class="o">...</span> <span class="k">return</span> <span class="s1">'#ff0000'</span> <span class="o">...</span> <span class="k">else</span><span class="p">:</span> <span class="o">...</span> <span class="k">return</span> <span class="s1">'#000000'</span> </pre></div> <p>We can now create <code>MyCarWithMethods</code> objects and call their <code>hexcolor()</code> method, just as expected:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">c</span> <span class="o">=</span> <span class="n">MyCarWithMethods</span><span class="p">(</span><span class="s1">'red'</span><span class="p">,</span> <span class="mi">1234</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">c</span><span class="o">.</span><span class="n">hexcolor</span><span class="p">()</span> <span class="s1">'#ff0000'</span> </pre></div> <p>However, this might be a little clunky. It might be worth doing if you want a class with immutable properties. But it’s also easy to shoot yourself in the foot here.</p> <p>For example, adding a new <em>immutable</em> field is tricky because of how namedtuples are structured internally. The easiest way to create hierarchies of namedtuples is to use the base tuple’s <code>._fields</code> property:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">Car</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span><span class="s1">'Car'</span><span class="p">,</span> <span class="s1">'color mileage'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ElectricCar</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span> <span class="o">...</span> <span class="s1">'ElectricCar'</span><span class="p">,</span> <span class="n">Car</span><span class="o">.</span><span class="n">_fields</span> <span class="o">+</span> <span class="p">(</span><span class="s1">'charge'</span><span class="p">,))</span> </pre></div> <p>This gives the desired result:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">ElectricCar</span><span class="p">(</span><span class="s1">'red'</span><span class="p">,</span> <span class="mi">1234</span><span class="p">,</span> <span class="mf">45.0</span><span class="p">)</span> <span class="n">ElectricCar</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="s1">'red'</span><span class="p">,</span> <span class="n">mileage</span><span class="o">=</span><span class="mi">1234</span><span class="p">,</span> <span class="n">charge</span><span class="o">=</span><span class="mf">45.0</span><span class="p">)</span> </pre></div> <h3>Built-in Helper Methods</h3> <p>Besides the <code>_fields</code> property each namedtuple instance also provides a few more helper methods you might find useful. Their names all start with an underscore character (<code>_</code>) which usually signals that a method or property is “private” and not part of the stable public interface of a class or module.</p> <p>With namedtuples the underscore naming convention has a different meaning though: These helper methods and properties <em>are</em> part of namedtuple’s public interface. The helpers were named that way to avoid naming collisions with user-defined tuple fields. So go ahead and use them if you need them!</p> <p>I want to show you a few scenarios where the namedtuple helper methods might come in handy. Let’s start with the <code>_asdict()</code> helper. It returns the contents of a namedtuple as a dictionary:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span><span class="o">.</span><span class="n">_asdict</span><span class="p">()</span> <span class="n">OrderedDict</span><span class="p">([(</span><span class="s1">'color'</span><span class="p">,</span> <span class="s1">'red'</span><span class="p">),</span> <span class="p">(</span><span class="s1">'mileage'</span><span class="p">,</span> <span class="mf">3812.4</span><span class="p">)])</span> </pre></div> <p>This is great for avoiding typos when generating JSON-output, for example:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">my_car</span><span class="o">.</span><span class="n">_asdict</span><span class="p">())</span> <span class="s1">'{"color": "red", "mileage": 3812.4}'</span> </pre></div> <p>Another useful helper is the <code>_replace()</code> function. It creates a (shallow) copy of a tuple and allows you to selectively replace some of its fields:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span><span class="o">.</span><span class="n">_replace</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="s1">'blue'</span><span class="p">)</span> <span class="n">Car</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="s1">'blue'</span><span class="p">,</span> <span class="n">mileage</span><span class="o">=</span><span class="mf">3812.4</span><span class="p">)</span> </pre></div> <p>Lastly, the <code>_make()</code> classmethod can be used to create new instances of a namedtuple from a sequence or iterable:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">Car</span><span class="o">.</span><span class="n">_make</span><span class="p">([</span><span class="s1">'red'</span><span class="p">,</span> <span class="mi">999</span><span class="p">])</span> <span class="n">Car</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="s1">'red'</span><span class="p">,</span> <span class="n">mileage</span><span class="o">=</span><span class="mi">999</span><span class="p">)</span> </pre></div> <h3>When to Use Namedtuples</h3> <p>Namedtuples can be an easy way to clean up your code and to make it more readable by enforcing a better structure for your data.</p> <p>I find, for example, that going from ad-hoc data types like dictionaries with a fixed format to namedtuples helps me express my intentions more clearly. Often when I attempt this refactoring I magically come up with a better solution for the problem I’m facing.</p> <p>Using namedtuples over unstructured tuples and dicts can also make my coworkers’ lives easier because they make the data being passed around “self-documenting” (to a degree).</p> <p>On the other hand I try not to use namedtuples for their own sake if they don’t help me write “cleaner”, more readable, and more maintainable code. Too much of a good thing can be a bad thing.</p> <p>However if you use them with care, namedtuples can undoubtedly make your Python code better and more expressive.</p> <h3>Things to Remember</h3> <ul> <li><code>collection.namedtuple</code> is a memory-efficient shortcut to defining an immutable class in Python manually.</li> <li>Namedtuples can help clean up your code by enforcing an easier to understand structure on your data.</li> <li>Namedtuples provide a few useful helper methods that all start with an <code>_</code> underscore—but are part of the public interface. It’s okay to use them.</li> </ul></footer></article>https://dbader.org/blog/writing-clean-python-with-namedtuplesTue, 14 Feb 2017 00:00:00 GMTLambda Functions in Python: What Are They Good For?https://dbader.org/blog/python-lambda-functions<article><header><h1>Lambda Functions in Python: What Are They Good For?</h1> <p>An introduction to “lambda” expressions in Python: What they’re good for, when you should use them, and when it’s best to avoid them.</p> </header><section><figure><img alt="Lambda expressions in Python" src="/blog/figures/python-lambda-functions.png" width="1280" height="720"></figure> <p>The <code>lambda</code> keyword in Python provides a shortcut for declaring small anonymous functions. Lambda functions behave just like regular functions declared with the <code>def</code> keyword. They can be used whenever function objects are required.</p> <p>For example, this is how you’d define a simple lambda function carrying out an addition:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">add</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">add</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="mi">8</span> </pre></div> <p>You could declare the same <code>add</code> function with the <code>def</code> keyword:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span> <span class="o">...</span> <span class="k">return</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">add</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="mi">8</span> </pre></div> <p>Now you might be wondering: <strong>Why the big fuss about lambdas?</strong> If they’re just a slightly more terse version of declaring functions with <code>def</code>, what’s the big deal?</p> <p>Take a look at the following example and keep the words <em>function expression</em> in your head while you do that:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span><span class="p">)(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="mi">8</span> </pre></div> <p>Okay, what happened here? I just used <code>lambda</code> to define an “add” function inline and then immediately called it with the arguments <code>5</code> and <code>3</code>.</p> <p>Conceptually the <em>lambda expression</em> <code>lambda x, y: x + y</code> is the same as declaring a function with <code>def</code>, just written inline. The difference is I didn’t bind it to a name like <code>add</code> before I used it. I simply stated the expression I wanted to compute and then immediately evaluated it by calling it like a regular function.</p> <p>Before you move on, you might want to play with the previous code example a little to really let the meaning of it sink in. I still remember this took me a while to wrap my head around. So don’t worry about spending a few minutes in an interpreter session.</p> <p>There’s another syntactic difference between lambdas and regular function definitions: Lambda functions are restricted to a single expression. This means a lambda function can’t use statements or annotations—not even a <code>return</code> statement.</p> <p>How do you return values from lambdas then? Executing a lambda function evaluates its expression and then automatically returns its result. So there’s always an <em>implicit</em> return statement. That’s why some people refer to lambdas as <em>single expression functions</em>.</p> </section><footer><h3>Lambdas You Can Use</h3> <p>When should you use lambda functions in your code? Technically, any time you’re expected to supply a function object you can use a lambda expression. And because a lambda expression can be anonymous, you don’t even need to assign it to a name.</p> <p>This can provide a handy and “unbureaucratic” shortcut to defining a function in Python. My most frequent use case for lambdas is writing short and concise <em>key funcs</em> for <a href="python-min-max-and-nested-lists">sorting iterables by an alternate key</a>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">sorted</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="o">-</span><span class="mi">5</span><span class="p">,</span> <span class="mi">6</span><span class="p">),</span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="o">-</span><span class="mi">4</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="o">-</span><span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">]</span> </pre></div> <p>Like regular nested functions, lambdas also work as <em>lexical closures</em>.</p> <p>What’s a lexical closure? Just a fancy name for a function that remembers the values from the enclosing lexical scope even when the program flow is no longer in that scope. Here’s a (fairly academic) example to illustrate the idea:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">make_adder</span><span class="p">(</span><span class="n">n</span><span class="p">):</span> <span class="o">...</span> <span class="k">return</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span> <span class="o">+</span> <span class="n">n</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_3</span> <span class="o">=</span> <span class="n">make_adder</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_5</span> <span class="o">=</span> <span class="n">make_adder</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_3</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span> <span class="mi">7</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">plus_5</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span> <span class="mi">9</span> </pre></div> <p>In the above example the <code>x + n</code> lambda can still access the value of <code>n</code> even though it was defined in the <code>make_adder</code> function (the enclosing scope).</p> <p>Sometimes, using a lambda function instead of a nested function declared with <code>def</code> can express one’s intent more clearly. But to be honest this isn’t a common occurrence—at least in the kind of code that I like to write.</p> <h3>But Maybe You Shouldn’t…</h3> <p>Now on the one hand I’m hoping this article got you interested in exploring Python’s lambda functions. On the other hand I feel like it’s time to put up another caveat: Lambda functions should be used sparingly and with extraordinary care.</p> <p>I know I wrote my fair share of code using lambdas that looked “cool” but was actually a liability for me and my coworkers. If you’re tempted to use a lambda spend a few seconds (or minutes) to think if this is really the cleanest and most maintainable way to achieve the desired result.</p> <p>For example, doing something like this to save two lines of code is just silly. Sure, it technically works and it’s a nice enough “trick”. But it’s also going to confuse the next gal or guy having to ship a bugfix under a tight deadline:</p> <div class="codehilite"><pre><span></span><span class="c1"># Harmful:</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">class</span> <span class="nc">Car</span><span class="p">:</span> <span class="o">...</span> <span class="n">rev</span> <span class="o">=</span> <span class="k">lambda</span> <span class="bp">self</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s1">'Wroom!'</span><span class="p">)</span> <span class="o">...</span> <span class="n">crash</span> <span class="o">=</span> <span class="k">lambda</span> <span class="bp">self</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s1">'Boom!'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span> <span class="o">=</span> <span class="n">Car</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_car</span><span class="o">.</span><span class="n">crash</span><span class="p">()</span> <span class="s1">'Boom!'</span> </pre></div> <p>I feel similarly about complicated <code>map()</code> or <code>filter()</code> constructs using lambdas. Usually it’s much cleaner to go with a list comprehension or generator expression:</p> <div class="codehilite"><pre><span></span><span class="c1"># Harmful:</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">list</span><span class="p">(</span><span class="nb">filter</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">,</span> <span class="nb">range</span><span class="p">(</span><span class="mi">16</span><span class="p">)))</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">14</span><span class="p">]</span> <span class="c1"># Better:</span> <span class="o">&gt;&gt;&gt;</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span> <span class="k">if</span> <span class="n">x</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">]</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">14</span><span class="p">]</span> </pre></div> <p>If you find yourself doing anything remotely complex with a lambda expression, consider defining a real function with a proper name instead.</p> <p>Saving a few keystrokes won’t matter in the long run. Your colleagues (and your future self) will appreciate clean and readable code more than terse wizardry.</p> <h3>Things to Remember</h3> <ul> <li>Lambda functions are single-expression functions that are not necessarily bound to a name (anonymous).</li> <li>Lambda functions can’t use regular Python statements and always include an implicit <code>return</code> statement.</li> <li>Always ask yourself: <em>Would using a regular (named) function or a list/generator expression offer more clarity?</em></li> </ul></footer></article>https://dbader.org/blog/python-lambda-functionsTue, 07 Feb 2017 00:00:00 GMTWhy Learn Python? Here Are 8 Data-Driven Reasonshttps://dbader.org/blog/why-learn-python<article><header><h1>Why Learn Python? Here Are 8 Data-Driven Reasons</h1> <p>Is Python worth learning? We’ve interviewed experts and surveyed the job market to identify the key reasons why you should learn Python today.</p> </header><section><figure><img alt="Why Learn Python? Here Are 8 Data-Driven Reasons" src="/blog/figures/python-worth-learning-header.png" width="1280" height="720"></figure> <p>Python had a great year in 2016. The latest <a href="http://stackoverflow.com/research/developer-survey-2016" target="_blank">Stack Overflow Developer Survey</a> ranked Python as the 6th most popular and the 4th most wanted technology of the year.</p> <p>Python is also one of the hottest skills to have according to <a href="http://insights.dice.com/2016/02/01/whats-hot-and-not-in-tech-skills/" target="_blank">research by Dice</a>, and the 2nd most popular programming language in the world based on the <a href="http://pypl.github.io/PYPL.html" target="_blank">PYPL Popularity of Programming Language Index</a>.</p> <p>So why the hype? What makes Python so popular? Should you stop what you’re doing and start learning Python right now? I’ve searched far and wide to find out why Python is one of the world’s most loved and most used technologies. Without further ado, here’s why Python is worth learning in 2017 and the years ahead:</p> </section><section><h2>1. You Can Use Python for Pretty Much Anything</h2> <p>One significant advantage of learning Python is that it’s a general-purpose language that can be applied in a large variety of projects. Below are just some of the most common fields where Python has found its use:</p> <ul> <li>Data science</li> <li>Scientific and mathematical computing</li> <li>Web development</li> <li>Finance and trading</li> <li>System automation and administration</li> <li>Computer graphics</li> <li>Basic game development</li> <li>Security and penetration testing</li> <li>General and application-specific scripting</li> <li>Mapping and geography (GIS software)</li> </ul> <p>In preparation for this post, I posted the question <em>“Is Python worth learning?”</em> on Google+, Quora, and LinkedIn in order to collect some professional opinions on the matter. Here’s one of the responses I got that supports my point:</p> <blockquote> <p>“I had the opportunity to start learning Python 6 years ago. Since that time, I’ve used Python for everything from work related stuff to home automation tasks, and I have never stumbled upon a problem that can’t be solved with Python.”</p> </blockquote> <p>— <a href="https://www.linkedin.com/in/abensrhir" target="_blank">Anass Bensrhir</a>, Senior Data Scientist and Managing Director at <a href="http://www.bolddata.net/" target="_blank">Bold Data</a></p> </section><section><h2>2. Python Is Widely Used in Data Science</h2> <figure><img alt="Python developer job roles" src="/blog/figures/python-worth-learning-1.png" width="825" height="575"></figure> <p>(<a href="https://www.packtpub.com/skill-up-2016" target="_blank">Source</a>)</p> <p>Python’s application in data science and data engineering is what’s really fuelling its popularity today. Pandas, NumPy, SciPy, and other tools combined with the ability to prototype quickly and then “glue” systems together enable data engineers to maintain high efficiency when using Python.</p> <p><a href="https://plus.google.com/u/0/+JustinMcGrath" target="_blank">Justin McGrath</a>, a researcher at the University of Illinois, Champaign-Urbana agrees:</p> <blockquote> <p>“<strong>Python is probably going to become the de facto standard for scientific and statistical analyses</strong>. If you’re going into those fields, it’s certainly worth learning.”</p> </blockquote> </section><section><h2>3. Python Pays Well</h2> <p>It’s all well and good, but what about the pay, I hear you ask? It turns out <a href="https://www.daxx.com/article/python-developer-salary-usa" target="_blank">Python engineers have some of the highest salaries in the industry</a>, at least in the US.</p> <p>At nearly $103,500 per year, Python is the second best-paying programming language in the country (beating out Java, C++, and JavaScript) according to <a href="https://gooroo.io/" target="_blank">Gooroo</a>, a skill and salary analytics platform.</p> <figure><img alt="Average salary for Python developers" src="/blog/figures/python-worth-learning-2.png" width="591" height="369"></figure> <p><a href="http://www.indeed.com/salaries/Python-Developer-Salaries?from=salaries-search" target="_blank">Indeed’s salary calculator</a> gives an even larger figure—a whopping $116,000 per year. Of course, tech salaries differ greatly from one state to another. So to add some context, here’s a breakdown of how much Python engineers make in the states featured on Indeed:</p> <figure><img alt="Average Python developer salaries in the USA" src="/blog/figures/python-worth-learning-3.jpg" width="1240" height="711"></figure> <p>(<a href="figures/python-worth-learning-3.jpg" target="_blank">Click to view a larger version of the above image.</a>)</p> <figure><img alt="Python developer salaries in the USA (ranking)" src="/blog/figures/python-worth-learning-4.png" width="705" height="806"></figure> <p>(<a href="figures/python-worth-learning-4.png" target="_blank">Click to view a larger version of the above image.</a>)</p> </section><section><h2>4. Demand for Python Developers Is High (And Growing)</h2> <p>Based on <a href="https://www.indeed.com/jobtrends/q-python.html" target="_blank">Indeed’s job trends</a>, it looks like having Python under your belt can help you land a job in very short terms. The graph below displays a steady growth in the number of job postings featuring Python since 2012, and there has been a strong spike in popularity over the last six months.</p> <figure><img alt="Python developer job postings" src="/blog/figures/python-worth-learning-5.png" width="754" height="527"></figure> <p>What’s more, the demand for Python skills clearly outstrips jobseeker interest. The job market outlook for Python developers is excellent at the moment.</p> <figure><img alt="Python developer jobseeker interest" src="/blog/figures/python-worth-learning-6.png" width="754" height="527"></figure> </section><section><h2>5. Python Saves Time</h2> <p>I’m pretty sure that the majority of the developers who’ve used Python would agree that making anything with this language takes a lot less time <em>and</em> code than most other technologies.</p> <p>Even the classic “Hello, world” program illustrates this point:</p> <div class="codehilite"><pre><span></span><span class="k">print</span><span class="p">(</span><span class="s2">"Hello, world"</span><span class="p">)</span> </pre></div> <p>For comparison, this is what the same program looks like in Java:</p> <div class="codehilite"><pre><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">HelloWorld</span> <span class="o">{</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span> <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Hello, world"</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> </pre></div> </section><section><h2>6. Python Is Beginner Friendly</h2> <p>Following up from the previous point, Python’s famously clean and readable syntax makes it newbie-friendly. A well-written Python program can look like it came straight out of an algorithms text book. There’s little superfluous boilerplate, allowing beginners and experts alike to focus on the job at hand—instead of the code.</p> <p>Python’s efficiency and readability have also made it <a href="http://cacm.acm.org/blogs/blog-cacm/176450-python-is-now-the-most-popular-introductory-teaching-language-at-top-u-s-universities/fulltext" target="_blank">the number one most commonly taught introductory language at top US universities</a>. This will have ramifications on the future job market and likely make Python an even more popular technology choice.</p> </section><section><h2>7. All the Big Names Use Python</h2> <p>Ever wanted to work for a tech giant like Google or Facebook? Python could be your way in, as these companies, as well as YouTube, IBM, Yahoo, Dropbox, Quora, Mozilla, Instagram, and many others all use Python for a wide array of purposes, and are constantly hiring Python developers.</p> <figure><img alt="Programming Languages Company Size Breakdown" src="/blog/figures/python-worth-learning-7.png" width="715" height="580"></figure> <p>(<a href="https://www.packtpub.com/skill-up-2016" target="_blank">Source</a>)</p> <p>Dropbox’s code base, for instance, uses Python for almost everything, including analytics, the server backend, the API backend, and the desktop clients.</p> </section><section><h2>8. Python Has an Amazing Ecosystem</h2> <p>Last but not least, there’s a huge number of resources developed for Python that keep getting updated, including <a href="https://docs.python.org/3/library/" target="_blank">an impressive standard library</a> with built-in functionality, a built-in <a href="https://docs.python.org/2.7/library/unittest.html" target="_blank">unit testing framework</a>, and more than enough frameworks and environments that allow you to focus on writing the website or app at hand.</p> <p><a href="https://www.djangoproject.com/" target="_blank">Django</a> is the most commonly used Python web framework, but there’s also <a href="http://flask.pocoo.org/" target="_blank">Flask</a>, <a href="https://trypyramid.com" target="_blank">Pyramid</a>, <a href="http://www.web2py.com/" target="_blank">web2py</a>, <a href="https://pypi.python.org/pypi/Zope2" target="_blank">Zope 2</a>, and a few more.</p> </section><section><h2>What Do Python Community Leaders Think?</h2> <p>I thought it would also be a good idea to top things up with a few expert opinions on the advantages of Python as well as its future. It’s always a good idea to get a second (or third) opinion. So I reached out to several influencers and leaders in the Python space.</p> <p>I asked each of these experts three questions:</p> <ol> <li>What advantages does Python have over other programming languages?</li> <li>What future do you see for Python in 3–5 years?</li> <li>What will the job market look like for a Python developer in the coming years?</li> </ol> <p>Here’s what I was able to learn:</p> <div class="testimonial testimonial-box"> <p></p><div class="testimonial-attribution" style="margin-top: 0;"> <img src="/img/testimonial-mk.jpg" width="512" height="512"> <div> <span class="name">Michael Kennedy</span> <span class="job">Python Coach and Host of the Talk Python and Python Bytes Podcasts</span> </div> </div> <hr> <h3>“You start easy but you rarely outgrow Python like you do other easy to learn languages”</h3> <hr> <p><strong>What advantages does Python have over other programming languages?</strong></p> <p>I often think of programming languages as falling into two buckets.</p> <p>The first group would be the “With great power comes great responsibility” type of languages. This would be C, C++, and to a lesser degree C# and Java. The others are “I just need to ship something, don’t waste my time with minutia” languages. Visual Basic (pre-VB.NET) and JavaScript seem solidly in this camp, although JavaScript appears to be trying to escape with the massive decoupling seen in typical Node.js code and TypeScript.</p> <p>You choose C++ or C# if you need to really control the system and build large professional software. Is it mission critical enterprise software running the company with 100k lines of code? You might choose these. If you need a quick app to get the job done, like writing that “forms over data” app for something internal, VB 6 used to be a great answer for finishing that in a week, but coding yourself into a box if it grows too big or needs low level capabilities.</p> <p>Python is one of the few languages that is:</p> <ul> <li>Easy to learn</li> <li>Solves that “Don’t waste my time” set of problems well</li> <li>Yet, is also well designed with OOP and solid modern language features</li> <li>Can grow in power to match the powerful languages in capabilities</li> </ul> <p>In short, it’s one of the few languages that spans the spectrum of these capabilities. You start easy but you rarely outgrow Python like you do other easy to learn languages.</p> <p>We could also go into things like data science, scientific computing, web development, microcontrollers, things like Raspberry Pi, and how Python spans more technologies and areas of focus than most programming languages do.</p> <p>But the full spectrum aspect is the most powerful to me.</p> <p><strong>What future do you see for Python in 3–5 years?</strong></p> <p>In terms of predictions, I’m willing to make a few:</p> <ol> <li>Python will continue to expand into new areas of computing. It will be the primary IoT programming language.</li> <li>We will see Python interpreters/runtimes evolve and innovate. The YouTube team just released a <a href="https://talkpython.fm/episodes/show/95/grumpy-running-python-on-go" target="_blank">project</a> running Python on the Go runtime for example.</li> <li>The Python 3 vs Python 2 schism that has turned off countless new developers and generally been a cloud over community will be closed, and Python 3 will be just “Python”.</li> </ol> <p><strong>What will the job market look like for a Python developer in the coming years?</strong></p> <p>Given the growth numbers as well as the wide areas of computing that Python occupies, I think the job perspectives for Python developers are very solid.</p> <p>Some folks may feel Python is kind of a niche language or a small time scripting language. But very major applications are written in Python, including Dropbox and Youtube.</p> <p>Other areas outside web development where Python shines are places like the Large Hadron Collider where the team that found the Higgs Boson and won the Nobel Prize made heavy use of Python. Netflix uses Python to manage their AWS servers which cumulatively handle up to 35% of the bandwidth of the United States during the evenings.</p> <p>You’ll find that some locations in the world are more Python-centric than others. But there are many opportunities for Python developers.</p> <hr> <p><a href="https://twitter.com/mkennedy" target="_blank"><strong>Michael Kennedy</strong></a> is a Python coach and host of the popular <a href="https://talkpython.fm/" target="_blank">Talk Python</a> and <a href="https://pythonbytes.fm/" target="_blank">Python Bytes</a> podcasts.</p> </div> <div class="testimonial testimonial-box"> <p></p><div class="testimonial-attribution" style="margin-top: 0;"> <img src="/img/sublime-python/testimonial-ag.jpg" width="400" height="400"> <div> <span class="name">Ankur Gupta</span> <span class="job">Curator at ImportPython</span> </div> </div> <hr> <h3>“There is a demand-supply mismatch for Python developers with 2 to 6 years of experience”</h3> <hr> <p><strong>What advantages does Python have over other programming languages?</strong></p> <p>Python is an easier language to learn compared to, say, C++, C, C#, or Java, but that’s not it. We often tend to credit syntax, core team, feature roadmap, etc for the success of a certain language.</p> <p>They’re beyond doubt important, but when it comes to Python, it’s the global, diverse, and vibrant community that make it so widely adopted. Initiatives like <a href="https://djangogirls.org" target="_blank">Django Girls</a> and the scale at which they operate are unique. There are at least three dozen free books on Python, thousands of free videos to learn from, as well as the PyCon events all around the world.</p> <p>Active local and online regional Python communities are the biggest advantage that Python has over other languages. It’s the people behind the language that make it special.</p> <p><strong>What future do you see for Python in 3–5 years?</strong></p> <p>10 years ago, mentioning Python was guaranteed to invite blank stares. But today, Python is a pretty mainstream language. I think Python is here to stay.</p> <p>In 3–5 years I foresee:</p> <ul> <li>2.x codebase becoming a minority</li> <li>Python developers being available in abundance thanks to schools and colleges that teach Python as an introductory language</li> <li>People using different Python runtime interpreters instead of just CPython</li> </ul> <p><strong>What will the job market look like for a Python developer in the coming years?</strong></p> <p>Back in 2007–2008, I’d get no more than 3–4 calls a month concerning Python job openings, and most of those calls had to do with Python scripting for test automation (India). But if I were to look for a job today, I’m sure my phone would ring multiple times per day.</p> <p>There is a demand-supply mismatch for Python developers with 2 to 6 years of experience because of all these companies wanting to use Python for data science, data processing, machine learning, web application development, and so on.</p> <p>This situation will be gradually improving over the next couple of years, which means today is definitely the best time to be a Python developer.</p> <hr> <p><a href="https://twitter.com/originalankur" target="_blank"><strong>Ankur Gupta</strong></a> is the curator of the <a href="http://importpython.com/newsletter/" target="_blank">weekly newsletter over at ImportPython.com</a>, which keeps you updated on everything happening in the world of Python programming.</p> </div> <div class="testimonial testimonial-box"> <p></p><div class="testimonial-attribution" style="margin-top: 0;"> <img src="/img/testimonial-sv.jpg" width="460" height="460"> <div> <span class="name">Sebastian Vetter</span> <span class="job">Python Engineer at Eventbase, PyCon Speaker and Meetup Host</span> </div> </div> <hr> <h3>“The community around Python is the most welcoming and inclusive one out of all those that I’ve experienced”</h3> <hr> <p><strong>What advantages does Python have over other programming languages?</strong></p> <ul> <li><strong>Community.</strong> The community around Python is the most welcoming and inclusive one out of all those that I’ve experienced. Many times I’ve been inspired by the progressive effort at meetups and conferences to be inclusive to newcomers, underrepresented groups and minorities.</li> <li><strong>Readability.</strong> A lot of effort has gone into developing Python as a language that has readability as one of its main features, rather than considering it as an afterthought. As Robert C. Martin wrote in <em>Clean Code</em>, “the ratio of time spent reading versus writing is well over 10 to 1.”</li> <li><strong>Consistency.</strong> One of the things that I’ve always loved about Python is the fact that it uses whitespace to determine blocks instead of using various types of brackets. Although this is a little unintuitive when starting out, in my opinion, the advantage is that it ensures that Python code is relatively similar across different projects. It improves consistency and readability.</li> </ul> <p><strong>What future do you see for Python in 3–5 years?</strong></p> <p>In my opinion, the use of Python and the number of developers working with it will grow significantly in scientific fields. The number of science-related topics at Python conferences (and beyond) and releases of new tools to help the scientific community will make it easier to adopt the language. This will give the scientific community access to a very inclusive and welcoming developer community that will help improve the quality of development and simplify the tooling for scientific and research-related applications.</p> <p>The mobile space is going to be very interesting in about 3–5 years. As Russell Keith-Magee pointed out in his presentation “Python on the Move: the State of Mobile Python” at PyCon AU 2015, the future of Python as a language will most likely depend in part on how the community moves into the mobile development space. Although the Python community is very diverse and the language is used in a lot of different fields, we currently don’t have any decent support for mobile platforms. Looking at Russell’s efforts to bridge this gap with his project under the BeeWare umbrella, I’m confident that this gap will be closed within the next few years, and we’ll be able to maintain a strong position even in these new areas.</p> <p>Over the last several years, there’s been a lot of disagreement over Python 3 and whether it’s a step in the right direction. I do understand some of the critical arguments made against Python 3. Several highly qualified Pythonistas with vastly more experience than myself have raised valid concerns and pointed out flaws. Regardless of these concerns, I’m convinced that the adoption of Python 3 will pick up steam over the next two or three years, moving faster towards it being the mainstream version. This is indicated by projects like Django dropping support for Python 2.7 within 2017 with their release of Django 2.0 and the broader adoption of asyncio and coroutine-based frameworks and libraries.</p> <p>Making the Python community a more inclusive space for individuals of underrepresented groups such as women and other minorities will help us build a community made up of all different types of people. I’m sure that over the next 5 years, we’ll see the first major benefits of these initiatives contributing to a much stronger community. Making everyone welcome and embracing the differences in perspectives and experiences will serve as a model for companies, proving that such an environment results in better software and happier employees. I also think that individuals from within the Python community who’ve experienced this atmosphere will impact their employers by demanding a similar environment in their professional lives, drawing from the support of the community.</p> <p><strong>What will the job market look like for a Python developer in the coming years?</strong></p> <p>The next few years will most likely see a much more diverse landscape of Python jobs. With the increased application of Python in scientific fields, more research positions will become available. In addition, I think the growing need of programming skills within the scientific community will lead to having a combination of researchers and programmers to produce a skilled workforce that is capable in the scientific aspect as well as development best practises and tooling.</p> <p>The position of Data Scientist is going to become more and more important in the tech industry and will therefore increase the demand within the Python community specifically. We already have a large number of scientists use Python as their main language for their research in our community. Their skills in statistics and the use of the language will make them prime candidates for positions that are related to data-driven systems. With the demand for such systems growing fast, there will be a high demand for these individuals, and anybody within the Python community willing to level up on either the development aspects or the scientific skills.</p> <p>The Python community is strongly committed to improving its inclusiveness and diversity. Mandating and enforcing codes of conduct at conferences and meetups as well as openly stating the inclusive nature of communities around projects like the Django framework are helping to improve the representation of underprivileged individuals within the community. I hope and believe that this will, over the next few years, help make the community a place that will thrive, because individuals from these underrepresented groups will feel safe and welcome. This will make the Python community an exceptional pool to tap into for companies that are making an effort to improve the diversity of their development and science teams.</p> <hr> <p><a href="http://www.roadsi.de" target="_blank"><strong>Sebastian Vetter</strong></a> is a Senior Python Engineer at <a href="https://www.eventbase.com/" target="_blank">Eventbase</a>, PyCon speaker and Python meet-up host.</p> </div></section><footer></footer></article>https://dbader.org/blog/why-learn-pythonTue, 31 Jan 2017 00:00:00 GMTThe 4 Major Ways to Do String Formatting in Pythonhttps://dbader.org/blog/python-string-formatting<article><header><h1>The 4 Major Ways to Do String Formatting in Python</h1> <p>Remember the Zen of Python and how there should be “one obvious way to do something in Python”? You might scratch your head when you find out that there are *four* major ways to do string formatting in Python.</p> </header><section><figure><img alt="String Formatting in Python (updated for Python 3.6 and above)" src="/blog/figures/python-string-formatting-36.png" width="1280" height="720"></figure> <p>In this article I’ll demonstrate how these four string formatting approaches work and what their respective strengths and weaknesses are. I’ll also give you my simple “rule of thumb” for how I pick the best general purpose string formatting approach.</p> <p>Let’s jump right in, as we’ve got a lot to cover. In order to have a simple toy example for experimentation, let’s assume we’ve got the following variables (or constants, really) to work with:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">errno</span> <span class="o">=</span> <span class="mi">50159747054</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">'Bob'</span> </pre></div> <p>And based on these variables we’d like to generate an output string containing a simple error message:</p> <div class="codehilite"><pre><span></span><span class="s1">'Hey Bob, there is a 0xbadc0ffee error!'</span> </pre></div> <p>Hey… now <em>that</em> error could really spoil a dev’s Monday morning. But we’re here to discuss string formatting. So let’s get to work.</p> </section><section><h2><a class="anchor" name="old-style"></a>#1 – “Old Style” String Formatting (%-operator)</h2> <p>Strings in Python have a unique built-in operation that can be accessed with the <code>%</code>-operator. This lets you do simple positional formatting very easily. If you’ve ever worked with a <code>printf</code>-style function in C you’ll recognize how this works instantly. Here’s a simple example:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="s1">'Hello, </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">name</span> <span class="s2">"Hello, Bob"</span> </pre></div> <p>I’m using the <code>%s</code> format specifier here to tell Python where to substitute the value of <code>name</code>, represented as a string.</p> <p>There are other format specifiers available that let you control the output format. For example it’s possible to convert numbers to hexadecimal notation or to add whitespace padding to generate nicely formatted tables and reports (<a href="https://docs.python.org/3/library/stdtypes.html#old-string-formatting" target="_blank">cf. Python Docs: “printf-style String Formatting”</a>).</p> <p>Here, we can use the <code>%x</code> format specifier to convert an int value to a string and to represent it as a hexadecimal number:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="s1">'</span><span class="si">%x</span><span class="s1">'</span> <span class="o">%</span> <span class="n">errno</span> <span class="s1">'badc0ffee'</span> </pre></div> <p>The “old style” string formatting syntax changes slightly if you want to make multiple substitutions in a single string. Because the <code>%</code>-operator only takes one argument you need to wrap the right-hand side in a tuple, like so:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="s1">'Hey </span><span class="si">%s</span><span class="s1">, there is a 0x</span><span class="si">%x</span><span class="s1"> error!'</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">errno</span><span class="p">)</span> <span class="s1">'Hey Bob, there is a 0xbadc0ffee error!'</span> </pre></div> <p>It’s also possible to refer to variable substitutions by name in your format string, if you pass a mapping to the <code>%</code>-operator:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="s1">'Hey </span><span class="si">%(name)s</span><span class="s1">, there is a 0x</span><span class="si">%(errno)x</span><span class="s1"> error!'</span> <span class="o">%</span> <span class="p">{</span> <span class="o">...</span> <span class="s2">"name"</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span> <span class="s2">"errno"</span><span class="p">:</span> <span class="n">errno</span> <span class="p">}</span> <span class="s1">'Hey Bob, there is a 0xbadc0ffee error!'</span> </pre></div> <p>This makes your format strings easier to maintain and easier to modify in the future. You don’t have to worry about making sure the order you’re passing in the values matches up with the order the values are referenced in the format string. Of course the downside is that this technique requires a little more typing.</p> <p>I’m sure you’ve been wondering why this <code>printf</code>-style formatting is called “old style” string formatting. It was technically superseded by “new style” formatting, which we’re going to talk about in a minute.</p> </section><section><h2><a class="anchor" name="new-style"></a>#2 – “New Style” String Formatting (str.format)</h2> <p>Python 3 introduced a new way to do string formatting that was also later back-ported to Python 2.7. This “new style” string formatting gets rid of the <code>%</code>-operator special syntax and makes the syntax for string formatting more regular. Formatting is now handled by calling a <code>format()</code> function on a string object (<a href="https://docs.python.org/3/library/stdtypes.html#str.format" target="_blank">cf. Python Docs: “str.format”</a>).</p> <p>You can use the <code>format()</code> function to do simple positional formatting, just like you could with “old style” formatting:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="s1">'Hello, {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="s1">'Hello, Bob'</span> </pre></div> <p>Or, you can refer to your variable substitutions by name and use them in any order you want. This is quite a powerful feature as it allows for re-arranging the order of display without changing the arguments passed to the format function:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="s1">'Hey {name}, there is a 0x{errno:x} error!'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span> <span class="o">...</span> <span class="n">name</span><span class="o">=</span><span class="n">name</span><span class="p">,</span> <span class="n">errno</span><span class="o">=</span><span class="n">errno</span><span class="p">)</span> <span class="s1">'Hey Bob, there is a 0xbadc0ffee error!'</span> </pre></div> <p>This also shows that the syntax to format an int variable as a hexadecimal string has changed. Now we need to pass a <em>format spec</em> by adding a <code>:x</code> suffix. The format string syntax has become more powerful without complicating the simpler use cases. It pays off to read up on this <em>string formatting mini-language</em> in the Python documentation (<a href="https://docs.python.org/3/library/string.html#string-formatting" target="_blank">cf. Python Docs: “Format String Syntax”</a>).</p> <p>In Python 3, this “new style” string formatting is to be preferred over <code>%</code>-style formatting. While <a href="https://docs.python.org/3/library/stdtypes.html?highlight=sprintf#printf-style-string-formatting" target="_blank">“old style” formatting has been de-emphasized</a> it has <em>not</em> been deprecated. It is still supported in the latest versions of Python. According to <a href="https://lists.gt.net/python/dev/969817" target="_blank">this discussion on the Python dev email list</a> and <a href="https://bugs.python.org/issue14123" target="_blank">this issue on the Python dev bug tracker</a>, <code>%</code>-formatting is going to stick around for a long time to come.</p> <p>Still, the official Python 3.X documentation doesn’t exactly recommend “old style” formatting or speak too fondly of it:</p> <blockquote> <p>The formatting operations described here exhibit a variety of quirks that lead to a number of common errors (such as failing to display tuples and dictionaries correctly). Using the newer formatted string literals or the str.format() interface helps avoid these errors. These alternatives also provide more powerful, flexible and extensible approaches to formatting text. (Source: <a href="https://docs.python.org/3/library/stdtypes.html?highlight=sprintf#printf-style-string-formatting" target="_blank">Python 3 Docs</a>)</p> </blockquote> <p>This is why I’d personally try to stick with <code>str.format</code> for new code moving forward. Starting with Python 3.6 there’s yet another way to format your strings. I’ll tell you all about it in the next section.</p> </section><section><h2><a class="anchor" name="interpolation"></a>#3 – Literal String Interpolation (Python 3.6+)</h2> <p><a href="cool-new-features-in-python-3-6">Python 3.6 adds a new string formatting approach</a> called <em>Formatted String Literals</em>. This new way of formatting strings lets you use embedded Python expressions inside string constants. Here’s a simple example to give you a feel for the feature:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">f</span><span class="s1">'Hello, {name}!'</span> <span class="s1">'Hello, Bob!'</span> </pre></div> <p>This new formatting syntax is powerful. Because you can embed arbitrary Python expressions you can even do inline arithmetic with it. See here for example:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">5</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">b</span> <span class="o">=</span> <span class="mi">10</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">f</span><span class="s1">'Five plus ten is {a + b} and not {2 * (a + b)}.'</span> <span class="s1">'Five plus ten is 15 and not 30.'</span> </pre></div> <p>Formatted string literals are a Python parser feature that converts f-strings into a series of string constants and expressions. They then get joined up to build the final string.</p> <p>Imagine we had the following <code>greet()</code> function that contains an f-string:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">greet</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">question</span><span class="p">):</span> <span class="o">...</span> <span class="k">return</span> <span class="n">f</span><span class="s2">"Hello, {name}! How's it {question}?"</span> <span class="o">...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greet</span><span class="p">(</span><span class="s1">'Bob'</span><span class="p">,</span> <span class="s1">'going'</span><span class="p">)</span> <span class="s2">"Hello, Bob! How's it going?"</span> </pre></div> <p>When we disassemble the function and inspect what’s going on behind the scenes we can see that the f-string in the function gets transformed into something similar to the following:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">greet</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">question</span><span class="p">):</span> <span class="o">...</span> <span class="k">return</span> <span class="s2">"Hello, "</span> <span class="o">+</span> <span class="n">name</span> <span class="o">+</span> <span class="s2">"! How's it "</span> <span class="o">+</span> <span class="n">question</span> <span class="o">+</span> <span class="s2">"?"</span> </pre></div> <p>The real implementation is slightly faster than that because it uses the <a href="https://bugs.python.org/issue27078" target="_blank"><code>BUILD_STRING</code> opcode as an optimization</a>. But functionally they’re the same:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">dis</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">dis</span><span class="o">.</span><span class="n">dis</span><span class="p">(</span><span class="n">greet</span><span class="p">)</span> <span class="mi">2</span> <span class="mi">0</span> <span class="n">LOAD_CONST</span> <span class="mi">1</span> <span class="p">(</span><span class="s1">'Hello, '</span><span class="p">)</span> <span class="mi">2</span> <span class="n">LOAD_FAST</span> <span class="mi">0</span> <span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="mi">4</span> <span class="n">FORMAT_VALUE</span> <span class="mi">0</span> <span class="mi">6</span> <span class="n">LOAD_CONST</span> <span class="mi">2</span> <span class="p">(</span><span class="s2">"! How's it "</span><span class="p">)</span> <span class="mi">8</span> <span class="n">LOAD_FAST</span> <span class="mi">1</span> <span class="p">(</span><span class="n">question</span><span class="p">)</span> <span class="mi">10</span> <span class="n">FORMAT_VALUE</span> <span class="mi">0</span> <span class="mi">12</span> <span class="n">LOAD_CONST</span> <span class="mi">3</span> <span class="p">(</span><span class="s1">'?'</span><span class="p">)</span> <span class="mi">14</span> <span class="n">BUILD_STRING</span> <span class="mi">5</span> <span class="mi">16</span> <span class="n">RETURN_VALUE</span> </pre></div> <p>String literals also support the existing format string syntax of the <code>str.format()</code> method. That allows you to solve the same formatting problems we’ve discussed in the previous two sections:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">f</span><span class="s2">"Hey {name}, there's a {errno:#x} error!"</span> <span class="s2">"Hey Bob, there's a 0xbadc0ffee error!"</span> </pre></div> <p>Python’s new Formatted String Literals are similar to the JavaScript Template Literals added in ES2015. I think they’re quite a nice addition to the language and I’ve already started using them in my day to day (Python 3) work. You can learn more about Formatted String Literals in the official Python documentation (<a href="https://docs.python.org/3/reference/lexical_analysis.html#f-strings" target="_blank">cf. Python Docs: “Formatted string literals”</a>).</p> </section><section><h2><a class="anchor" name="template-strings"></a>#4 – Template Strings (standard library)</h2> <p>Here’s one more technique for string formatting in Python: Template Strings. It’s a simpler and less powerful mechanism, but in some cases this might be exactly what you’re looking for.</p> <p>Let’s take a look at a simple greeting example:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">string</span> <span class="kn">import</span> <span class="n">Template</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">t</span> <span class="o">=</span> <span class="n">Template</span><span class="p">(</span><span class="s1">'Hey, $name!'</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">t</span><span class="o">.</span><span class="n">substitute</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">name</span><span class="p">)</span> <span class="s1">'Hey, Bob!'</span> </pre></div> <p>You see here that we need to import the <code>Template</code> class from Python’s built-in <code>string</code> module. Template strings are not a core language feature but they’re supplied by a module in the standard library.</p> <p>Another difference is that template strings don’t allow format specifiers. So in order to get our error string example to work we need to transform our int error number into a hex-string ourselves:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">templ_string</span> <span class="o">=</span> <span class="s1">'Hey $name, there is a $error error!'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">Template</span><span class="p">(</span><span class="n">templ_string</span><span class="p">)</span><span class="o">.</span><span class="n">substitute</span><span class="p">(</span> <span class="o">...</span> <span class="n">name</span><span class="o">=</span><span class="n">name</span><span class="p">,</span> <span class="n">error</span><span class="o">=</span><span class="nb">hex</span><span class="p">(</span><span class="n">errno</span><span class="p">))</span> <span class="s1">'Hey Bob, there is a 0xbadc0ffee error!'</span> </pre></div> <p>That worked great. So when should you use template strings in your Python programs? In my opinion the best use case for template strings is when you’re handling format strings generated by users of your program. Due to their reduced complexity template strings are a safer choice.</p> <p>The more complex formatting mini-languages of the other string formatting techniques might introduce security vulnerabilities to your programs. For example, it’s <a href="http://lucumr.pocoo.org/2016/12/29/careful-with-str-format/" target="_blank">possible for format strings to access arbitrary variables in your program</a>.</p> <p>That means, if a malicious user can supply a format string they can potentially leak secret keys and other sensible information! Here’s a simple proof of concept of how this attack might be used:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">SECRET</span> <span class="o">=</span> <span class="s1">'this-is-a-secret'</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">class</span> <span class="nc">Error</span><span class="p">:</span> <span class="o">...</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="o">...</span> <span class="k">pass</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">err</span> <span class="o">=</span> <span class="n">Error</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">user_input</span> <span class="o">=</span> <span class="s1">'{error.__init__.__globals__[SECRET]}'</span> <span class="c1"># Uh-oh...</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">user_input</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">error</span><span class="o">=</span><span class="n">err</span><span class="p">)</span> <span class="s1">'this-is-a-secret'</span> </pre></div> <p>See how a hypothetical attacker was able to extract our secret string by accessing the <code>__globals__</code> dictionary? Scary, huh? Template Strings close this attack vector. And this makes them a safer choice if you’re handling format strings generated from user input:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">user_input</span> <span class="o">=</span> <span class="s1">'${error.__init__.__globals__[SECRET]}'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">Template</span><span class="p">(</span><span class="n">user_input</span><span class="p">)</span><span class="o">.</span><span class="n">substitute</span><span class="p">(</span><span class="n">error</span><span class="o">=</span><span class="n">err</span><span class="p">)</span> <span class="ne">ValueError</span><span class="p">:</span> <span class="s2">"Invalid placeholder in string: line 1, col 1"</span> </pre></div> </section><section><h2>Which String Formatting Method Should I Use?</h2> <p>I totally get that having so much choice for how to format your strings in Python can feel very confusing. This is an excellent cue to bust out this handy flowchart infographic I’ve put together for you:</p> <figure><img alt="String Formatting in Python -- Flowchart" src="/blog/figures/python-string-formatting-flowchart.png" width="706" height="989"></figure> <p>This flowchart is based on the following rule of thumb that I apply when I’m writing Python:</p> </section><footer><h3>Dan’s Python String Formatting Rule of Thumb:</h3> <blockquote> <p><em>If your format strings are user-supplied, use <a href="#template-strings">Template Strings (#4)</a> to avoid security issues. Otherwise, use <a href="#interpolation">Literal String Interpolation (#3)</a> if you’re on Python 3.6+, and <a href="#new-style">“New Style” str.format (#2)</a> if you’re not.</em></p> </blockquote></footer></article>https://dbader.org/blog/python-string-formattingTue, 24 Jan 2017 00:00:00 GMTAssert Statements in Pythonhttps://dbader.org/blog/python-assert-tutorial<article><header><h1>Assert Statements in Python</h1> <p>How to use assertions to help automatically detect errors in your Python programs in order to make them more reliable and easier to debug.</p> </header><section><figure><img alt="" src="/blog/figures/python-assert.png" width="1280" height="720"></figure> </section><section><h2>What Are Assertions &amp; What Are They Good For?</h2> <p>Python’s assert statement is a debugging aid that tests a condition. If the condition is true, it does nothing and your program just continues to execute. But if the assert condition evaluates to false, it raises an <code>AssertionError</code> exception with an optional error message.</p> <p>The proper use of assertions is to inform developers about <strong>unrecoverable</strong> errors in a program. They’re not intended to signal expected error conditions, like “file not found”, where a user can take corrective action or just try again.</p> <p>Another way to look at it is to say that assertions are <strong>internal self-checks</strong> for your program. They work by declaring some conditions as <em>impossible</em> in your code. If one of these conditions doesn’t hold that means there’s a bug in the program.</p> <p>If your program is bug-free, these conditions will never occur. But if they <em>do</em> occur the program will crash with an assertion error telling you exactly which “impossible” condition was triggered. This makes it much easier to track down and fix bugs in your programs.</p> <p><strong>To summarize</strong>: Python’s assert statement is a debugging aid, not a mechanism for handling run-time errors. The goal of using assertions is to let developers find the likely root cause of a bug more quickly. An assertion error should never be raised unless there’s a bug in your program.</p> </section><section><h2>Assert in Python — An Example</h2> <p>Here’s a simple example so you can see where assertions might come in handy. I tried to give this some semblance of a real world problem you might actually encounter in one of your programs.</p> <p>Suppose you were building an online store with Python. You’re working to add a discount coupon functionality to the system and eventually write the following <code>apply_discount</code> function:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">apply_discount</span><span class="p">(</span><span class="n">product</span><span class="p">,</span> <span class="n">discount</span><span class="p">):</span> <span class="n">price</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">product</span><span class="p">[</span><span class="s1">'price'</span><span class="p">]</span> <span class="o">*</span> <span class="p">(</span><span class="mf">1.0</span> <span class="o">-</span> <span class="n">discount</span><span class="p">))</span> <span class="k">assert</span> <span class="mi">0</span> <span class="o">&lt;=</span> <span class="n">price</span> <span class="o">&lt;=</span> <span class="n">product</span><span class="p">[</span><span class="s1">'price'</span><span class="p">]</span> <span class="k">return</span> <span class="n">price</span> </pre></div> <p>Notice the <code>assert</code> statement in there? It will guarantee that, no matter what, discounted prices cannot be lower than $0 and they cannot be higher than the original price of the product.</p> <p>Let’s make sure this actually works as intended if we call this function to apply a valid discount:</p> <div class="codehilite"><pre><span></span><span class="c1">#</span> <span class="c1"># Our example product: Nice shoes for $149.00</span> <span class="c1">#</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">shoes</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'name'</span><span class="p">:</span> <span class="s1">'Fancy Shoes'</span><span class="p">,</span> <span class="s1">'price'</span><span class="p">:</span> <span class="mi">14900</span><span class="p">}</span> <span class="c1">#</span> <span class="c1"># 25% off -&gt; $111.75</span> <span class="c1">#</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">apply_discount</span><span class="p">(</span><span class="n">shoes</span><span class="p">,</span> <span class="mf">0.25</span><span class="p">)</span> <span class="mi">11175</span> </pre></div> <p>Alright, this worked nicely. Now, let’s try to apply some <em>invalid</em> discounts:</p> <div class="codehilite"><pre><span></span><span class="c1">#</span> <span class="c1"># A "200% off" discount:</span> <span class="c1">#</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">apply_discount</span><span class="p">(</span><span class="n">shoes</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">)</span> <span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span> <span class="n">File</span> <span class="s2">"&lt;input&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o">&lt;</span><span class="n">module</span><span class="o">&gt;</span> <span class="n">apply_discount</span><span class="p">(</span><span class="n">prod</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">)</span> <span class="n">File</span> <span class="s2">"&lt;input&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">4</span><span class="p">,</span> <span class="ow">in</span> <span class="n">apply_discount</span> <span class="k">assert</span> <span class="mi">0</span> <span class="o">&lt;=</span> <span class="n">price</span> <span class="o">&lt;=</span> <span class="n">product</span><span class="p">[</span><span class="s1">'price'</span><span class="p">]</span> <span class="ne">AssertionError</span> <span class="c1">#</span> <span class="c1"># A "-30% off" discount:</span> <span class="c1">#</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">apply_discount</span><span class="p">(</span><span class="n">shoes</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.3</span><span class="p">)</span> <span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span> <span class="n">File</span> <span class="s2">"&lt;input&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o">&lt;</span><span class="n">module</span><span class="o">&gt;</span> <span class="n">apply_discount</span><span class="p">(</span><span class="n">prod</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.3</span><span class="p">)</span> <span class="n">File</span> <span class="s2">"&lt;input&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">4</span><span class="p">,</span> <span class="ow">in</span> <span class="n">apply_discount</span> <span class="k">assert</span> <span class="mi">0</span> <span class="o">&lt;=</span> <span class="n">price</span> <span class="o">&lt;=</span> <span class="n">product</span><span class="p">[</span><span class="s1">'price'</span><span class="p">]</span> <span class="ne">AssertionError</span> </pre></div> <p>As you can see, trying to apply an invalid discount raises an <code>AssertionError</code> exception that points out the line with the violated assertion condition. If we ever encounter one of these errors while testing our online store it will be easy to find out what happened by looking at the traceback.</p> <p>This is the power of assertions, in a nutshell.</p> </section><section><h2>Python’s Assert Syntax</h2> <p>It’s always a good idea to study up on how a language feature is actually implemented in Python before you start using it. So let’s take a quick look at the <a href="https://docs.python.org/3/reference/simple_stmts.html#the-assert-statement" target="_blank">syntax for the assert statement according to the Python docs</a>:</p> <div class="codehilite"><pre><span></span><span class="n">assert_stmt</span> <span class="p">::</span><span class="o">=</span> <span class="s2">"assert"</span> <span class="n">expression1</span> <span class="p">[</span><span class="s2">","</span> <span class="n">expression2</span><span class="p">]</span> </pre></div> <p>In this case <code>expression1</code> is the condition we test, and the optional <code>expression2</code> is an error message that’s displayed if the assertion fails.</p> <p>At execution time, the Python interpreter transforms each assert statement into roughly the following:</p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="n">__debug__</span><span class="p">:</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">expression1</span><span class="p">:</span> <span class="k">raise</span> <span class="ne">AssertionError</span><span class="p">(</span><span class="n">expression2</span><span class="p">)</span> </pre></div> <p>You can use <code>expression2</code> to pass an <strong>optional error message</strong> that will be displayed with the <code>AssertionError</code> in the traceback. This can simplify debugging even further—for example, I’ve seen code like this:</p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="n">cond</span> <span class="o">==</span> <span class="s1">'x'</span><span class="p">:</span> <span class="n">do_x</span><span class="p">()</span> <span class="k">elif</span> <span class="n">cond</span> <span class="o">==</span> <span class="s1">'y'</span><span class="p">:</span> <span class="n">do_y</span><span class="p">()</span> <span class="k">else</span><span class="p">:</span> <span class="k">assert</span> <span class="bp">False</span><span class="p">,</span> <span class="p">(</span><span class="s2">"This should never happen, but it does occasionally. "</span> <span class="s2">"We're currently trying to figure out why. "</span> <span class="s2">"Email dbader if you encounter this in the wild."</span><span class="p">)</span> </pre></div> <p>Is this ugly? Well, yes. But it’s definitely a valid and helpful technique if you’re faced with a <a href="https://en.wikipedia.org/wiki/Heisenbug" target="_blank">heisenbug-type issue</a> in one of your applications. 😉</p> </section><section><h2>Common Pitfalls With Using Asserts in Python</h2> <p>Before you move on, there are two important caveats with using assertions in Python that I’d like to call out.</p> <p>The first one has to do with introducing security risks and bugs into your applications, and the second one is about a syntax quirk that makes it easy to write <em>useless</em> assertions.</p> <p>This sounds (and potentially is) pretty horrible, so you might at least want to skim these two caveats or read their summaries below.</p> </section><section><h2>Caveat #1 – Don’t Use Asserts for Data Validation</h2> <p><strong>Asserts can be turned off globally in the Python interpreter. Don’t rely on assert expressions to be executed for data validation or data processing.</strong></p> <p>The biggest caveat with using asserts in Python is that <em><a href="https://docs.python.org/3/library/constants.html#__debug__" target="_blank">assertions can be globally disabled</a> with the <code>-O</code> and <code>-OO</code> command line switches, as well as the <code>PYTHONOPTIMIZE</code> environment variable in CPython</em>.</p> <p>This turns any assert statement into a null-operation: the assertions simply get compiled away and won’t be evaluated, which means that none of the conditional expressions will be executed.</p> <p>This is an intentional design decision used similarly by many other programming languages. As a side-effect it becomes extremely dangerous to use assert statements as a quick and easy way to validate input data.</p> <p>Let me explain—if your program uses asserts to check if a function argument contains a “wrong” or unexpected value this can backfire quickly and lead to bugs or security holes.</p> <p>Let’s take a look at a simple example. Imagine you’re building an online store application with Python. Somewhere in your application code there’s a function to delete a product as per a user’s request:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">delete_product</span><span class="p">(</span><span class="n">product_id</span><span class="p">,</span> <span class="n">user</span><span class="p">):</span> <span class="k">assert</span> <span class="n">user</span><span class="o">.</span><span class="n">is_admin</span><span class="p">(),</span> <span class="s1">'Must have admin privileges to delete'</span> <span class="k">assert</span> <span class="n">store</span><span class="o">.</span><span class="n">product_exists</span><span class="p">(</span><span class="n">product_id</span><span class="p">),</span> <span class="s1">'Unknown product id'</span> <span class="n">store</span><span class="o">.</span><span class="n">find_product</span><span class="p">(</span><span class="n">product_id</span><span class="p">)</span><span class="o">.</span><span class="n">delete</span><span class="p">()</span> </pre></div> <p>Take a close look at this function. What happens if assertions are disabled?</p> <p>There are two serious issues in this three-line function example, caused by the incorrect use of assert statements:</p> <ol> <li><strong>Checking for admin privileges with an assert statement is dangerous.</strong> If assertions are disabled in the Python interpreter, this turns into a null-op. Therefore <em>any user can now delete products</em>. The privileges check doesn’t even run. This likely introduces a security problem and opens the door for attackers to destroy or severely damage the data in your customer’s or company’s online store. Not good.</li> <li><strong>The <code>product_exists()</code> check is skipped when assertions are disabled.</strong> This means <code>find_product()</code> can now be called with invalid product ids—which could lead to more severe bugs depending on how our program is written. In the worst case this could be an avenue for someone to launch Denial of Service attacks against our store. If the store app crashes if we attempt to delete an unknown product, it might be possible for an attacker to bombard it with invalid delete requests and cause an outage.</li> </ol> <p>How might we avoid these problems? The answer is to not use assertions to do data validation. Instead we could do our validation with regular if-statements and raise validation exceptions if necessary. Like so:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">delete_product</span><span class="p">(</span><span class="n">product_id</span><span class="p">,</span> <span class="n">user</span><span class="p">):</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">user</span><span class="o">.</span><span class="n">is_admin</span><span class="p">():</span> <span class="k">raise</span> <span class="n">AuthError</span><span class="p">(</span><span class="s1">'Must have admin privileges to delete'</span><span class="p">)</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">store</span><span class="o">.</span><span class="n">product_exists</span><span class="p">(</span><span class="n">product_id</span><span class="p">):</span> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'Unknown product id'</span><span class="p">)</span> <span class="n">store</span><span class="o">.</span><span class="n">find_product</span><span class="p">(</span><span class="n">product_id</span><span class="p">)</span><span class="o">.</span><span class="n">delete</span><span class="p">()</span> </pre></div> <p>This updated example also has the benefit that instead of raising unspecific <code>AssertionError</code> exceptions, it now raises semantically correct exceptions like <code>ValueError</code> or <code>AuthError</code> (which <a href="python-custom-exceptions">we’d have to define ourselves</a>).</p> </section><section><h2>Caveat #2 – Asserts That Never Fail</h2> <p>It’s easy to accidentally write Python assert statements that always evaluate to true. I’ve been bitten by this myself in the past. I wrote <a href="catching-bogus-python-asserts">a longer article about this specific issue you can check out by clicking here</a>.</p> <p>Alternatively, here’s the executive summary:</p> <p><strong>When you pass a tuple as the first argument in an <code>assert</code> statement, the assertion always evaluates as true and therefore never fails.</strong></p> <p>For example, this assertion will never fail:</p> <div class="codehilite"><pre><span></span><span class="k">assert</span><span class="p">(</span><span class="mi">1</span> <span class="o">==</span> <span class="mi">2</span><span class="p">,</span> <span class="s1">'This should fail'</span><span class="p">)</span> </pre></div> <p>This has to do with non-empty tuples always being truthy in Python. If you pass a tuple to an assert statement it leads to the assert condition to always be true—which in turn leads to the above assert statement being useless because it can never fail and trigger an exception.</p> <p>It’s relatively easy to accidentally write bad multi-line asserts due to this unintuitive behavior. This quickly leads to broken test cases that give a false sense of security in our test code. Imagine you had this assertion somewhere in your unit test suite:</p> <div class="codehilite"><pre><span></span><span class="k">assert</span> <span class="p">(</span> <span class="n">counter</span> <span class="o">==</span> <span class="mi">10</span><span class="p">,</span> <span class="s1">'It should have counted all the items'</span> <span class="p">)</span> </pre></div> <p>Upon first inspection this test case looks completely fine. However, this test case would never catch an incorrect result: it always evaluates to <code>True</code>, regardless of the state of the counter variable.</p> <p>Like I said, it’s rather easy to shoot yourself in the foot with this (mine still hurts). Luckily, there are some countermeasures you can apply to prevent this syntax quirk from causing trouble:</p> <p><a href="catching-bogus-python-asserts">&gt;&gt; Read the full article on bogus assertions to get the dirty details.</a></p> </section><section><h2>Python Assertions — Summary</h2> <p>Despite these caveats I believe that Python’s assertions are a powerful debugging tool that’s frequently underused by Python developers.</p> <p>Understanding how assertions work and when to apply them can help you write more maintainable and easier to debug Python programs. It’s a great skill to learn that will help bring your Python to the next level and make you a more well-rounded Pythonista.</p></section><footer></footer></article>https://dbader.org/blog/python-assert-tutorialWed, 18 Jan 2017 00:00:00 GMTComprehending Python’s Comprehensionshttps://dbader.org/blog/list-dict-set-comprehensions-in-python<article><header><h1>Comprehending Python’s Comprehensions</h1> <p>One of my favorite features in Python are list comprehensions. They can seem a bit arcane at first but when you break them down they are actually a very simple construct.</p> </header><section><figure><img alt="Comprehending Python's List, Dict, Set Comprehensions" src="/blog/figures/comprehending-comprehensions.png" width="1280" height="720"></figure> <p>The key to understanding list comprehensions is that they’re just <code>for</code>-loops over a collection expressed in a more terse and compact syntax. Let’s take the following list comprehension as an example:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">squares</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span> <span class="o">*</span> <span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)]</span> </pre></div> <p>It computes a list of all integer square numbers from 0 to 9:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">squares</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">9</span><span class="p">,</span> <span class="mi">16</span><span class="p">,</span> <span class="mi">25</span><span class="p">,</span> <span class="mi">36</span><span class="p">,</span> <span class="mi">49</span><span class="p">,</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">81</span><span class="p">]</span> </pre></div> <p>If we wanted to build the same list using a plain <code>for</code>-loop we’d probably write something like this:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">squares</span> <span class="o">=</span> <span class="p">[]</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span> <span class="o">...</span> <span class="n">squares</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">x</span> <span class="o">*</span> <span class="n">x</span><span class="p">)</span> </pre></div> <p>That’s a pretty straightforward loop, right? If you try and generalize some of this structure you might end up with a template similar to this:</p> <div class="codehilite"><pre><span></span><span class="p">(</span><span class="n">values</span><span class="p">)</span> <span class="o">=</span> <span class="p">[</span> <span class="p">(</span><span class="n">expression</span><span class="p">)</span> <span class="k">for</span> <span class="p">(</span><span class="n">item</span><span class="p">)</span> <span class="ow">in</span> <span class="p">(</span><span class="n">collection</span><span class="p">)</span> <span class="p">]</span> </pre></div> <p>The above list comprehension is equivalent to the following plain <code>for</code>-loop:</p> <div class="codehilite"><pre><span></span><span class="p">(</span><span class="n">values</span><span class="p">)</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="p">(</span><span class="n">item</span><span class="p">)</span> <span class="ow">in</span> <span class="p">(</span><span class="n">collection</span><span class="p">):</span> <span class="p">(</span><span class="n">values</span><span class="p">)</span><span class="o">.</span><span class="n">append</span><span class="p">(</span> <span class="p">(</span><span class="n">expression</span><span class="p">)</span> <span class="p">)</span> </pre></div> <p>Again, a fairly simple cookiecutter pattern you can apply to most for loops. Now there’s one more useful element we need to add to this template, and that is element filtering with <em>conditions</em>.</p> <p>List comprehensions can filter values based on some arbitrary condition that decides whether or not the resulting value becomes a part of the output list. Here’s an example:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">even_squares</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span> <span class="o">*</span> <span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="k">if</span> <span class="n">x</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">]</span> </pre></div> <p>This list comprehension will compute a list of the squares of all even integers from 0 to 9.</p> <p>If you’re not familiar with what the <em>modulo</em> (<code>%</code>) operator does—it returns the remainder after division of one number by another. In this example the <code>%</code>-operator gives us an easy way to test if a number is even by checking the remainder after we divide the number by 2.</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">even_squares</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">16</span><span class="p">,</span> <span class="mi">36</span><span class="p">,</span> <span class="mi">64</span><span class="p">]</span> </pre></div> <p>Similarly to the first example, this new list comprehension can be transformed into an equivalent <code>for</code>-loop:</p> <div class="codehilite"><pre><span></span><span class="n">even_squares</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span> <span class="k">if</span> <span class="n">x</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="n">even_squares</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">x</span> <span class="o">*</span> <span class="n">x</span><span class="p">)</span> </pre></div> <p>Let’s try and generalize the above <em>list comprehension to for-loop</em> transform again. This time we’re going to add a filter condition to our template to decide which values end up in the resulting list.</p> <p>Here’s the list comprehension template:</p> <div class="codehilite"><pre><span></span><span class="n">values</span> <span class="o">=</span> <span class="p">[</span><span class="n">expression</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">collection</span> <span class="k">if</span> <span class="n">condition</span><span class="p">]</span> </pre></div> <p>And we can transform this list comprehension into a <em>for</em>-loop with the following pattern:</p> <div class="codehilite"><pre><span></span><span class="n">values</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">collection</span><span class="p">:</span> <span class="k">if</span> <span class="n">condition</span><span class="p">:</span> <span class="n">values</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">expression</span><span class="p">)</span> </pre></div> <p>Again, this is a straightforward transformation—we simply apply our cookiecutter pattern again. I hope this dispelled some of the “magic” in how list comprehensions work. They’re really quite a useful tool.</p> <p>Before you move on I want to point out that Python not only supports <em>list</em> comprehensions but also has similar syntax for <em>sets</em> and <em>dictionaries</em>.</p> <p>Here’s what a <em>set comprehension</em> looks like:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="p">{</span> <span class="n">x</span> <span class="o">*</span> <span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="o">-</span><span class="mi">9</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="p">}</span> <span class="nb">set</span><span class="p">([</span><span class="mi">64</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">36</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">49</span><span class="p">,</span> <span class="mi">9</span><span class="p">,</span> <span class="mi">16</span><span class="p">,</span> <span class="mi">81</span><span class="p">,</span> <span class="mi">25</span><span class="p">,</span> <span class="mi">4</span><span class="p">])</span> </pre></div> <p>And this is a <em>dict comprehension</em>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="p">{</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span> <span class="o">*</span> <span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span> <span class="p">}</span> <span class="p">{</span><span class="mi">0</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">:</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">3</span><span class="p">:</span> <span class="mi">9</span><span class="p">,</span> <span class="mi">4</span><span class="p">:</span> <span class="mi">16</span><span class="p">}</span> </pre></div> <p>Both are useful tools in practice. There’s one caveat to Python’s comprehensions—as you get more proficient at using them it becomes easier and easier to write code that’s difficult to read. If you’re not careful you might have to deal with monstrous list, set, dict comprehensions soon. Remember, too much of a good thing is usually a bad thing.</p> <p>After much chagrin I’m personally drawing the line at one level of nesting for comprehensions. I found that in most cases it’s better (as in “more readable” and “easier to maintain”) to use <em>for</em>-loops beyond that point.</p> </section><section><h2>📺🐍 Learn More With This Video Tutorial</h2> <p>I recorded a step-by-step video tutorial that teaches you how list comprehensions work in Python to go along with the article. Watch it embedded below or <a href="https://www.youtube.com/watch?v=1HlyKKiGg-4" target="_blank">on my YouTube channel</a>:</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/1HlyKKiGg-4?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> </section><section><h2>Key Takeaways</h2> <ul> <li>Comprehensions are a key feature in Python. Understanding and applying them will make your code much more Pythonic.</li> <li>Comprehensions are just fancy syntax for a simple <code>for</code>-loop pattern. Once you understand the pattern, you’ll develop an intuitive understanding for comprehensions.</li> <li>There are more than just list comprehensions.</li> </ul></section><footer></footer></article>https://dbader.org/blog/list-dict-set-comprehensions-in-pythonWed, 11 Jan 2017 00:00:00 GMTThe Difference Between “is” and “==” in Pythonhttps://dbader.org/blog/difference-between-is-and-equals-in-python<article><header><h1>The Difference Between “is” and “==” in Python</h1> <p>Python has two operators for equality comparisons, “is” and “==” (equals). In this article I’m going to teach you the difference between the two and when to use each with a few simple examples.</p> </header><section><figure><img alt='When to use "==" and "is" in Python' src="/blog/figures/python-is-vs-equals.png" width="1280" height="720"></figure> <p>When I was a kid, our neighbors had two twin cats.</p> <p>Both cats looked seemingly identical—same charcoal fur, same piercing green eyes. Some personality quirks aside, you just couldn’t tell them apart just from looking at them. But of course they were two different cats, two separate beings, even though they looked exactly the same.</p> <p>There’s a difference in meaning between <em>equal</em> and <em>identical</em>. And this difference is important when you want to understand how Python’s <code>is</code> and <code>==</code> comparison operators behave.</p> <p><strong>The <code>==</code> operator compares by checking for <u>equality</u></strong>: If these cats were Python objects and we’d compare them with the <code>==</code> operator, we’d get “both cats are equal” as an answer.</p> <p><strong>The <code>is</code> operator, however, compares <u>identities</u></strong>: If we compared our cats with the <code>is</code> operator, we’d get “these are two different cats” as an answer.</p> <p>But before I get all tangled up in this ball of twine of a cat analogy, let’s take a look at some real Python code.</p> <p>First, we’ll create a new list object and name it <code>a</code>, and then define another variable <code>b</code> that points to the same list object:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">b</span> <span class="o">=</span> <span class="n">a</span> </pre></div> <p>Let’s inspect these two variables. We can see they point to identical looking lists:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">b</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> </pre></div> <p>Because the two list objects look the same we’ll get the expected result when we compare them for equality using the <code>==</code> operator:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="o">==</span> <span class="n">b</span> <span class="bp">True</span> </pre></div> <p>However, that doesn’t tell us whether <code>a</code> and <code>b</code> are actually pointing to the same object. Of course, we know they do because we assigned them earlier, but suppose we didn’t know—how might we find out?</p> <p>The answer is comparing both variables with the <code>is</code> operator. This confirms both variables are in fact pointing to one list object:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="ow">is</span> <span class="n">b</span> <span class="bp">True</span> </pre></div> <p>Let’s see what happens when we create an identical copy of our list object. We can do that by calling <code>list()</code> on the existing list to create a copy we’ll name <code>c</code>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">c</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> </pre></div> <p>Again you’ll see that the new list we just created looks identical to the list object pointed to by <code>a</code> and <code>b</code>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">c</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> </pre></div> <p>Now this is where it gets interesting—let’s compare our list copy <code>c</code> with the initial list <code>a</code> using the <code>==</code> operator. What answer do you expect to see?</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="o">==</span> <span class="n">c</span> <span class="bp">True</span> </pre></div> <p>Okay, I hope this was what you expected. What this result tells us is that <code>c</code> and <code>a</code> have the same contents. They’re considered equal by Python. But are they actually pointing to the same object? Let’s find out with the <code>is</code> operator:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="ow">is</span> <span class="n">c</span> <span class="bp">False</span> </pre></div> <p>Boom—this is where we get a different result. Python is telling us that <code>c</code> and <code>a</code> are pointing to two different objects, even though their contents might be the same.</p> <p>So, to recap let’s try and break the difference between <code>is</code> and <code>==</code> down to two short definitions:</p> <ul> <li> <p>An <code>is</code> expression evaluates to <code>True</code> if two variables point to the same (identical) object.</p> </li> <li> <p>An <code>==</code> expression evaluates to <code>True</code> if the objects referred to by the variables are equal (have the same contents).</p> </li> </ul> <p>Just remember, think of twin cats (dogs should work, too) whenever you need to decide between using <code>is</code> and <code>==</code> in Python. You’ll be fine.</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/CZ8bZPqtwU0?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div></section><footer></footer></article>https://dbader.org/blog/difference-between-is-and-equals-in-pythonWed, 28 Dec 2016 00:00:00 GMTHow to Speak at a Python Conferencehttps://dbader.org/blog/how-to-speak-at-a-python-conference<article><header><h1>How to Speak at a Python Conference</h1> <p>My tutorial on how you can get a first-time speaking gig at a tech conference like PyCon as a software developer.</p> </header><section><figure><img alt="How To Speak at a Python Conference" src="/blog/figures/how-to-speak-python-conference-header.png" width="1280" height="720"></figure> <p>I gave <a href="https://www.youtube.com/watch?v=Di50Y5b20p8" target="_blank">a talk at PyCon Germany this year</a> and I was just chatting with my friend Sergei who wants to get into presenting at tech conferences and is looking for a way to get started. Here’s Segei’s question:</p> <blockquote> <p>My goal for now it to get ready for the spring season of conferences, and pick one where I could present something.</p> <p>I want to try presenting stuff on conferences, as it is a great public speaking experience for me and… free attendance of a good conference :)</p> <p><strong>Where do you start if you want to present something?</strong></p> </blockquote> <p>If you’re thinking about speaking at a tech conference that’s a fantastic idea! It’s a lot of fun (it’s a lot of work, too)—but it’s totally worth it.</p> <p>It’s great for getting in touch with people, making new connections, finding a new or better job, and to improve your public speaking skills. Plus, you get tagged as a speaker on your conference badge—a great conversation starter 😜</p> </section><section><h2>Alright, I’m sold—How do I become a first-time conference speaker?</h2> <p>For most tech conferences you can become a speaker by applying with a talk proposal a few months before the conference takes place.</p> <p>Basically, all the legwork is on you. You’re picking the topic and pitching it to the conference organizers and they decide which talks they accept. If you’ve built a bit of a reputation for yourself then conference organizers might contact you and ask you to speak at their conference. But for first time speakers that’s unlikely.</p> <p>The key thing here is to apply early enough and before the “call for proposals” deadline. The deadline usually ends well in advance before the conference takes place. It absolutely helps and increases your chances to get in if you plan ahead of time.</p> </section><section><h2>Where can I find upcoming conferences to speak at?</h2> <p>For conferences related to Python there’s the Python Events calendar at:</p> <ul> <li><a href="http://www.pycon.org/" target="_blank">http://www.pycon.org/</a>; and</li> <li><a href="https://www.python.org/events/" target="_blank">https://www.python.org/events/</a>.</li> </ul> <p>Use the events calendar to find a conference that looks interesting. Then check out the conference website to see what the application process is like and when the call for proposals (submission deadline) ends.</p> <p>At this point you should know which event you want to speak at, who to send your talk proposal to, and when the submission deadline is. The next step is to put together and send in your talk proposal.</p> </section><section><h2>How do I write a good talk proposal?</h2> <p>The most important thing you need to realize about talk proposals is that <em>they’re all about the organizers</em>. You need to put yourself in the shoes of the organizers when you’re writing the proposal—that’s the biggest trick to getting your first talk accepted.</p> <p>Your goal is to convince the organizers that:</p> <ul> <li>inviting you as a speaker won’t embarrass them; and</li> <li>their audience is likely going to enjoy your talk.</li> </ul> <p>Keep this in mind while writing your proposal and it’ll increase your chances to get your talk accepted immensely. Whatever information you can give to convince the organizers your talk will do these two things for them helps.</p> <p>For example, be sure to mention the intended audience for your talk and to share your previous speaking experience (meetup/work/university presentations). Also mention your own expertise on the talk’s topic.</p> <p>If you don’t have previous public speaking experience it can be a bit tough to get into conferences at first. To “hack” this system you could also point people to a YouTube video tutorial you recorded. That way the organizers can get a sense for your presentation skills (I always include a link to my <a href="/screencasts">YouTube channel</a> for that reason.)</p> <p>To make this a bit easier for you I’ve put together a conference proposal template you can use. <a href="https://www.getdrip.com/forms/46062574/submissions/new" data-drip-show-form="46062574"><strong>Click here and I’ll email it over to you.</strong></a></p> <p>My template will help you cover all of the relevant info. I used this template to get a talk into PyCon Germany not long ago, for example. Good luck 😃</p> </section><section><h2>What happens after I send in my proposal?</h2> <p>After you’ve sent in your proposal all you need to do is wait for an answer. If you get accepted—congratulations, you’ll be speaking at a conference soon!</p> <p>Usually you won’t get feedback when your talk gets rejected (it’s kind of similar to a job interview that way). But don’t be too upset if your talk is rejected! You can always try again with the same talk at a different conference.</p> <p>My gut feeling is that the biggest factor in whether or not your talk gets accepted is “quality indicators” like your previous speaking experience—so be sure to talk about that in your proposal.</p> <p>It’ll get much easier to get your talks accepted once you’ve spoken a few times.</p> </section><section><h2>Can I attend for free if I’m speaking?</h2> <p>Most non-commercial tech conferences don’t waive the attendance fee even if you’re a speaker. I believe it’s different with keynote speakers but if you’re a regular speaker at a small to medium size conference you’ll often pay the full fee or a slightly reduced fee. Still worth it though 😊</p> </section><section><h2>Got any more tips?</h2> <p>Sure do! Watch this video I recorded to get some extra tips on how to bootstrap your speaking career (there’s more videos on <a href="https://www.youtube.com/playlist?list=PLP8GkvaIxJP37FpJZcj_NhP9-r1a_AkqJ" target="_blank">this playlist</a>):</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/0-BkCkAiso8?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> </section><section><h2>Extra Resources</h2> <ul> <li><a href="https://www.getdrip.com/forms/46062574/submissions/new" data-drip-show-form="46062574">PDF talk proposal template (got me accepted at PyCon Germany)</a></li> <li><a href="https://www.python.org/events/" target="_blank">Python Events Calendar</a></li> <li><a href="http://katemats.com/public-speaking-at-a-conference/" target="_blank">Kate Matsudaira: “So you want to speak at a conference”</a></li> <li><a href="http://wunder.schoenaberselten.com/2016/02/16/how-to-prepare-and-write-a-tech-conference-talk/" target="_blank">Lena Reinhard: “How to prepare and write a tech conference talk”</a></li> <li><a href="https://medium.com/@karen_meep/how-to-start-speaking-in-tech-conferences-f64a9f3a84a6" target="_blank">Karen Meep: “How to start speaking (in tech conferences)”</a></li> <li><a href="https://www.ashedryden.com/blog/what-you-need-to-know-about-speaking-at-conferences" target="_blank">Ashe Dryden: “What you need to know about speaking at conferences”</a></li> </ul></section><footer></footer></article>https://dbader.org/blog/how-to-speak-at-a-python-conferenceTue, 13 Dec 2016 00:00:00 GMTCool new features in Python 3.6https://dbader.org/blog/cool-new-features-in-python-3-6<article><header><h1>Cool new features in Python 3.6</h1> <p>Python 3.6 adds a couple of new features and improvements that’ll affect the day to day work of Python coders. In this article I’ll give you an overview of the new features I found the most interesting.</p> </header><section><figure><img alt="" src="/blog/figures/python-36-features.png" width="1280" height="720"></figure> </section><section><h2>Improved numeric literals</h2> <p>This is a syntactic tweak that makes numeric literals easier to read. You can now add underscores to numbers in order to group them to your liking. This is handy for expressing large quantities or constants in binary or hexadecimal:</p> <div class="codehilite"><pre><span></span>&gt;&gt;&gt; <span class="nv">six_figures</span> <span class="o">=</span> 100_000 &gt;&gt;&gt; six_figures <span class="m">100000</span> </pre></div> <div class="codehilite"><pre><span></span>&gt;&gt;&gt; <span class="nv">programmer_error</span> <span class="o">=</span> 0xbad_c0ffee &gt;&gt;&gt; <span class="nv">flags</span> <span class="o">=</span> 0b_0111_0101_0001_0101 </pre></div> <p>Remember, this change doesn’t introduce any new semantics. It’s just a way to represent numeric literals differently in your source code. A small but neat addition.</p> <p>You can learn more about this change in <a href="https://www.python.org/dev/peps/pep-0515/" target="_blank">PEP 515</a>.</p> </section><section><h2>String interpolation</h2> <p>Python 3.6 adds yet another way to format strings called <em>Formatted String Literals</em>. This new way of formatting strings lets you use embedded Python expressions inside string constants. Here’s are two simple examples to give you a feel for the feature:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">'Bob'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">f</span><span class="s1">'Hello, </span><span class="si">{name}</span><span class="s1">!'</span> <span class="s1">'Hello, Bob!'</span> </pre></div> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">5</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">b</span> <span class="o">=</span> <span class="mi">10</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">f</span><span class="s1">'Five plus ten is {a + b} and not {2 * (a + b)}.'</span> <span class="s1">'Five plus ten is 15 and not 30.'</span> </pre></div> <p>String literals also support the existing format string syntax of the <code>str.format()</code> method. That allows you to do things like:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">error</span> <span class="o">=</span> <span class="mi">50159747054</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">f</span><span class="s1">'Programmer Error: </span><span class="si">{error:#x}</span><span class="s1">'</span> <span class="s1">'Programmer Error: 0xbadc0ffee'</span> </pre></div> <p>Python’s new Formatted String Literals are similar to the <a href="https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Template_literals" target="_blank">JavaScript Template Literals added in ES2015/ES6</a>. I think they’re quite a nice addition to the language and I look forward to using them in my day to day work.</p> <p>You can learn more about this change in <a href="https://www.python.org/dev/peps/pep-0498/" target="_blank">PEP 498</a>.</p> </section><section><h2>Type annotations for variables</h2> <p>Starting with Python 3.5 you could add type annotations to functions and methods:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">my_add</span><span class="p">(</span><span class="n">a</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">b</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span> <span class="o">...</span> <span class="k">return</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span> </pre></div> <p>In Python 3.6 you can use a syntax similar to type annotations for function arguments to type-hint standalone variables:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">python_version</span> <span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mf">3.6</span> </pre></div> <p>Nothing has changed in terms of the semantics–CPython simply records the type as a type annotation but doesn’t validate or check types in any way.</p> <p>Type-checking is purely optional and you’ll need <a href="https://www.youtube.com/watch?v=2xWhaALHTvU" target="_blank">a tool like Mypy</a> for that, which basically works like a <a href="/blog/python-code-linting">code linter</a>.</p> <p>You can learn more about this change in <a href="https://www.python.org/dev/peps/pep-0526/" target="_blank">PEP 526</a>.</p> </section><section><h2>Watch a video summary of the best new features in Python 3.6</h2> <p></p><div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/klKdMxjDaa0?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> </section><section><h2>Other notable changes</h2> <p>I think Python 3.6 will be an interesting release. There are many more interesting additions and improvements that are worth checking out. You can learn more about them in the links below or by reading the <a href="https://docs.python.org/3.6/whatsnew/3.6.html" target="_blank">official “What’s New In Python 3.6” announcement</a>.</p> <ul> <li><a href="https://docs.python.org/3.6/whatsnew/3.6.html#whatsnew36-pep525" target="_blank">Syntax for asynchronous generators</a></li> <li><a href="https://docs.python.org/3.6/whatsnew/3.6.html#whatsnew36-pep530" target="_blank">Syntax for asynchronous comprehensions</a></li> <li><a href="https://docs.python.org/3.6/whatsnew/3.6.html#whatsnew36-compactdict" target="_blank">Dictionaries are faster and use 20% to 25% less memory</a></li> <li><a href="https://docs.python.org/3.6/whatsnew/3.6.html#whatsnew36-typing" target="_blank">The typing module is considered a stable API</a></li> <li><a href="https://docs.python.org/3.6/library/secrets.html#module-secrets" target="_blank">A new “secrets” module for generating cryptographically secure authentication tokens</a></li> <li><a href="https://docs.python.org/3.6/whatsnew/3.6.html" target="_blank">…and more</a></li> </ul></section><footer></footer></article>https://dbader.org/blog/cool-new-features-in-python-3-6Thu, 08 Dec 2016 00:00:00 GMT3 Reasons why you need a programming bloghttps://dbader.org/blog/3-reasons-why-you-need-a-programming-blog<article><header><h1>3 Reasons why you need a programming blog</h1> <p>One of the best things I ever did for my dev career: A little story and three reasons why you should start a programming portfolio website right now.</p> </header><section><figure><img alt="" src="/blog/figures/programming-blog-reasons.png" width="1280" height="720"></figure> <p>At PyCon Germany I chatted with Astrid, a freelance Python (Django) developer looking for ways to improve her career and to find more contracts.</p> <p>Astrid seemed quite frustrated with her situation—it was tough for her to get the contracts and jobs she really wanted.</p> <p>Often when she sent out her resume for more desirable gigs she wouldn’t even receive an answer. It sounded like she was stuck with a certain quality of clients and couldn’t really push past that invisible barrier.</p> <p>I always love to help a sister (or brother) out and went into full-on diagnosing mode. Usually I just end up spouting unsolicited advice in these situations but with Astrid I think I actually hit the nail on the head… 😉</p> <p>Eventually I asked Astrid if she had a website or blog as a “programmer portfolio” of sorts.</p> <p>She did not.</p> <p>And I think that was a BIG mistake –</p> <p>Looking back I’d say starting my personal website at dbader.org was probably the best thing I ever did for my programming career:</p> </section><section><h2>Reason #1: Employers loved it–it made it much easier to get interviews</h2> <p>In fact once I had my website up for a while companies started contacting me through it. And they were no longer the crappy recruiter emails I got through LinkedIn, but from managers and dev leads at companies that I found actually interesting.</p> </section><section><h2>Reason #2: It was easier to get started than I thought</h2> <p>I launched my site with just 3 articles I wrote over the holidays hanging out with my family one year. I was surprised to find I got more (not less) traffic over time even though I didn’t post new stuff constantly. More people started linking to my posts and they ranked higher in Google (also search engines seem to favor content that has been around for a while). It was incredibly fun to see that growth and to find new ways of reaching developers.</p> </section><section><h2>Reason #3: It put me in touch with so many fine folks (like you!)</h2> <p>Most of the places I lived in didn’t have strong software dev / meetup communities. Starting a website was a fantastic way to make friends with other developers around the world and to exchange ideas.</p> </section><section><h2>How you can get started today</h2> <p>I know it seems super difficult to get everything set up in the beginning. And the work involved can seem kind of boring at first… “it’s just a website”.</p> <p>What finally got me started with setting up my own website was turning it into a programming exercise.</p> <p>Instead of using a pre-fab framework like Wordpress I wrote my own Python framework for generating the website.</p> <p>I figured even if I wouldn’t follow through with the site I’d learn some web development skills in the process… And this was exactly true 😃</p> <p>Putting myself in Astrid’s shoes again I really believe every software developer should have a personal website. The time investment is so small in comparison to the awesome benefits and opportunities it can generate for you.</p> <p>If you’re sold on the idea of starting a programming blog but you don’t know how to go about it yet then check out this video I created for you.</p> <p>In the video embedded below I’m going over my own website as an example and how it looks very different today compared to when I started it in 2012.</p> <p>It doesn’t take much to get started with your own programming blog or portfolio website and the benefits can be huge.</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/wqpBeGeOMgM?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div></section><footer></footer></article>https://dbader.org/blog/3-reasons-why-you-need-a-programming-blogWed, 30 Nov 2016 00:00:00 GMTA Python Riddle: The Craziest Dict Expression in the Westhttps://dbader.org/blog/python-mystery-dict-expression<article><header><h1>A Python Riddle: The Craziest Dict Expression in the West</h1> <p>Let’s pry apart this slightly unintuitive Python dictionary expression to find out what’s going on in the uncharted depths of the Python interpreter.</p> </header><section><figure><img alt="" src="/blog/figures/python-dict-riddle.png" width="661" height="371"></figure> <p>A while ago I shared this Python one-liner as a “Python riddle” <a href="https://twitter.com/dbader_org/status/784108741373427718" target="_blank">on Twitter</a> and it got some interesting reactions:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="p">{</span><span class="bp">True</span><span class="p">:</span> <span class="s1">'yes'</span><span class="p">,</span> <span class="mi">1</span><span class="p">:</span> <span class="s1">'no'</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">:</span> <span class="s1">'maybe'</span><span class="p">}</span> </pre></div> <p>Take a quick moment to think about what this dict expression will evaluate to.</p> <p>I’ll wait here 😃</p> <p>Ok, ready?</p> <p>It’ll evaluate to this dictionary:</p> <div class="codehilite"><pre><span></span><span class="p">{</span><span class="bp">True</span><span class="p">:</span> <span class="s1">'maybe'</span><span class="p">}</span> </pre></div> <p>I’ll admit I was pretty surprised about this result the first time I saw it. But it all makes sense when you investigate what happens step by step.</p> </section><section><h2>Where baby dictionaries come from</h2> <p>Let’s think about why we get this (I want to say “slightly unintuitive”) result:</p> <p>When Python processes our dictionary expression it first constructs a new empty dictionary object; and then assigns the keys and values to it in the order given in the dict expression.</p> <p>When we break it down our dict expression is equivalent to this sequence of statements:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">xs</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">xs</span><span class="p">[</span><span class="bp">True</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'yes'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">xs</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'no'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">xs</span><span class="p">[</span><span class="mf">1.0</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'maybe'</span> </pre></div> <p>Now this gets a lot more interesting when we realize that <em>all keys</em> we’re using in this example are considered to be <em>equal</em> by Python:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="bp">True</span> <span class="o">==</span> <span class="mi">1</span> <span class="o">==</span> <span class="mf">1.0</span> <span class="bp">True</span> </pre></div> <p>Okay, but–wait a minute here. We can intuitively accept that <code>1.0 == 1</code>, but why would <code>True</code> be considered equal to <code>1</code> as well?</p> <p>The answer to that is Python treats <code>bool</code> as a subclass of <code>int</code><sup id="fnref:bool-indexes"><a class="footnote-ref" href="#fn:bool-indexes" rel="footnote">1</a></sup>. This is the case in <a href="https://docs.python.org/2/reference/datamodel.html#the-standard-type-hierarchy" target="_blank">Python 2</a> and <a href="https://docs.python.org/3/reference/datamodel.html#the-standard-type-hierarchy" target="_blank">Python 3</a>:</p> <blockquote> <p>The Boolean type is a subtype of the integer type, and Boolean values behave like the values 0 and 1, respectively, in almost all contexts, the exception being that when converted to a string, the strings “False” or “True” are returned, respectively.</p> </blockquote> <p>So, as far as Python is concerned, <code>True</code>, <code>1</code>, and <code>1.0</code> all represent <em>the same dictionary key</em>. As the interpreter evaluates the dictionary expression it repeatedly overwrites the value for the key <code>True</code>.</p> <p>But why do we still get <code>True</code> as the key in the final dictionary? Shouldn’t the key also change to <code>1.0</code> at the end through the repeated assignments?</p> <p>To explain this outcome we need to know that Python doesn’t replace the object instance for a key when a new value is assigned to it:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">ys</span> <span class="o">=</span> <span class="p">{</span><span class="mf">1.0</span><span class="p">:</span> <span class="s1">'no'</span><span class="p">}</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ys</span><span class="p">[</span><span class="bp">True</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'yes'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ys</span> <span class="p">{</span><span class="mf">1.0</span><span class="p">:</span> <span class="s1">'yes'</span><span class="p">}</span> </pre></div> <p>This is presumably done as a performance optimization–if the keys are considered identical then why update the original. From this example we saw that the initial <code>True</code> object is never replaced as the key. Therefore the dictionary’s string representation still prints the key as <code>True</code> (instead of <code>1</code> or <code>1.0</code>).</p> </section><section><h2>Wait, what about the hash code?</h2> <p>With what we know now it looks like the values in the resulting dict are getting overwritten because they compare as equal. However, it turns out that this effect isn’t caused by the <code>__eq__</code> equality check alone, either.</p> <p>Let’s define the following class as our little detective tool:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">AlwaysEquals</span><span class="p">:</span> <span class="k">def</span> <span class="fm">__eq__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span> <span class="sd">"""Make this object equal to any other object."""</span> <span class="k">return</span> <span class="bp">True</span> <span class="k">def</span> <span class="fm">__hash__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="sd">"""Give each object a unique hash code."""</span> <span class="k">return</span> <span class="nb">id</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> </pre></div> <p>All instances of this class will pretend they’re equal to any other object:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">AlwaysEquals</span><span class="p">()</span> <span class="o">==</span> <span class="n">AlwaysEquals</span><span class="p">()</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">AlwaysEquals</span><span class="p">()</span> <span class="o">==</span> <span class="mi">42</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">AlwaysEquals</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'waaat?'</span> <span class="bp">True</span> </pre></div> <p>However they will also each return a unique hash value:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="p">[</span><span class="nb">hash</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span> <span class="k">for</span> <span class="n">obj</span> <span class="ow">in</span> <span class="p">[</span><span class="n">AlwaysEquals</span><span class="p">(),</span> <span class="n">AlwaysEquals</span><span class="p">(),</span> <span class="n">AlwaysEquals</span><span class="p">()]]</span> <span class="p">[</span><span class="mi">4574298968</span><span class="p">,</span> <span class="mi">4574287912</span><span class="p">,</span> <span class="mi">4574287072</span><span class="p">]</span> </pre></div> <p>That’ll allow us to test if dictionary keys are overwritten based on their equality comparison result alone. And you’ll see that the keys are <em>not</em> getting overwritten even though they always compare as equal:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="p">{</span><span class="n">AlwaysEquals</span><span class="p">():</span> <span class="s1">'yes'</span><span class="p">,</span> <span class="n">AlwaysEquals</span><span class="p">():</span> <span class="s1">'no'</span><span class="p">}</span> <span class="p">{</span> <span class="o">&lt;</span><span class="n">AlwaysEquals</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x110a3c588</span><span class="o">&gt;</span><span class="p">:</span> <span class="s1">'yes'</span><span class="p">,</span> <span class="o">&lt;</span><span class="n">AlwaysEquals</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x110a3cf98</span><span class="o">&gt;</span><span class="p">:</span> <span class="s1">'no'</span> <span class="p">}</span> </pre></div> <p>We can also flip this idea around and check to see if returning the same hash value is enough to cause keys to get overwritten:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="k">class</span> <span class="nc">SameHash</span><span class="p">:</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="fm">__hash__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="o">&gt;&gt;&gt;</span> <span class="s2">"""Give each object the same hash code."""</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">return</span> <span class="mi">1</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="o">=</span> <span class="n">SameHash</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">b</span> <span class="o">=</span> <span class="n">SameHash</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">a</span> <span class="o">==</span> <span class="n">b</span> <span class="bp">False</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">hash</span><span class="p">(</span><span class="n">a</span><span class="p">),</span> <span class="nb">hash</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">({</span><span class="n">a</span><span class="p">:</span> <span class="s1">'a'</span><span class="p">,</span> <span class="n">b</span><span class="p">:</span> <span class="s1">'b'</span><span class="p">})</span> <span class="p">{</span> <span class="o">&lt;</span><span class="n">SameHash</span> <span class="n">instance</span> <span class="n">at</span> <span class="mh">0x7f7159020cb0</span><span class="o">&gt;</span><span class="p">:</span> <span class="s1">'a'</span><span class="p">,</span> <span class="o">&lt;</span><span class="n">SameHash</span> <span class="n">instance</span> <span class="n">at</span> <span class="mh">0x7f7159020cf8</span><span class="o">&gt;</span><span class="p">:</span> <span class="s1">'b'</span> <span class="p">}</span> </pre></div> <p>As this example shows, the “keys get overwritten” effect isn’t caused by the hash value alone either.</p> <p>A better explanation of what’s going on here is that dictionaries check for equality <em>and</em> compare the hash value to determine if two keys are the same:</p> <blockquote> <p>An object is hashable if it has a hash value which never changes during its lifetime (it needs a <code>__hash__()</code> method), and can be compared to other objects (it needs an <code>__eq__()</code> method). Hashable objects which compare equal must have the same hash value.</p> <p>Hashability makes an object usable as a dictionary key and a set member, because these data structures use the hash value internally.</p> <p>(Source: <a href="https://docs.python.org/3/glossary.html#term-hashable" target="_blank">https://docs.python.org/3/glossary.html#term-hashable</a>)</p> </blockquote> </section><section><h2>Umm okay, what’s the executive summary here?</h2> <p>Let’s try and summarize our findings:</p> <blockquote> <p><code>{True: 'yes', 1: 'no', 1.0: 'maybe'}</code> evaluates to <code>{True: 'maybe'}</code> because the keys <code>True</code>, <code>1</code>, and <code>1.0</code> all compare as equal <em>and</em> they all have the same hash value.</p> </blockquote> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="bp">True</span> <span class="o">==</span> <span class="mi">1</span> <span class="o">==</span> <span class="mf">1.0</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="p">(</span><span class="nb">hash</span><span class="p">(</span><span class="bp">True</span><span class="p">),</span> <span class="nb">hash</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> <span class="nb">hash</span><span class="p">(</span><span class="mf">1.0</span><span class="p">))</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> </pre></div> <p>That’s how we end up with this slightly surprising result as the dictionary’s final state:</p> <div class="codehilite"><pre><span></span><span class="p">{</span><span class="bp">True</span><span class="p">:</span> <span class="s1">'yes'</span><span class="p">,</span> <span class="mi">1</span><span class="p">:</span> <span class="s1">'no'</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">:</span> <span class="s1">'maybe'</span><span class="p">}</span> <span class="o">==</span> <span class="p">{</span><span class="bp">True</span><span class="p">:</span> <span class="s1">'maybe'</span><span class="p">}</span> </pre></div> </section><section><h2>It’s a Python Trick!</h2> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/CZ8bZPqtwU0?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p>I understand that all of this can be a bit mind-boggling at first. Try playing through the examples I gave one by one in a Python REPL. I’m sure it’ll help expand your knowledge of Python!</p> <p>I love these short one-line examples that teach you a ton about a language once you wrap your head around them. They’re almost like a <a href="https://en.wikipedia.org/wiki/K%C5%8Dan" target="_blank">Zen kōan</a> 😃</p> <p>There’s one more thing I want to tell you about:</p> <p>I’ve started a series of these Python “tricks” delivered over email. You can sign up at <a href="/python-tricks">dbader.org/python-tricks</a> and I’ll send you a new Python trick as a code screenshot every couple of days.</p> <p>This is still an experiment and a work in progress but I’ve heard some really positive feedback from the developers who’ve tried it out so far.</p> <p><em>Thanks to JayR, Murat, and kurashu89 for their feedback on this article.</em></p> <div class="footnote"> <hr> <ol> <li id="fn:bool-indexes"> <p>Yes, this also means you can do things like this and use bools as indexes: <code>('no', 'yes')[True] == 'yes'</code> But you probably <em>shouldn’t</em> do it for the sake of clarity 😉 <a class="footnote-backref" href="#fnref:bool-indexes" rev="footnote" title="Jump back to footnote 1 in the text">↩</a></p> </li> </ol> </div></section><footer></footer></article>https://dbader.org/blog/python-mystery-dict-expressionWed, 23 Nov 2016 00:00:00 GMTThe Ultimate List of Python Podcastshttps://dbader.org/blog/ultimate-list-of-python-podcasts<article><header><h1>The Ultimate List of Python Podcasts</h1> <p>I couldn’t find a good and updated list of Python developer or Python programming podcasts online. So I created my own list with the best Python podcasts.</p> </header><section><figure><img alt="The Ultimate List of Python Podcasts" src="/blog/figures/ultimate-list-of-python-podcasts.png" width="1280" height="720"></figure> <p>I enjoy listening to all kinds of podcasts when I’m at the gym or driving. There are some really good podcasts about Python development out there but I just couldn’t find a good (and updated) list.</p> <p>I initially created this list on forum posts and by searching the iTunes podcast directory and I will continue to grow it with user feedback. I plan to keep this list updated, so feel free to shoot me an email if you think anything is missing or if you’d like to see your own Python podcast added.</p> <p>My criteria for inclusion on this list are:</p> <ul> <li>episode download links must work; and</li> <li>the podcast must be active (new episodes are coming out) <em>OR</em> at least have an interesting archive with old episodes worth listening to.</li> </ul> <p>Enjoy the podcasts! 🎙🐍</p> </section><section><h2><a href="https://talkpython.fm/" target="_blank">Talk Python To Me</a></h2> <blockquote> <p>Talk Python to Me is a weekly podcast hosted by Michael Kennedy. The show covers a wide array of Python topics as well as many related topics (e.g. MongoDB, AngularJS, DevOps). The format is a casual 45 minute conversation with industry experts.</p> </blockquote> <ul> <li>Website: <a href="https://talkpython.fm/" target="_blank">talkpython.fm</a></li> <li>Twitter: <a href="https://twitter.com/TalkPython" target="_blank">@TalkPython</a></li> <li>Subscribe: <a href="https://talkpython.fm/episodes/rss" target="_blank">RSS</a> ⋅ <a href="https://itunes.apple.com/podcast/id979020229" target="_blank">iTunes</a> ⋅ <a href="https://overcast.fm/itunes979020229/" target="_blank">Overcast.fm</a></li> </ul> </section><section><h2><a href="https://www.podcastinit.com/" target="_blank">Podcast.__init__</a></h2> <blockquote> <p>A podcast about Python and the people who make it great. Hosted by Tobias Macey and Christopher Patti.</p> </blockquote> <ul> <li>Website: <a href="https://www.podcastinit.com/" target="_blank">www.podcastinit.com</a></li> <li>Twitter: <a href="https://twitter.com/Podcast__init__" target="_blank">@Podcast__init__</a></li> <li>Subscribe: <a href="https://www.podcastinit.com/feed/mp3/" target="_blank">RSS</a> ⋅ <a href="https://itunes.apple.com/podcast/id981834425" target="_blank">iTunes</a> ⋅ <a href="https://overcast.fm/itunes981834425/" target="_blank">Overcast.fm</a></li> </ul> </section><section><h2><a href="https://pythonbytes.fm/" target="_blank">Python Bytes</a></h2> <blockquote> <p>Python Bytes is a weekly podcast hosted by Michael Kennedy and Brian Okken. Python Bytes podcast delivers headlines directly to your earbuds. If you want to stay up on the Python developer news but don’t have time to scour reddit, twitter, and other news sources, just subscribe and you’ll get the best picks delivered weekly.</p> </blockquote> <ul> <li>Website: <a href="https://pythonbytes.fm/" target="_blank">pythonbytes.fm</a></li> <li>Twitter: <a href="https://twitter.com/PythonBytes" target="_blank">@PythonBytes</a></li> <li>Subscribe: <a href="https://pythonbytes.fm/episodes/rss" target="_blank">RSS</a> ⋅ <a href="https://itunes.apple.com/podcast/id1173690032" target="_blank">iTunes</a> ⋅ <a href="https://overcast.fm/itunes1173690032/" target="_blank">Overcast.fm</a></li> </ul> </section><section><h2><a href="http://python.madewithopinion.com/" target="_blank">The Python Experience</a></h2> <blockquote> <p>The personal ramblings of the perpetual computer programming newb, finally finding his way “home” to Python, and determined to do things more Pythonically.</p> </blockquote> <ul> <li>Website: <a href="http://python.madewithopinion.com/" target="_blank">python.madewithopinion.com</a></li> <li>Twitter: <a href="https://twitter.com/miklevin" target="_blank">@miklevin</a></li> <li>Subscribe: <a href="http://python.madewithopinion.com/feeds/python/rss/" target="_blank">RSS</a> ⋅ <a href="https://itunes.apple.com/podcast/id1165128787" target="_blank">iTunes</a> ⋅ <a href="https://overcast.fm/itunes1165128787/" target="_blank">Overcast.fm</a></li> </ul> </section><section><h2><a href="http://pythontesting.net/podcast" target="_blank">Test &amp; Code</a></h2> <blockquote> <p>A podcast about Software Development, Software Testing, and Python. How did you become a software developer/tester/engineer/lead, etc? Odds are we all are missing some important information to do our jobs most effectively. This podcast is an attempt to fill those education gaps. I focus on testing and process questions like “How do I know it works?”, “How do I effectively test?”, and the like. But really, anything in the software development realm is fair game.</p> </blockquote> <ul> <li>Website: <a href="http://pythontesting.net/podcast" target="_blank">pythontesting.net/podcast</a></li> <li>Twitter: <a href="https://twitter.com/TestPodcast" target="_blank">@TestPodcast</a></li> <li>Subscribe: <a href="http://pythontesting.net/feed/podcast/" target="_blank">RSS</a> ⋅ <a href="https://itunes.apple.com/podcast/id1029487211" target="_blank">iTunes</a> ⋅ <a href="https://overcast.fm/itunes1029487211/" target="_blank">Overcast.fm</a></li> </ul> </section><section><h2><a href="https://www.kennethreitz.org/import-this/" target="_blank">Import This: A Python Podcast for Humans</a></h2> <blockquote> <p>Featuring Kennneth Reitz and a random Python co-host.</p> </blockquote> <ul> <li>Website: <a href="https://www.kennethreitz.org/import-this/" target="_blank">www.kennethreitz.org/import-this/</a></li> <li>Twitter: <a href="https://twitter.com/kennethreitz" target="_blank">@kennethreitz</a></li> <li>Subscribe: <a href="http://feeds.soundcloud.com/users/soundcloud:users:82237854/sounds.rss" target="_blank">RSS</a> ⋅ <a href="https://itunes.apple.com/podcast/id829754832" target="_blank">iTunes</a> ⋅ <a href="https://overcast.fm/itunes829754832/" target="_blank">Overcast.fm</a></li> </ul> </section><section><h2><a href="http://radiofreepython.com/" target="_blank">Radio Free Python</a></h2> <blockquote> <p>A monthly podcast focused on the Python programming language and its community.</p> </blockquote> <p>(No updates since 2013 but an interesting archive of episodes.)</p> <ul> <li>Website: <a href="http://radiofreepython.com/" target="_blank">radiofreepython.com</a></li> <li>Twitter: <a href="https://twitter.com/RadioFreePython" target="_blank">@RadioFreePython</a>]</li> <li>Subscribe: <a href="http://radiofreepython.com/high_quality.rss" target="_blank">RSS</a></li> </ul> </section><section><h2><a href="http://frompythonimportpodcast.com/" target="_blank">From python import podcast</a></h2> <blockquote> <p>A small-batch artisanal podcast for irreverent pythonistas. Easy-going, conversational, often silly, and occasionally earning our iTunes “explicit” tag, From Python Import Podcast is news, analysis, discussion, and general shenanigans about the Python language and community. Put on your headphones and come hang out with us!</p> </blockquote> <p>(No updates since 2014 but an interesting archive of episodes.)</p> <ul> <li>Website: <a href="http://frompythonimportpodcast.com/" target="_blank">frompythonimportpodcast.com</a></li> <li>Twitter: <a href="https://twitter.com/__fpip__" target="_blank">@__fpip__</a></li> <li>Subscribe: <a href="http://feeds.feedburner.com/FromPythonImportPodcastmp3" target="_blank">RSS</a> ⋅ <a href="http://itunes.apple.com/podcast/id525611633" target="_blank">iTunes</a> ⋅ <a href="https://overcast.fm/itunes525611633/" target="_blank">Overcast.fm</a></li> </ul> <p><em>If you think anything is missing from this list or if you’d like to see your own Python podcast added, then please email me at <a href="mailto:mail@dbader.org">mail@dbader.org</a>.</em></p></section><footer></footer></article>https://dbader.org/blog/ultimate-list-of-python-podcastsTue, 15 Nov 2016 00:00:00 GMTHow code linting will make you awesome at Pythonhttps://dbader.org/blog/python-code-linting<article><header><h1>How code linting will make you awesome at Python</h1> <p>In Python code reviews I’ve seen over and over that it can be tough for developers to format their Python code in a consistent way: extra whitespace, irregular indentation, and other “sloppiness” then often leads to actual bugs in the program.</p> </header><section><figure><img alt="" src="/blog/figures/python-code-linting-header.png" width="1280" height="719"></figure> <p>Luckily automated tools can help with this common problem. <em>Code linters</em> make sure your Python code is always formatted consistently – and their benefits go way beyond that.</p> </section><section><h2>What code linters can do for you</h2> <p>A <em>code linter</em> is a program that analyses your source code for potential errors. The kinds of errors a linter can detect include:</p> <ul> <li>syntax errors;</li> <li>structural problems like the use of undefined variables;</li> <li>best practice or code style guideline violations.</li> </ul> <p>I find code linting to be an indispensable productivity tool for writing Python. It’s possible to integrate linting into your editing environment. This gives you immediate feedback on your code right when you type it:</p> <figure><img alt="" src="/blog/figures/python-code-linting.jpg" width="958" height="220"></figure> <p>For some classes of errors, linting can shorten the usual <em>write code, run code, catch error, fix error</em> loop to <em>write code, see and fix error</em>. This difference might not seem much – but in the course of a day these time savings add up quickly and can have a huge impact on your productivity.</p> <p>In short, code linters are great!</p> </section><section><h2>Which Python linter should I use?</h2> <p>Python has several good options for code linters. The ones I’m listing here are available for free and are open-source software:</p> <ul> <li> <p><a href="https://pypi.python.org/pypi/flake8" target="_blank">Flake8</a> is my personal favorite these days. It’s fast and has a low rate of false positives. Flake8 is actually a combination of several other tools, mainly the <a href="https://pypi.python.org/pypi/pyflakes" target="_blank">Pyflakes</a> static analysis tool and the <a href="https://pypi.python.org/pypi/pycodestyle" target="_blank">Pycodestyle</a> (former pep8) code style checker.</p> </li> <li> <p><a href="https://pypi.python.org/pypi/pylint" target="_blank">Pylint</a> is another good choice. It takes a little more effort to set up than Flake8 and also triggers more false positives. On the other hand it provides a more comprehensive analysis. Definitely not a bad choice – but I’d stick with Flake8 if you’re just starting out.</p> </li> </ul> </section><section><h2>I’m sold – what’s the quickest way to get started?</h2> <p>If you’re not using a linter yet you’re missing out on some really great benefits. But don’t worry, I’ve got your back – I recorded a <a href="https://www.youtube.com/watch?v=TDUf93vqq3g" target="_blank">5 minute Python linting video tutorial</a> you can watch below.</p> <p>In the video I’ll give you the run down on how to set up the <a href="https://pypi.python.org/pypi/flake8" target="_blank">Flake8</a> Python linter from scratch. With a few simple steps you’ll be able run a code linter on your own Python programs. I’ll also demonstrate how linter feedback can be integrated with your code editor (I’m using <a href="/blog/tags/sublimetext">Sublime Text 3</a> in the video).</p> <p>I’ve seen great results from using linters. I believe they’re one of the quickest ways to improve your Python skills. Spend 5 minutes to try out Flake8 – I’m sure it’ll be well worth your time 😊</p> <p>Enjoy the video:</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/TDUf93vqq3g?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div></section><footer></footer></article>https://dbader.org/blog/python-code-lintingThu, 10 Nov 2016 00:00:00 GMTA Python refactoring gone wronghttps://dbader.org/blog/python-refactoring-gone-wrong<article><header><h1>A Python refactoring gone wrong</h1> <p>Ever witnessed a colleague make a refactoring to “clean up” some Python code only to make it worse and harder to understand?</p> </header><section><figure><img alt="" src="/blog/figures/python-refactoring-gone-wrong.png" width="1280" height="720"></figure> <p>I know I did. And I’ve also <em>been that colleague</em> to others many times 😊</p> <p>There’s often a fine line between making code better by “cleaning it up” and just shuffling around or even making it slightly worse. Refactoring is hard!</p> <p>It’s challenging to come up with good examples for this – so I was delighted when I got this Python question from Bev:</p> <p><br></p> <hr> <p><em>I came across something in Python that I’m having difficulty understanding. As part of a code rewrite in a Python related Youtube video, an “if” statement was changed to:</em></p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="nb">any</span><span class="p">([</span><span class="bp">self</span><span class="o">.</span><span class="n">temperature</span> <span class="o">&gt;</span> <span class="n">MAX_TEMPERATURE</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">pressure</span> <span class="o">&gt;</span> <span class="n">MAX_PRESSURE</span><span class="p">]):</span> </pre></div> <p><em>Why was that used rather than the simpler:</em></p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">temperature</span> <span class="o">&gt;</span> <span class="n">MAX_TEMPERATURE</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">pressure</span> <span class="o">&gt;</span> <span class="n">MAX_PRESSURE</span><span class="p">):</span> </pre></div> <p><em>Why create a list and call a function in the if statement when there are only two (2) comparisons?</em></p> <hr> <p><br></p> <p>I agree with Bev, this is a surprising change and I don’t think it’s for the better!</p> <p>It complicates the code for no apparent gain.</p> <p>Let’s take a look at the definition for the <code>any()</code> function first:</p> <blockquote> <p>Return True if any element of the iterable is true. If the iterable is empty, return False (Source: <a href="https://docs.python.org/3/library/functions.html#any" target="_blank">Python docs</a>)</p> </blockquote> <p><code>any()</code> – and its colleague, <code>all()</code> – are handy if you need to check the elements of an iterable (like a list or a generator) for truthiness.</p> <p>In some cases using <code>any()</code> can help avoid having to write a loop to check the elements of the iterable individually<sup id="fnref:short-circuit"><a class="footnote-ref" href="#fn:short-circuit" rel="footnote">1</a></sup>. Imagine you needed to do something like this:</p> <div class="codehilite"><pre><span></span><span class="n">result</span> <span class="o">=</span> <span class="bp">False</span> <span class="k">for</span> <span class="n">elem</span> <span class="ow">in</span> <span class="n">my_iterable</span><span class="p">:</span> <span class="k">if</span> <span class="n">elem</span><span class="p">:</span> <span class="n">result</span> <span class="o">=</span> <span class="bp">True</span> <span class="k">break</span> </pre></div> <p>You could replace these 5 lines with a simple assignment using the <code>any()</code> function:</p> <div class="codehilite"><pre><span></span><span class="n">result</span> <span class="o">=</span> <span class="nb">any</span><span class="p">(</span><span class="n">my_iterable</span><span class="p">)</span> </pre></div> <p>In this case it makes sense to go with the <code>any()</code> solution because it is shorter and easier to understand.</p> <p>Yet, too much of a good thing can make you sick… 😷</p> <p>In my mind it doesn’t make much sense to use <code>any()</code> if all you have is a fixed size list with 2-3 elements, like in the example Bev found:</p> <ul> <li> <p>Constructing a list so we have an iterable to pass to <code>any()</code> is confusing. It adds visual clutter and is a non-standard pattern.</p> </li> <li> <p>On top of that, using <code>any()</code> is also going to be slower than a nice and simple “or” expression: The Python interpreter first needs to construct that list and then call <code>any()</code> on it<sup id="fnref:memory"><a class="footnote-ref" href="#fn:memory" rel="footnote">2</a></sup>.</p> </li> </ul> <p>In summary, I think this refactoring was <em>not</em> helpful. I’d say it actually made the code worse by making it slightly harder to read and understand.</p> <p>As developers we need to be careful and deliberate about using the tools and patterns we know. “Fancy” often won’t equal “better”.</p> <p>Great question, Bev! I think you were spot-on in doubting this refactoring 😃</p> <div class="footnote"> <hr> <ol> <li id="fn:short-circuit"> <p>It’s good to note that <code>any()</code> has <a href="https://en.wikipedia.org/wiki/Short-circuit_evaluation" target="_blank">short-circuit evaluation semantics</a>, so using it is equivalent to a chain of <code>or</code> operations. (Thanks to <a href="https://axil.github.io/" target="_blank">Lev Maximov</a> for his feedback.) <a class="footnote-backref" href="#fnref:short-circuit" rev="footnote" title="Jump back to footnote 1 in the text">↩</a></p> </li> <li id="fn:memory"> <p>In all fairness, this won’t make a big real-world difference in performance or memory usage – but it’s always good to keep these things in mind. <a class="footnote-backref" href="#fnref:memory" rev="footnote" title="Jump back to footnote 2 in the text">↩</a></p> </li> </ol> </div></section><footer></footer></article>https://dbader.org/blog/python-refactoring-gone-wrongThu, 03 Nov 2016 00:00:00 GMTPython Code Review: Unplugged – Episode 3https://dbader.org/blog/python-code-review-unplugged-episode-3<article><header><h1>Python Code Review: Unplugged – Episode 3</h1> <p>In this third episode of my video code review series I take a look at a reader’s web scraping project and start adding some unit tests to it.</p> </header><section><div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/03Ki2WpVRBc?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p>This is a Python code review I did for Sunny’s web scraping project on GitHub. Sunny reached out to me after watching one of my previous code review videos, asking me if I could give him some feedback on his web scraping pet project.</p> <p>In this episode you’ll see Flake8 and Python code linting tools make a comeback. Also I’m doing an intro to adding Pytest unit tests to an existing Python code base in the second half of the video.</p> <p>By the way, I love how eager Sunny was to get feedback on his Python code:</p> <blockquote> <p>As a pet project I wanted to learn all the best practice as much as I could about python, trying to code as pythonic as I can, checking pep8 and autopep8 (didn’t knew about flake8!!) Eventually I re-factored using classes and methods. If you’ll see that it originally did everything in one monolithic function in the 2.x pull request on github. I still have a few things that I want to implement, so I welcome all kinds of feedback about it!</p> </blockquote> <p>This is exactly the right mindset that turns people into productive and successful software engineers. Even tiny – but constant – improvements add up and compound over time.</p> <p>I’ve seen this in friends and colleagues alike. Those developers who seek out constant small improvements eventually go on and do amazing things.</p> <p>Enjoy the video! And be sure to check out my other <a href="/screencasts">Python screencasts</a> if you liked this code review 😊</p> <p><strong>Links &amp; Resources</strong>:</p> <ul> <li><a href="https://github.com/sunnz" target="_blank">Sunny’s GitHub account</a></li> <li><a href="https://github.com/sunnz/auction-results.py/pull/2" target="_blank">The GitHub pull-request from the video</a></li> <li><a href="http://doc.pytest.org/en/latest/" target="_blank">Pytest Python testing framework</a></li> <li><a href="http://flake8.pycqa.org/en/latest/" target="_blank">Flake8 Python code linter</a></li> <li><a href="https://SublimeTextPython.com" target="_blank">How to integrate code linting with your editor setup</a></li> </ul> <p>» <a href="tags/code-review">Click here to watch my other Python Code Review: Unplugged videos</a></p></section><footer></footer></article>https://dbader.org/blog/python-code-review-unplugged-episode-3Wed, 26 Oct 2016 00:00:00 GMTClick & jump to any file or folder from the terminalhttps://dbader.org/blog/click-and-jump-to-any-file-or-folder-from-the-terminal<article><header><h1>Click &amp; jump to any file or folder from the terminal</h1> <p>iTerm2 for macOS has a little known feature that lets you <strong>open files and folders simply by Cmd+Clicking on them in the terminal</strong>. Among other things, this is super handy for debugging tests.</p> </header><section><figure><img alt="" src="/blog/figures/iterm-jump-to-file.gif" width="1280" height="720"></figure> <p>With this so called <em>Semantic History</em> feature you can configure <a href="https://www.iterm2.com/" target="_blank">iTerm2</a> to open folders and files in their default application when you press Cmd and then click on them.</p> <p>So if you click on a folder name it will open in the Finder, and if you click on a <code>.py</code> file, for example, it will open in your editor.</p> <p>The amazingly cool part is that this also works with line numbers, so if you click on something like <code>test_myapp.py:42</code> in the terminal your editor opens <code>test_myapp.py</code> and moves the cursor to line 42! 😀</p> <p>This unbelievably handy if you’re running your unit tests from the command line. I use it all the time to click and jump to failed test cases with the Pytest test runner, for example.</p> <p>Here’s how to set up <em>Semantic History</em> in iTerm2:</p> <ul> <li>Open the iTerm2 preferences by clicking on <strong>iTerm2 → Preferences</strong> in the menu bar (or press <strong>Cmd+,</strong>)</li> <li>Click on <strong>Profiles</strong> in the top row, then click <strong>Advanced</strong> all the way to the right. Find the section that says <strong>Semantic History</strong>.</li> <li>Under <strong>Semantic History</strong>, set the first option to <strong>Open with editor… </strong>and then pick your favorite editor (I use <strong>Sublime Text 3</strong>).</li> <li>Close the preferences window – that’s it!</li> </ul> <p>If you need some more help setting this up and a quick demo of what you can do with this feature, watch my video below:</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/Gw43Q0WZptI?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p>Like I said, I found this “click to jump to file” feature extremely helpful for working with tests.</p> <p>I usually run my Python tests with <a href="http://doc.pytest.org/en/latest/" target="_blank">Pytest</a> and it prints test failure messages in a format that iTerm2 understands. So I can simply Cmd+click on a failed test assertion and that’ll open up the test case Sublime Text, placing the cursor at the exact line that caused the test to fail.</p> <p>This feature should be completely language agnostic by the way. You’ll be able to use it with any test runner or programming language – and any editor.</p> <p><strong>P.S.</strong> Unfortunately iTerm2 is only available on macOS. I’d love to learn if there’s a way to get the same functionality on Windows or Linux, so far I haven’t been able to find anything. If you know how to do this on Linux or Windows please get in touch and tell me how to do it :) Thanks!</p></section><footer></footer></article>https://dbader.org/blog/click-and-jump-to-any-file-or-folder-from-the-terminalMon, 17 Oct 2016 00:00:00 GMTSublime Text plugin review: Djaneirohttps://dbader.org/blog/sublime-text-djaneiro-review<article><header><h1>Sublime Text plugin review: Djaneiro</h1> <p>A review of Djaneiro, a Sublime Text plugin for Django development.</p> </header><section><figure><img alt="" src="/blog/figures/djaneiro-snippets.gif" width="1280" height="720"></figure> <p>I’ll admit I was skeptical at first when a friend of mine recommended <a href="https://packagecontrol.io/packages/Djaneiro" target="_blank">Djaneiro</a> to enhance my Django development workflow in Sublime Text.</p> <p>I’d been happy with the <a href="/products/sublime-python-guide/">Python development setup</a> I built for myself over the years and I didn’t really understand what Djaneiro was going to add to that.</p> <p>But when I tried out Djaneiro I was impressed how helpful it turned out to be! I decided to write <a href="tags/plugin-review">another Sublime Text plugin review</a> to share my findings.</p> <p>Djaneiro’s main selling points are adding:</p> <ul> <li><strong>syntax highlighting for Django HTML templates</strong>; and</li> <li><strong>code completion snippets for Django HTML templates and Python files</strong>.</li> </ul> <p>In this review I’ll explain how Djaneiro can make your Django development workflow more productive and I’ll go over the pros and cons of the plugin as I experienced them. After that I’ll take a look at alternatives to Djaneiro in the Sublime Text plugin landscape. At the end I’ll share my final verdict and rating.</p> </section><section><h2>Pros</h2> <p><strong>Syntax highlighting for Django templates</strong>: In its default configuration Sublime Text doesn’t have syntax definitions for Django’s HTML templating syntax.</p> <p>This means that typos and syntax errors in templates are harder to catch visually. As you can see in the screenshot below (in the editing pane on the left), the standard HTML syntax highlighting in Sublime Text 3 uses a uniform white color for Django’s template tags.</p> <p>Djaneiro adds a <em>HTML (Django)</em> syntax that properly highlights Django’s template tags. As you can see in the right-hand editing pane in the screenshot, proper syntax highlighting makes these templates quite a bit easier to read. Also, syntax errors and typos stand out more due to the proper highlighting.</p> <p>This simple change adds a lot of value – I found that I was making fewer typos in my templates with Djaneiro’s syntax highlighting. Also, templates seemed easier to read and scan quickly with Djaneiro installed.</p> <figure><img alt="" src="/blog/figures/djaneiro-html-syntax.png" width="1516" height="588"></figure> <p><strong>Improved syntax highlighting for Django Python files</strong>: Djaneiro also makes some small tweaks to the default Python syntax highlighting. For example, it knows the standard Django settings constants like <code>INSTALLED_APPS</code> and highlights them differently so that they stand out more and typos are easier to find.</p> <p>This also happens for things like field definitions when writing Django model classes, which I found handy. In summary I found that the syntax highlighting changes introduced by Djaneiro make it easier to grasp the structure of the code I’m writing.</p> <figure><img alt="" src="/blog/figures/djaneiro-python-syntax.png" width="1680" height="669"></figure> <p><strong>Code completion snippets for Django templates and Python files</strong>: Another helpful feature provided by Djaneiro is a library of pre-made code completion snippets for common Django code and patterns.</p> <p>For example, you can insert an <code>{% if _____ %} {% endif %}</code> block by typing <code>if</code> as an abbreviation and hitting the auto-complete key (<em>Tab</em> by default). You can see a quick demo of that in the screenshot below. Generally, I found the <a href="https://packagecontrol.io/packages/Djaneiro" target="_blank">list of snippets included with Djaneiro</a> to be comprehensive and well-chosen.</p> <p>Besides snippets for Django HTML templates Djaneiro also includes a snippet library for Django Python code. These snippets let you quickly scaffold out whole view definitions or a barebones model classes, for example.</p> <p>Once you’ve gotten used to these snippets they can save you <em>a lot</em> of typing. Be sure to check out the full list of snippets in the <a href="https://github.com/squ1b3r/Djaneiro" target="_blank">Djaneiro README</a>.</p> <figure><img alt="" src="/blog/figures/djaneiro-snippets.gif" width="1280" height="720"></figure> </section><section><h2>Cons</h2> <p><strong>Snippets might get in the way</strong>: Because Djaneiro adds quite a substantial number of new code snippets I found myself triggering some of them accidentally, especially in the beginning. I really don’t want to hold this up against Djaneiro because the snippets do add a lot of value once I learned to use them well.</p> <p>It’s possible to <a href="http://stackoverflow.com/questions/27995926/how-do-i-disable-snippets-in-sublime-text-3" target="_blank">disable individual code snippets in Sublime Text</a> but unfortunately this process is a bit involved.</p> <p>If you find that the snippets get in your way occasionally you can temporarily switch them off by selecting a different syntax highlighting definition. Just open the Sublime Text <em>Command Palette</em>, type <em>Set syntax</em>, and select the default HTML or Python syntax.</p> </section><section><h2>Alternatives</h2> <p>There are a few more <a href="https://packagecontrol.io/search/django" target="_blank">Django-specific plugins available on Package Control</a> but Djaneiro seems to be the most popular and also the most powerful of the pack.</p> </section><section><h2>The verdict</h2> <p>I’ve grown quite fond of Djaneiro since I started using it. I immediately loved the improved syntax highlighting for Django templates and I’d say Djaneiro is worth installing for that feature alone.</p> <p>Once I’d gotten the hang of Djaneiro’s code snippets and their shortcodes I felt a noticeable improvement in my productivity. The snippets added by Djaneiro cover many things I encountered in day to day Django development. It’s great not having to go through the work of writing these snippets myself.</p> <p>I’d recommend any Django developer using Sublime Text to at least try out Djaneiro for a few days. There aren’t any substantial downsides to it and I’m sure it will make you more productive. You can <a href="https://packagecontrol.io/packages/Djaneiro" target="_blank">install Djaneiro via Package Control</a>.</p> <p>🐍 🐍 🐍 🐍 🐍 (5 out of 5 snakes)</p></section><footer></footer></article>https://dbader.org/blog/sublime-text-djaneiro-reviewTue, 11 Oct 2016 00:00:00 GMTThe Complete Guide to Setting up Sublime Text for Python Developers – Now Available!https://dbader.org/blog/python-sublime-text-setup-guide<article><header><h1>The Complete Guide to Setting up Sublime Text for Python Developers – Now Available!</h1> <p>Hey folks, I’m super excited to announce the launch of my first book – It’s called “The Complete Guide to Setting up Sublime Text for Python Developers”.</p> </header><section><div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/Vpv99Xmh3Gs?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p>It’s a detailed, step-by-step guidebook aimed at getting you to a kickass, professional-grade Python development setup built around Sublime Text in the shortest amount of time possible.</p> <p>I created this because I’ve been using Sublime Text for almost four years now in my Python workflow and I think it’s an <em>amazing</em> combo.</p> <p>However I kept getting so many emails and questions about this development setup when I used it in my screencasts.</p> <p>That made me realize how difficult it can be to set up an enjoyable Python development environment – and I decided to do something about it by writing the ULTIMATE setup guide for Sublime Text + Python 😃.</p> <p>If you want to become a better and more productive developer then this guide is really going to help you get more out of your Python workflow.</p> <p>Check out <strong><a href="https://SublimeTextPython.com" target="_blank">SublimeTextPython.com</a></strong> to see what it’s all about! Thanks so much for your support! Enjoy the guide and let me know what you think!</p></section><footer></footer></article>https://dbader.org/blog/python-sublime-text-setup-guideTue, 04 Oct 2016 00:00:00 GMTPython Code Review: Unplugged – Episode 2https://dbader.org/blog/python-code-review-unplugged-episode-2<article><header><h1>Python Code Review: Unplugged – Episode 2</h1> <p>This is the second episode of my video code review series where I record myself giving feedback and refactoring a reader’s Python code.</p> </header><section><figure><img alt="" src="/blog/figures/python-code-review-unplugged.png" width="1280" height="720"></figure> <p>The response to the <a href="/blog/live-python-code-review">first Code Review: Unplugged video</a> was super positive. I got a ton of emails and comments on YouTube saying that the video worked well as a teaching tool and that I should do more of them.</p> <p>And so I did just that 😃. <a href="https://twitter.com/ImmiltonPereira" target="_blank">Milton</a> sent me a link to his Python 3 project on GitHub and I recorded another code review based on his code. You can watch it below:</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/sarl7bi50xw?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p>Milton is on the right track with his Python journey. I liked how he used functions to split up his web scraper program into functions that each handle a different phase, like <em>fetch the html</em>, <em>parse it</em>, and <em>generate the output file</em>.</p> <p>The main thing that this code base could benefit from would be <strong>consistent formatting</strong>. Making the formatting as regular and consistent as possible really helps with keeping the “mental overhead” low when you’re working on the code or handing it off to someone else.</p> <p>And the beautiful thing is that there’s an easy fix for this, too. I demo a tool called <a href="http://flake8.pycqa.org/en/latest/" target="_blank">Flake8</a> in the video. Flake8 is a code linter and code style checker – and it’s great for making sure your code has consistent formatting and avoids common pitfalls or anti-patterns.</p> <p>You can even integrate Flake8 into your editing environment so that it checks your code as you write it.</p> <p>(<em>Shameless plug</em>: <a href="https://SublimeTextPython.com" target="_blank">The book I’m working</a> on has a whole chapter on integrating Flake8 into the Sublime Text editor. Check it out if you’d like to learn how to set up a Python development environment just like the one I’m using in the video).</p> <p>Besides formatting, the video also covers things like writing a great GitHub README, how to name functions and modules, and the use of constants to simplify your Python code. So be sure to watch the whole thing when you get the chance.</p> <p>Again, I left the video completely unedited. That’s why I’m calling this series <em>Code Review: Unplugged</em>. It’s definitely not a polished tutorial or course. But based on the feedback I got so far that seems to be part of the appeal.</p> <p><strong>Links &amp; Resources</strong>:</p> <ul> <li><a href="https://github.com/miltonpereira/Scrapper/pull/4" target="_blank">The pull request with the code from the video</a></li> <li><a href="http://flake8.pycqa.org/en/latest/" target="_blank">Flake8 Python code linter</a></li> <li><a href="https://twitter.com/ImmiltonPereira" target="_blank">Milton’s Twitter account</a></li> <li><a href="/blog/write-a-great-readme-for-your-github-project">How to write a great GitHub README</a></li> </ul> <p><strong>One more quick tip for you</strong>: You can turn these videos into a fun Python exercise for yourself. Just pause the video before I dig into the code and do your own code review first. Spend 10 to 20 minutes taking notes and refactoring the code and then continue with the video to compare your solution with mine. Let me know how this worked out! 😊</p> <p>» <a href="tags/code-review">Click here to watch my other Python Code Review: Unplugged videos</a></p></section><footer></footer></article>https://dbader.org/blog/python-code-review-unplugged-episode-2Tue, 27 Sep 2016 00:00:00 GMTHow do I make my own command line commands using Python?https://dbader.org/blog/how-to-make-command-line-commands-with-python<article><header><h1>How do I make my own command line commands using Python?</h1> <p>The Python script you just wrote would make a great little command line tool – but having to type “python myscript.py” all the time to launch your program gets daunting fast. Here’s how you can make your Python script feel like a <em>real</em> shell command.</p> </header><section><figure><img alt="" src="/blog/figures/python-cmdline-command.png" width="1280" height="719"></figure> <p>At the end of this short tutorial you’ll know how to write a Python program that simulates the Unix <code>echo</code> command and can be started from the command line just like it:</p> <div class="codehilite"><pre><span></span>$ myecho Hello, World! </pre></div> <p>Sweet, let’s jump right in!</p> <p>Imagine you have the following short Python script called <code>myecho.py</code> that just prints the command line arguments you pass to it back to the console:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span> <span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="n">arg</span><span class="p">)</span> </pre></div> <p>You can run this command just fine by passing it to the Python interpreter like so:</p> <div class="codehilite"><pre><span></span>$ python myecho.py Hello, World! myecho.py Hello, World! </pre></div> <p>But how can you give your users a more polished experience that allows them simply type <code>myecho Hello, World!</code> and get the same result?</p> <p>Easy – there are three things you need to do:</p> </section><section><h2>Step 1: Mark your Python file as executable</h2> <p>The first thing you’ll need to do is mark your script as executable in the file system, like so:</p> <div class="codehilite"><pre><span></span>$ chmod +x myecho.py </pre></div> <p>This sets the <em>executable</em> flag on <code>myecho.py</code>, which tells the shell that it’s a program that can be run directly from the command line. Let’s try it:</p> <div class="codehilite"><pre><span></span>$ ./myecho.py Hello, World! </pre></div> <p>We need to prefix our command with <code>./</code> because usually the current directory is not included in the <code>PATH</code> environment variable on Unix. This is a security feature<sup id="fnref:1"><a class="footnote-ref" href="#fn:1" rel="footnote">1</a></sup>.</p> <p>Anyway – the result will be that you get a crazy error message when you try to run <code>myecho.py</code>. It’ll probably look something like this:</p> <div class="codehilite"><pre><span></span>./myecho.py: line 4: syntax error near unexpected token `print' ./myecho.py: line 4: ` print(arg)' </pre></div> <p>The reason for that is that now the system doesn’t know it’s supposed to execute a <em>Python</em> script. So instead it takes a wild guess and tries to run your Python script like a shell script with the <code>/bin/sh</code> interpreter.</p> <p>That’s why you’re getting these odd syntax errors. But there’s an easy fix for this in the next step. You just need to …</p> </section><section><h2>Step 2: Add an interpreter “shebang”</h2> <p>Okay, admittedly this sounds completely crazy if you’ve never heard of Unix shebangs before…😃 But it’s actually a really simple concept and super useful:</p> <p>Whenever you run a script file on an Unix-like operating system (like Linux or macOS) the program loader responsible for loading and executing your script checks the first line for an <em>interpreter directive</em>. Here’s an example:</p> <div class="codehilite"><pre><span></span><span class="ch">#!/bin/sh</span> </pre></div> <p>You’ve probably seen those before. These interpreter directives are also called <em>shebangs</em> in Unix jargon<sup id="fnref:2"><a class="footnote-ref" href="#fn:2" rel="footnote">2</a></sup>. They tell the program loader which interpreter should execute the script.</p> <p>You can use this mechanism to your advantage by adding a shebang line that points to the system Python interpreter:</p> <div class="codehilite"><pre><span></span><span class="ch">#!/usr/bin/env python</span> </pre></div> <p>You may be wondering why you should be using <code>env</code> to load the Python interpreter instead of simply using an absolute path like <code>/usr/local/bin/python</code>.</p> <p>The reason for that is that the Python interpreter will be installed in different locations on different systems. On a Mac using <a href="https://brew.sh/" target="_blank">Homebrew</a> it might be in <code>/usr/local/bin/python</code>. On a Ubuntu Linux box it might be in <code>/usr/bin/python</code>.</p> <p>Using another level of indirection through <code>env</code> you can select the Python interpreter that’s on the <code>PATH</code> environment variable. That’s usually the right way to go about it<sup id="fnref:3"><a class="footnote-ref" href="#fn:3" rel="footnote">3</a></sup>.</p> <p>Okay, so now that you’ve added that <code>#!/usr/bin/env python</code> line your script should look like this:</p> <div class="codehilite"><pre><span></span><span class="ch">#!/usr/bin/env python</span> <span class="kn">import</span> <span class="nn">sys</span> <span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="n">arg</span><span class="p">)</span> </pre></div> <p>Let’s try to run it again!</p> <div class="codehilite"><pre><span></span>$ ./myecho.py Hello, World! ./myecho.py Hello, World! </pre></div> <p>Yes! Success!</p> <p>Now that you’re using the interpreter directive shebang in the script you can also drop the <code>.py</code> extension. This will make your script look even more like a system tool:</p> <div class="codehilite"><pre><span></span>$ mv myecho.py myecho </pre></div> <p>This is starting to look pretty good now:</p> <div class="codehilite"><pre><span></span>$ ./myecho Hello, World! ./myecho Hello, World! </pre></div> </section><section><h2>Step 3: Make sure your program is on the PATH</h2> <p>The last thing you need to change to make your Python script really seem like a shell command or system tool is to make sure it’s on your <code>PATH</code>.</p> <p>That way you’ll be able to launch it from any directory by simply running <code>myecho Hello, World!</code>, just like the “real” <code>echo</code> command.</p> <p>Here’s how to achieve that.</p> <p>I don’t recommend that you try to copy your script to a system directory like <code>/usr/bin/</code> or <code>/usr/local/bin</code> because that can lead to all kinds of odd naming conflicts (and, in the worst case, break your operating system install).</p> <p>So instead, what you’ll want to do is to create a <code>bin</code> directory in your user’s home directory and then add that to the <code>PATH</code>.</p> <p>First, you need to create the <code>~/bin</code> directory:</p> <div class="codehilite"><pre><span></span>$ mkdir -p ~/bin </pre></div> <p>Next, copy your script to <code>~/bin</code>:</p> <div class="codehilite"><pre><span></span>$ cp myecho ~/bin </pre></div> <p>Finally, add <code>~/bin</code> to your <code>PATH</code>:</p> <div class="codehilite"><pre><span></span><span class="nb">export</span> <span class="nv">PATH</span><span class="o">=</span><span class="nv">$PATH</span><span class="s2">":</span><span class="nv">$HOME</span><span class="s2">/bin"</span> </pre></div> <p>Adding <code>~/bin</code> to the <code>PATH</code> like this is only temporary, however. It won’t stick across terminal sessions or system restarts. If you want to make your command permanently available on a system, do the following:</p> <ul> <li>Add the this line to <code>.profile</code> or <code>.bash_profile</code> in your home directory: <code>export PATH=$PATH":$HOME/bin"</code>.</li> <li>You can either use an editor to do it or run the following command to do that: <code>echo 'export PATH=$PATH":$HOME/bin"' &gt;&gt; .profile</code></li> <li>Changes to <code>.profile</code> or <code>.bash_profile</code> only go into effect when your shell reloads these files. You can trigger a reload by either opening a new terminal window or running this command: <code>source .profile</code></li> </ul> <p>Okay great, now we finally get the result we wanted all along:</p> <div class="codehilite"><pre><span></span>$ myecho Hello, World! /Users/youruser/bin/myecho Hello, World! </pre></div> <p>Whew. When I write these tutorials I’m always surprised <em>how much work seemingly “simple” things take</em> when you’re trying to get to the bottom of them. So don’t be too hard on yourself if some of these steps felt a little arcane at first 😃. All of this stuff becomes second nature once you’ve dealt with it a few times.</p> <div class="footnote"> <hr> <ol> <li id="fn:1"> <p>Here’s an awesome in-depth explanation for this security feature and why it is necessary: <a href="http://www.linfo.org/dot_slash.html" target="_blank">www.linfo.org/dot_slash.html</a> <a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text">↩</a></p> </li> <li id="fn:2"> <p>You can read all about Unix shebangs here: <a href="https://en.wikipedia.org/wiki/Shebang_(Unix)" target="_blank">wikipedia.org/wiki/Shebang_(Unix)</a> <a class="footnote-backref" href="#fnref:2" rev="footnote" title="Jump back to footnote 2 in the text">↩</a></p> </li> <li id="fn:3"> <p>Learn more about <code>env</code> and its merits here: <a href="https://en.wikipedia.org/wiki/Env" target="_blank">wikipedia.org/wiki/Env</a> <a class="footnote-backref" href="#fnref:3" rev="footnote" title="Jump back to footnote 3 in the text">↩</a></p> </li> </ol> </div></section><footer></footer></article>https://dbader.org/blog/how-to-make-command-line-commands-with-pythonFri, 23 Sep 2016 00:00:00 GMTUsing get() to return a default value from a Python dicthttps://dbader.org/blog/python-dict-get-default-value<article><header><h1>Using get() to return a default value from a Python dict</h1> <p>Python’s dictionaries have a “get” method to look up a key while providing a fallback value. This short screencast tutorial gives you a real-world example where this might come in handy.</p> </header><section><div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/y4LXXpekUxs?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p>Imagine we have the following data structure mapping user IDs to user names:</p> <div class="codehilite"><pre><span></span><span class="n">name_for_userid</span> <span class="o">=</span> <span class="p">{</span> <span class="mi">382</span><span class="p">:</span> <span class="s2">"Alice"</span><span class="p">,</span> <span class="mi">950</span><span class="p">:</span> <span class="s2">"Bob"</span><span class="p">,</span> <span class="mi">590</span><span class="p">:</span> <span class="s2">"Dilbert"</span><span class="p">,</span> <span class="p">}</span> </pre></div> <p>Now we’d like to write a function <code>greeting()</code> which returns a greeting for a user given their user ID. Our first implementation might look something like this:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">greeting</span><span class="p">(</span><span class="n">userid</span><span class="p">):</span> <span class="k">return</span> <span class="s2">"Hi </span><span class="si">%s</span><span class="s2">!"</span> <span class="o">%</span> <span class="n">name_for_userid</span><span class="p">[</span><span class="n">userid</span><span class="p">]</span> </pre></div> <p>This implementation works if the user ID is a valid key in <code>name_for_userid</code>, but it throws an exception if we pass in an invalid user ID:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">greeting</span><span class="p">(</span><span class="mi">382</span><span class="p">)</span> <span class="s2">"Hi Alice!"</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greeting</span><span class="p">(</span><span class="mi">33333333</span><span class="p">)</span> <span class="ne">KeyError</span><span class="p">:</span> <span class="mi">33333333</span> </pre></div> <p>Let’s modify our greeting function to return a default greeting if the user ID cannot be found. Our first idea might be to simply do a “key in dict” membership check:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">greeting</span><span class="p">(</span><span class="n">userid</span><span class="p">):</span> <span class="k">if</span> <span class="n">userid</span> <span class="ow">in</span> <span class="n">name_for_userid</span><span class="p">:</span> <span class="k">return</span> <span class="s2">"Hi </span><span class="si">%s</span><span class="s2">!"</span> <span class="o">%</span> <span class="n">name_for_userid</span><span class="p">[</span><span class="n">userid</span><span class="p">]</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="s2">"Hi there!"</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greeting</span><span class="p">(</span><span class="mi">382</span><span class="p">)</span> <span class="s2">"Hi Alice!"</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greeting</span><span class="p">(</span><span class="mi">33333333</span><span class="p">)</span> <span class="s2">"Hi there!"</span> </pre></div> <p>While this implementation gives us the expected result, it isn’t great:</p> <ul> <li>it’s <em>inefficient</em> because it queries the dictionary twice</li> <li>it’s <em>verbose</em> as part of the greeting string are repeated, for example</li> <li>it’s not <em>pythonic</em> – the official Python documentation recommends an “easier to ask for forgiveness than permission” (EAFP) coding style:</li> </ul> <blockquote> <p>“This common Python coding style assumes the existence of valid keys or attributes and catches exceptions if the assumption proves false.” (<a href="https://docs.python.org/3/glossary.html" target="_blank">Python glossary: “EAFP”</a>)</p> </blockquote> <p>Therefore a better implementation that follows <em>EAFP</em> could use a <em>try…except</em> block to catch the <code>KeyError</code> instead of doing a membership test:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">greeting</span><span class="p">(</span><span class="n">userid</span><span class="p">):</span> <span class="k">try</span><span class="p">:</span> <span class="k">return</span> <span class="s2">"Hi </span><span class="si">%s</span><span class="s2">!"</span> <span class="o">%</span> <span class="n">name_for_userid</span><span class="p">[</span><span class="n">userid</span><span class="p">]</span> <span class="k">except</span> <span class="ne">KeyError</span><span class="p">:</span> <span class="k">return</span> <span class="s2">"Hi there"</span> </pre></div> <p>Again, this implementation would be correct – but we can come up with a <em>cleaner</em> solution still! Python’s dictionaries have a <a href="https://docs.python.org/3/library/stdtypes.html#dict.get" target="_blank"><code>get()</code> method</a> on them which supports a default argument that can be used as a fallback value:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">greeting</span><span class="p">(</span><span class="n">userid</span><span class="p">):</span> <span class="k">return</span> <span class="s2">"Hi </span><span class="si">%s</span><span class="s2">!"</span> <span class="o">%</span> <span class="n">name_for_userid</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">userid</span><span class="p">,</span> <span class="s2">"there"</span><span class="p">)</span> </pre></div> <p>When <code>get()</code> is called it checks if the given key exists in the dict. If it does, the value for that key is returned. If it does <em>not</em> exist then the value of the <em>default</em> argument is returned instead.</p> <p>As you can see, this implementation of <code>greeting</code> works as intended:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">greeting</span><span class="p">(</span><span class="mi">950</span><span class="p">)</span> <span class="s2">"Hi Bob!"</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">greeting</span><span class="p">(</span><span class="mi">333333</span><span class="p">)</span> <span class="s2">"Hi there!"</span> </pre></div> <p>Our final implementation of <code>greeting()</code> is concise, clean, and only uses features from the Python standard library. Therefore I believe it is the best solution for this particular situation.</p> <p><em>P.S. If you enjoyed this screencast and you’d like to see more just like it then subscribe to my » <a href="/screencasts">YouTube channel with free screencasts and video tutorials for Python developers</a> «</em></p></section><footer></footer></article>https://dbader.org/blog/python-dict-get-default-valueWed, 14 Sep 2016 00:00:00 GMTWatch me do a “live” Python code review for a readerhttps://dbader.org/blog/live-python-code-review<article><header><h1>Watch me do a “live” Python code review for a reader</h1> <p>This is a bit of an experiment – but you might find it interesting!</p> </header><section><figure><img alt="" src="/blog/figures/python-code-review-unplugged.png" width="1280" height="720"></figure> <p>A couple of days ago I had a Twitter conversation with Labeeb who’s just getting into Python. (Good news, so far he loves it!)</p> <p>I think we started out with a classic “Emacs vs Sublime” discussion (😂) until Labeeb mentioned he was struggling with some aspects of Object-Oriented Programming in Python.</p> <p>I asked him to send me some sample code and offered to take a look and give him some feedback.</p> <p>He later emailed me an implementation of Conway’s Game of Life (that’s a great coding exercise by the way).</p> <p>After taking a quick look at his code, I decided the best way forward would be to <strong>record myself doing a code review pass and to send Labeeb the screencast recording</strong>.</p> <p>So I did just that. And it turned out to be a pretty… interesting experience. I was happy to hear that Labeeb liked the video:</p> <blockquote> <p>“The video is extremely helpful. The way you present things was very much appealing. :)”</p> </blockquote> <p>That made me feel all warm and fuzzy inside. I thought I was onto something… and that this video could help other Python developers, too.</p> <p>(After asking Labeeb for permission) <strong>I’m now sharing the raw and unfiltered “live” code review video with you here</strong>:</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/Tf70szZwWgA?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p>Note that the video is completely unedited. This is really more of a <em>”Code Review: Unplugged”</em> session than a polished tutorial or course. But based on the feedback I got so far that seems to be part of the appeal, haha.</p> <p>I figured it was worth the experiment if there’s a chance it’ll help more people… Shoot me a quick email and let me know how you liked it, I might do more of these in the future if it’s helpful.</p> <p><strong>P.S.</strong> Labeeb is currently doing a programming mentorship program. He’s looking to get a job as a data analyst working with Python. You can check out his GitHub here: <a href="https://github.com/labeebee" target="_blank">github.com/labeebee</a> Give him some love, I think he’s on the right track 😊</p> <p><strong>P.P.S.</strong> This has now turned into a series. <a href="tags/code-review">Click here to watch my other Python Code Review: Unplugged videos</a>.</p></section><footer></footer></article>https://dbader.org/blog/live-python-code-reviewThu, 08 Sep 2016 00:00:00 GMTWhat books should I read to move past the beginner stage in Python?https://dbader.org/blog/python-intermediate-level-books<article><header><h1>What books should I read to move past the beginner stage in Python?</h1> <p>Recommendations for intermediate-level Python books that help you get past the basics so you can start working on small projects.</p> </header><section><figure><img alt="" src="/blog/figures/intermediate-python-books.png" width="1280" height="721"></figure> <p>I want to answer a question that I got on Twitter the other day:</p> <p>Someone asked for book recommendations to move past the “beginner” stage in Python – The person was looking for intermediate-level books that would help them get past the basics so they could improve their skills by working on small projects.</p> <p>Let me start by saying that I really like this approach to learning a new programming language!</p> <p>It’s a good idea to start working on real projects as soon as possible, even if they’re small. There’s only so much you can learn from repeated <em>let’s implement this algorithm</em> exercises.</p> <p>These are the books I recommended:</p> <ul> <li> <p><a href="http://amzn.to/2cSSPKf" target="_blank">Automate the Boring Stuff with Python</a> by Al Sweigart has some great “project like” exercises. It covers common real world tasks like web scraping or filling out online forms. This really helps keep your motivation up and getting a sense of accomplishment. The book is <a href="https://automatetheboringstuff.com/" target="_blank">free to read online</a> under a Creative Commons license (but you can buy a copy to support Al).</p> </li> <li> <p><a href="http://amzn.to/2cok5MI" target="_blank">Effective Python</a> by Brett Slatkin is also a great book that will help take your Python skills to the next level. It focuses on teaching you to write more pythonic code and learning the community best practices, without running the danger of overusing some of Python’s more arcane features to the detriment of your code. It’s all about hitting that sweet spot and Brett teaches this lesson well!</p> </li> <li> <p><a href="http://amzn.to/2c5X8eY" target="_blank">Fluent Python</a> by Luciano Ramalho is intended as a hands-on guide covering the features that make Python special. I like how Luciano focuses on teaching the <em>pythonic</em> way to do things, which helps if you’re trying to “unlearn” patterns you’ve picked up from working with other languages. (<a href="http://www.snowboardingcoder.com/django/" target="_blank">Jim Anderson</a> emailed me to recommend this book. Thanks Jim!)</p> </li> <li> <p><a href="http://amzn.to/2cig6VV" target="_blank">Python Cookbook, 3rd Ed.</a> by David Beazley and Brian Jones is more project-based again. It’s chock-full of recipes for common tasks across various application domains like data processing or network programming. This is probably the most advanced-level book of the three, covering topics like metaprogramming. But there’s just so much information in there that I’m sure you’ll learn something useful from it even with beginner-level Python skills.</p> </li> </ul> <p>I hope that helped you out!</p> <p><strong>P.S.</strong> What are <em>your</em> favorite books and resources for moving from junior/entry-level Python to intermediate and beyond? I’m thinking about writing a longer article about this topic and would love to hear about your best resources and learning strategies. Leave a comment below if you’ve got a minute!</p> <p><strong>Update (2017):</strong> I wrote my own Python book for intermediate developers looking to write clean and Pythonic code. This is a bit of shameless plug, but if you like this list I’m sure you’ll love <a href="/products/python-tricks-book/">Python Tricks: The Book – A Buffet of Awesome Python Features</a>.</p></section><footer></footer></article>https://dbader.org/blog/python-intermediate-level-booksWed, 31 Aug 2016 00:00:00 GMTMake your Python code more readable with custom exception classeshttps://dbader.org/blog/python-custom-exceptions<article><header><h1>Make your Python code more readable with custom exception classes</h1> <p>In this short screencast I’ll walk you through a simple code example that demonstrates how you can use custom exception classes in your Python code to make it easier to understand, easier to debug, and more maintainable.</p> </header><section><div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/hLLaw9BI-EE?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p>Let’s say we want to validate an input string representing a person’s name in our application. A simple toy example might look like this:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">validate</span><span class="p">(</span><span class="n">name</span><span class="p">):</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="o">&lt;</span> <span class="mi">10</span><span class="p">:</span> <span class="k">raise</span> <span class="ne">ValueError</span> </pre></div> <p>If the validation fails it throws a <code>ValueError</code>. That feels kind of Pythonic already… We’re doing great!</p> <p><strong>However, there’s one downside to this piece of code</strong>: Imagine one of our teammates calls this function as part of a library and doesn’t know much about its internals.</p> <p>When a name fails to validate it’ll look like this in the debug stacktrace:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">validate</span><span class="p">(</span><span class="s1">'joe'</span><span class="p">)</span> <span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span> <span class="n">File</span> <span class="s2">"&lt;input&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o">&lt;</span><span class="n">module</span><span class="o">&gt;</span> <span class="n">validate</span><span class="p">(</span><span class="s1">'joe'</span><span class="p">)</span> <span class="n">File</span> <span class="s2">"&lt;input&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">3</span><span class="p">,</span> <span class="ow">in</span> <span class="n">validate</span> <span class="k">raise</span> <span class="ne">ValueError</span> <span class="ne">ValueError</span> </pre></div> <p><strong>This stacktrace isn’t really all that helpful.</strong> Sure, we know that something went wrong and that the problem has to do with an “incorrect value” of sorts.</p> <p>But to be able to fix the problem our teammate almost certainly has to look up the implementation of <code>validate()</code>. But reading code costs time. And it adds up quickly…</p> <p><strong>Luckily we can do better!</strong> Let’s introduce a custom exception type for when a name fails validation. We’ll base our new exception class on Python’s built-in <code>ValueError</code>, but make it more explicit by giving it a different name:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">NameTooShortError</span><span class="p">(</span><span class="ne">ValueError</span><span class="p">):</span> <span class="k">pass</span> <span class="k">def</span> <span class="nf">validate</span><span class="p">(</span><span class="n">name</span><span class="p">):</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="o">&lt;</span> <span class="mi">10</span><span class="p">:</span> <span class="k">raise</span> <span class="n">NameTooShortError</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> </pre></div> <p>See how we’re passing <code>name</code> to the constructor of our custom exception class when we instantiate it inside <code>validate</code>? <strong>The updated code results in a much nicer stacktrace for our teammate</strong>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">validate</span><span class="p">(</span><span class="s1">'jane'</span><span class="p">)</span> <span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span> <span class="n">File</span> <span class="s2">"&lt;input&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o">&lt;</span><span class="n">module</span><span class="o">&gt;</span> <span class="n">validate</span><span class="p">(</span><span class="s1">'jane'</span><span class="p">)</span> <span class="n">File</span> <span class="s2">"&lt;input&gt;"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">3</span><span class="p">,</span> <span class="ow">in</span> <span class="n">validate</span> <span class="k">raise</span> <span class="n">NameTooShortError</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="n">NameTooShortError</span><span class="p">:</span> <span class="n">jane</span> </pre></div> <p>Now, imagine <em>you</em> are the teammate we were talking about… Even if you’re working on a code base by yourself, custom exception classes will make it easier to understand what’s going on when things go wrong. A few weeks or months down the road you’ll have a much easier time maintaining your code. I’ll vouch for that 😃</p> <p><em>P.S. If you enjoyed this screencast and you’d like to see more just like it then subscribe to my » <a href="/screencasts">YouTube channel with free screencasts and video tutorials for Python developers</a> «</em></p></section><footer></footer></article>https://dbader.org/blog/python-custom-exceptionsThu, 18 Aug 2016 00:00:00 GMTSublime Text for Python development — My 2016 reviewhttps://dbader.org/blog/sublime-text-for-python-development-2016-review<article><header><h1>Sublime Text for Python development — My 2016 review</h1> <p>When you ask for editor recommendations as a Python developer one of the top choices you’ll hear about is Sublime Text. In this post I’ll review the status of Python development with Sublime Text as of 2016.</p> </header><section><figure><img alt="" src="/blog/figures/sublime-text-2016-header.png" width="1741" height="663"></figure> <p><em>Disclaimer</em>: I’m a fan and long-time user of Sublime Text. I used a number of editors and IDEs for writing Python including PyCharm, IntelliJ with Python plugins, Atom, Visual Studio Code, BBEdit, and emacs. I’ve worked with colleagues who are big Vim proponents. And while I never got the hang of Vim, I feel like I’m qualified to give Sublime Text a relatively unbiased review. You’ve been warned though 😃.</p> </section><section><h2>What I like about Sublime Text</h2> <figure><img alt="" src="/blog/figures/sublime-text-2016-pros.png" width="1734" height="404"></figure> <ul> <li> <p><strong>Performance</strong>: Sublime is one of the fastest editors available. <a href="https://pavelfatin.com/typing-with-pleasure/" target="_blank">Pavel Fatin compared typing latencies between several popular editors</a> and Sublime Text is consistently among the fastest and most responsive ones in his list. My (unscientific) personal impression comparing Sublime with similar editors like Atom or VS Code confirms this. Also note that Sublime starts up super fast. I don’t restart my development environment too often, but when I do it’s nice to be back up and running within a few seconds — rather than waiting half a minute for a ginormous IDE to boot up.</p> </li> <li> <p><strong>Stability and reliability</strong>: I’ve been using Sublime as my main editor for almost four years and it’s always been rock solid for me in terms of stability. I don’t think I’ve ever lost any data due to a crash or some other issue. I think that’s impressive. I like my tools to be reliable.</p> </li> <li> <p><strong>Plugin ecosystem</strong>: Something that’s drawn me towards Sublime is its fantastic community that wrote thousands of plugins for it. That way you can build a custom editor setup that does exactly what you want and how you want it. Several fantastic packages for Python development are available. I’ve reviewed some of them here: <a href="tags/plugin-review">Sublime Text Plugin Reviews</a>.</p> </li> <li> <p><strong>Package Control</strong>: Sublime Text has <em>Package Control</em> which is a plugin manager that let’s you install and uninstall other plugins directly from within the editor. It’s kind of a “meta plugin” that makes tinkering with your setup super easy. Package Control comes with a <a href="https://packagecontrol.io/browse" target="_blank">directory of available plugins</a> which makes it easy to pick out the good ones based on popularity and recent activity.</p> </li> <li> <p><strong>Plugins are written in Python</strong>: Most Sublime plugins are written in Python. Sublime Text includes an embedded Python interpreter that’s used to run the plugin code. It’s nice being able to look under the hood and read through a plugin’s code to judge its quality. If you’re a Python developer and you’re interested in writing your own Sublime Text package then that’s also a bonus.</p> </li> <li> <p><strong>It’s pretty</strong>: There’s a wide variety of themes for Sublime Text available which allows you to set up the look and feel of your editor to your liking. On top of that, Sublime’s font rendering is excellent. I’m peculiar about the way my editor looks. If I’m going to be staring at this thing for several hours each day then it better be as pretty as it can be 😀. I found Sublime Text to be easier to “prettify” than other editors.</p> </li> <li> <p><strong>Soft learning curve</strong>: Compared to some other editing environments like Vim or Emacs, Sublime Text has a soft learning curve. This is great for beginners. In my experience it’s difficult to be successful with Vim or Emacs without going all-in and spending at least a few weeks or months learning the system. Sublime Text is much easier to pick up in comparison.</p> </li> <li> <p><strong>UI state restoration</strong>: Sublime Text remembers the state of your editor windows when you shut it down so that when you restart Sublime everything looks the way you left it, including modified or unsaved files. This feature is brilliant! I haven’t seen anything quite like it and it’s something that discouraged me from using Atom, for example. I often use new editor tabs as scratchpads for notes. And while those are temporary it’s nice not having to worry about losing them due to an editor crash or restart.</p> </li> <li> <p><strong>Multiple cursors</strong>: Like some other editors Sublime supports editing with multiple cursors at the same time. This is super handy when you want to rename a local variable, for example. Select the variable, hit <code>cmd+d</code> a couple of times to select all other occurrences and then type the new name. Done. The same approach works in other situations like re-formatting a several lines of code at once or cutting out parts from a log file.</p> </li> <li> <p><strong>Cross-platform</strong>: Sublime Text is available for Mac, Linux, and Windows. It’s nice being able to use a familiar editing environment across multiple platforms.</p> </li> <li> <p><strong>Handles large files</strong>: Sublime is good at dealing with large files, like an occasional giant CSV file or a log file you want to take a look at in a familiar environment. I like not having to switch to other tools (like <code>less</code>) for that job, knowing Sublime will handle the file just fine and won’t freeze or crash. Atom dealt with the same files much less gracefully. It often freezed for seconds at a time or even crashed.</p> </li> <li> <p><strong>Fast global search</strong>: Sublime’s global text search is fast. I find it comparable to tools like <a href="http://beyondgrep.com/" target="_blank">ack</a>, which is nice because that means I have to switch to the command line less. Sublime also indexes your source files and has a <em>Goto Symbol in Project</em> command that let’s you quickly jump to specific identifiers, functions, or classes. This feature is aware of Python’s syntax so it’s usually accurate.</p> </li> <li> <p><strong>Command palette</strong>: I’m bad at remembering keyboard shortcuts for commands I use infrequently. Sublime’s solution to that problem is the <em>Command Palette</em>. You can open it with <code>cmd+p</code> and find what you’re looking for with a fuzzy text search. Let’s say I want to rename a file and I can’t remember the keyboard shortcut for that – what I’ll do is open the Command Palette and type <code>ren</code> to select the <em>File: Rename</em> command and then hit <code>return</code>. Boom, this let’s me rename a file without ever moving my hands away from the keyboard – and without having to remember some arcane shortcut. This feature is a great time saver!</p> </li> </ul> </section><section><h2>Things I dislike</h2> <figure><img alt="" src="/blog/figures/sublime-text-2016-cons.png" width="1734" height="403"></figure> <ul> <li> <p><strong>Can be difficult to set up for a beginner</strong>: While using Sublime Text the way it comes out of the box is okay, getting most of the good stuff requires spending some time. It’s not as simple as installing an IDE like PyCharm that comes with batteries included. On the other hand, you can start with a simple setup using Sublime. Then simply add more plugins and custom configurations over time to turn it into a completely personalized tool.</p> </li> <li> <p><strong>It’s not free</strong>: I was on the fence about adding this point because I believe in paying for the tools that allow me to do my job better. I realize though that some people might find a free solution (like Atom, emacs, or vim) more attractive.</p> </li> <li> <p><strong>Not open-source, “bus factor”</strong>: Many of the Sublime Text alternatives are open-source which makes them more future proof. Sublime Text is developed by just one developer, ex-Googler Jon Skinner. And while Jon is clearly a genius and great at what he’s doing, it’s an open question what would happen if Jon decided (or was forced) to halt development of Sublime Text. Would the project just disappear? Would he be able to <a href="https://en.wikipedia.org/wiki/TextMate" target="_blank">pull a TextMate</a> and open-source the project? What if he decides to sell Sublime Text to a company and they do a bad job maintaining it? Essentially, one of the biggest problems with Sublime Text is that it has a bad <a href="https://en.wikipedia.org/wiki/Bus_factor" target="_blank">bus factor</a> — there’s just one developer working on it and its source code is not publicly available. Of course I hope the best for Jon and Sublime Text. My perspective on this issue is that I chose not to worry about it — I’d rather use the best tool for the job <em>now</em> than waste time trying to future proof my setup. If it doesn’t work out I can always switch later. (<strong>Edit</strong>: Will Bond, the creator of Package Control, <a href="https://www.sublimetext.com/blog/articles/sublime-text-3-build-3103" target="_blank">joined the ST team in February 2016</a>. This makes Sublime Text’s long-term survival more likely. But it still has a comparatively small team behind it and isn’t open-source like some of the alternatives. If ST ever stops being maintained we’ll probably see open-source reimplementations of the core editor functionality. There’s already projects like <a href="http://limetext.org/" target="_blank">Lime Text</a>, an open-source editor that aims to be compatible with Sublime’s plugin API.)</p> </li> <li> <p><strong>No great solution for “semantic auto-complete”</strong>: While there are packages that offer <a href="https://en.wikipedia.org/wiki/Intelligent_code_completion#IntelliSense" target="_blank">IntelliSense-like</a> code completion, those I’ve tried weren’t satisfying. Due to the dynamic nature of Python as a language it will be difficult to get to the point where the auto-complete works as well as it does for Java in IntelliJ or for C# in Visual Studio. So it’s difficult to strike this up against Sublime Text and it’s plugin ecosystem. However, if you’re relying on a feature like that then it may be worth trying out the PyCharm IDE. I found it’s implementation of Python auto-complete the most promising. (<strong>Update</strong>: I’ve done more research on Python code completion with Sublime Text and after trying out several plugins I think the <a href="https://packagecontrol.io/packages/Anaconda" target="_blank">Anacoda plugin</a> is the best solution. Configured correctly its auto-complete rivals that of PyCharm. I’m now happily using Anaconda in <a href="/products/sublime-python-guide/">my Python development workflow</a>.)</p> </li> </ul> </section><section><h2>Conclusion</h2> <figure><img alt="" src="/blog/figures/sublime-text-2016-conclusion.png" width="1734" height="403"></figure> <p>All things considered I believe Sublime Text is still the top editor choice for Python development. I haven’t found an alternative that would make me want to switch.</p> <p>In my mind Sublime Text offers the best combination of performance, stability, and ergonomics. With some tuning it can look attractive, too. It does everything I want out of my programming environment and has been a central tool for me over more than three years.</p> <p><em>By the way, if you’re looking for help setting up Sublime Text for Python development then check out this tutorial I wrote: » <a href="setting-up-sublime-text-for-python-development">Setting up Sublime Text for Python development</a> «</em></p></section><footer></footer></article>https://dbader.org/blog/sublime-text-for-python-development-2016-reviewTue, 09 Aug 2016 00:00:00 GMTHow to use Python’s min() and max() with nested listshttps://dbader.org/blog/python-min-max-and-nested-lists<article><header><h1>How to use Python’s min() and max() with nested lists</h1> <p>Let’s talk about using Python’s <code>min</code> and <code>max</code> functions on a list containing other lists. Sometimes this is referred to as a <em>nested list</em> or a <em>lists of lists</em>.</p> </header><section><figure><img alt="" src="/blog/figures/min-max-nested-lists.png" width="1280" height="720"></figure> <p>Finding the minimum or maximum element of a list of lists<sup id="fnref:1"><a class="footnote-ref" href="#fn:1" rel="footnote">1</a></sup> based on a <em>specific property of the inner lists</em> is a common situation that can be challenging for someone new to Python.</p> <p>To give us a more concrete example to work with, let’s say we have the following list of <em>item, weight</em> pairs<sup id="fnref:2"><a class="footnote-ref" href="#fn:2" rel="footnote">2</a></sup>:</p> <div class="codehilite"><pre><span></span><span class="n">nested_list</span> <span class="o">=</span> <span class="p">[[</span><span class="s1">'cherry'</span><span class="p">,</span> <span class="mi">7</span><span class="p">],</span> <span class="p">[</span><span class="s1">'apple'</span><span class="p">,</span> <span class="mi">100</span><span class="p">],</span> <span class="p">[</span><span class="s1">'anaconda'</span><span class="p">,</span> <span class="mi">1360</span><span class="p">]]</span> </pre></div> <p>We want Python to select the minimum and maximum element based on each item’s <em>weight</em> stored at index 1. We expect <code>min</code> and <code>max</code> to return the following elements:</p> <ul> <li><code>min(nested_list)</code> should be <code>['cherry', 7]</code></li> <li><code>max(nested_list)</code> should be <code>['anaconda', 1360]</code></li> </ul> <p><strong>But if we simply call <code>min</code> and <code>max</code> on that nested list we don’t get the results we expected.</strong></p> <p>The ordering we get seems to be based on the item’s <em>name</em>, stored at index 0:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">min</span><span class="p">(</span><span class="n">nested_list</span><span class="p">)</span> <span class="p">[</span><span class="s1">'anaconda'</span><span class="p">,</span> <span class="mi">1360</span><span class="p">]</span> <span class="c1"># Not what we expected!</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">max</span><span class="p">(</span><span class="n">nested_list</span><span class="p">)</span> <span class="p">[</span><span class="s1">'cherry'</span><span class="p">,</span> <span class="mi">7</span><span class="p">]</span> <span class="c1"># Not what we expected!</span> </pre></div> </section><section><h2>Alright, why does it pick the wrong elements?</h2> <p>Let’s stop for a moment to think about how Python’s <code>max</code> function works internally. The algorithm looks something like this:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">my_max</span><span class="p">(</span><span class="n">sequence</span><span class="p">):</span> <span class="sd">"""Return the maximum element of a sequence"""</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">sequence</span><span class="p">:</span> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'empty sequence'</span><span class="p">)</span> <span class="n">maximum</span> <span class="o">=</span> <span class="n">sequence</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">sequence</span><span class="p">:</span> <span class="k">if</span> <span class="n">item</span> <span class="o">&gt;</span> <span class="n">maximum</span><span class="p">:</span> <span class="n">maximum</span> <span class="o">=</span> <span class="n">item</span> <span class="k">return</span> <span class="n">maximum</span> </pre></div> <p>The interesting bit of behavior here can be found in the condition that selects a new maximum: <code>if item &gt; maximum:</code>.</p> <p>This condition works nicely if <code>sequence</code> only contains primitive types like <code>int</code> or <code>float</code> because comparing those is straightforward (in the sense that it’ll give an answer that we intuitively expect; like <code>3 &gt; 2</code>).</p> <p>However, if <code>sequence</code> contains <em>other sequences</em> then things get a little more complex. Let’s look at the Python docs to learn <a href="https://docs.python.org/3/tutorial/datastructures.html#comparing-sequences-and-other-types" target="_blank">how Python compares sequences</a>:</p> <blockquote> <p>Sequence objects may be compared to other objects with the same sequence type. The comparison uses <a href="https://en.wikipedia.org/wiki/Lexicographical_order" target="_blank">lexicographical ordering</a>: first the first two items are compared, and if they differ this determines the outcome of the comparison; if they are equal, the next two items are compared, and so on, until either sequence is exhausted.</p> </blockquote> <p>When <code>max</code> needs to compare two <em>sequences</em> to find the “larger” element then Python’s default comparison behavior might not be what we want<sup id="fnref:3"><a class="footnote-ref" href="#fn:3" rel="footnote">3</a></sup>.</p> <p>Now that we understand <em>why</em> we get an unexpected result we can think about ways to fix our code.</p> </section><section><h2>How can we change the comparison behavior?</h2> <p>We need to tell <code>max</code> to compare the items differently.</p> <p>In our example, Python’s <code>max</code> looks at the first item in each inner list (the string <code>cherry</code>, <code>apple</code>, or <code>anaconda</code>) and compares it with the current maximum element. That’s why it returns <code>cherry</code> as the maximum element if we just call <code>max(nested_list)</code>.</p> <p>How do we tell <code>max</code> to compare the second item of each inner list?</p> <p>Let’s imagine we had an updated version of <code>my_max</code> called <code>my_max_by_weight</code> that uses the <em>second element</em> of each inner list for comparison:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">my_max_by_weight</span><span class="p">(</span><span class="n">sequence</span><span class="p">):</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">sequence</span><span class="p">:</span> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'empty sequence'</span><span class="p">)</span> <span class="n">maximum</span> <span class="o">=</span> <span class="n">sequence</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">sequence</span><span class="p">:</span> <span class="c1"># Compare elements by their weight stored</span> <span class="c1"># in their second element.</span> <span class="k">if</span> <span class="n">item</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">&gt;</span> <span class="n">maximum</span><span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="n">maximum</span> <span class="o">=</span> <span class="n">item</span> <span class="k">return</span> <span class="n">maximum</span> </pre></div> <p>That would do the trick! We can see that <code>my_max_by_weight</code> selects the maximum element we expected:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">my_max_by_weight</span><span class="p">(</span><span class="n">nested_list</span><span class="p">)</span> <span class="p">[</span><span class="s1">'anaconda'</span><span class="p">,</span> <span class="mi">1360</span><span class="p">]</span> </pre></div> <p>Now imagine we needed to find the maximum of different kinds of lists.</p> <p>Perhaps the index (or <em>key</em>) we’re interested in won’t always be the second item. Maybe sometimes it’ll be the third or fourth item, or a different kind of lookup is necessary all together.</p> <p>Wouldn’t it be great if we could reuse the bulk of the code in our implementation of <code>my_max</code>? Some parts of it will always work the same, for example checking if an empty sequence was passed to the function.</p> </section><section><h2>How can we make max() more flexible?</h2> <p>Because Python allows us to treat functions as data we can extract the code selecting the comparison key into its own function. We’ll call that the <em>key func</em>. We can write different kinds of key funcs and pass them to <code>my_max</code> as necessary.</p> <p>This gives us complete flexibility! Instead of just being able to choose a specific list index for the comparison, like index 1 or 2, we can tell our function to select something else entirely – for example, the length of the item’s name.</p> <p>Let’s have a look at some code that implements this idea:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">identity</span><span class="p">(</span><span class="n">x</span><span class="p">):</span> <span class="k">return</span> <span class="n">x</span> <span class="k">def</span> <span class="nf">my_max</span><span class="p">(</span><span class="n">sequence</span><span class="p">,</span> <span class="n">key_func</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Return the maximum element of a sequence.</span> <span class="sd"> key_func is an optional one-argument ordering function.</span> <span class="sd"> """</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">sequence</span><span class="p">:</span> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'empty sequence'</span><span class="p">)</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">key_func</span><span class="p">:</span> <span class="n">key_func</span> <span class="o">=</span> <span class="n">identity</span> <span class="n">maximum</span> <span class="o">=</span> <span class="n">sequence</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">sequence</span><span class="p">:</span> <span class="c1"># Ask the key func which property to compare</span> <span class="k">if</span> <span class="n">key_func</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> <span class="o">&gt;</span> <span class="n">key_func</span><span class="p">(</span><span class="n">maximum</span><span class="p">):</span> <span class="n">maximum</span> <span class="o">=</span> <span class="n">item</span> <span class="k">return</span> <span class="n">maximum</span> </pre></div> <p>In the code example you can see how by default we let <code>my_max</code> use a key func we called <code>identity</code>, which just uses the whole, unmodified item to do the comparison.</p> <p>With <code>identity</code> as the key func we expect <code>my_max</code> to behave the same way <code>max</code> behaves.</p> <div class="codehilite"><pre><span></span><span class="n">nested_list</span> <span class="o">=</span> <span class="p">[[</span><span class="s1">'cherry'</span><span class="p">,</span> <span class="mi">7</span><span class="p">],</span> <span class="p">[</span><span class="s1">'apple'</span><span class="p">,</span> <span class="mi">100</span><span class="p">],</span> <span class="p">[</span><span class="s1">'anaconda'</span><span class="p">,</span> <span class="mi">1360</span><span class="p">]]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_max</span><span class="p">(</span><span class="n">nested_list</span><span class="p">)</span> <span class="p">[</span><span class="s1">'cherry'</span><span class="p">,</span> <span class="mi">7</span><span class="p">]</span> </pre></div> <p>And we can confirm that we’re still getting the same (incorrect) result as before, which is a pretty good indication that we didn’t screw up the implementation completely 😃.</p> <p>Now comes the cool part – we’re going to override the comparison behavior by writing a <code>key_func</code> that returns the second <em>sub</em>-element instead of the element itself<sup id="fnref:4"><a class="footnote-ref" href="#fn:4" rel="footnote">4</a></sup>:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">weight</span><span class="p">(</span><span class="n">x</span><span class="p">):</span> <span class="k">return</span> <span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_max</span><span class="p">(</span><span class="n">nested_list</span><span class="p">,</span> <span class="n">key_func</span><span class="o">=</span><span class="n">weight</span><span class="p">)</span> <span class="p">[</span><span class="s1">'anaconda'</span><span class="p">,</span> <span class="mi">1360</span><span class="p">]</span> </pre></div> <p>And voilà, this is the maximum element we expected to get!</p> <p>Just to demonstrate the amount of flexibility this refactoring gave us, here’s a <code>key_func</code> that selects the maximum element based on the length of the item’s name:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">name_length</span><span class="p">(</span><span class="n">x</span><span class="p">):</span> <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">my_max</span><span class="p">(</span><span class="n">nested_list</span><span class="p">,</span> <span class="n">key_func</span><span class="o">=</span><span class="n">name_length</span><span class="p">)</span> <span class="p">[</span><span class="s1">'anaconda'</span><span class="p">,</span> <span class="mi">1360</span><span class="p">]</span> </pre></div> </section><section><h2>Is there a shorthand for this stuff?</h2> <p>Instead of defining the key func explicitly with <code>def</code> and giving it a name we can also use Python’s <code>lambda</code> keyword to define a function anonymously. This shortens the code quite a bit (and won’t create a named function):</p> <div class="codehilite"><pre><span></span><span class="n">my_max</span><span class="p">(</span><span class="n">nested_list</span><span class="p">,</span> <span class="n">key_func</span><span class="o">=</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> <span class="o">&gt;&gt;&gt;</span> <span class="p">[</span><span class="s1">'anaconda'</span><span class="p">,</span> <span class="mi">1360</span><span class="p">]</span> </pre></div> <p>To make the naming a little slicker (albeit less expressive) imagine we’ll shorten the <code>key_func</code> arg to <code>key</code> and we’ve arrived at a code snippet that works with the <code>max</code> function in vanilla Python.</p> <p>This means we’ll no longer need our own re-implementation of Python’s <code>max</code> function to find the “correct” maximum element:</p> <div class="codehilite"><pre><span></span><span class="c1"># This is pure, vanilla Python:</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">max</span><span class="p">(</span><span class="n">nested_list</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> <span class="p">[</span><span class="s1">'anaconda'</span><span class="p">,</span> <span class="mi">1360</span><span class="p">]</span> </pre></div> <p>The same also works for Python’s built-in <code>min</code>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">min</span><span class="p">(</span><span class="n">nested_list</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> <span class="p">[</span><span class="s1">'cherry'</span><span class="p">,</span> <span class="mi">7</span><span class="p">]</span> </pre></div> <p>It even works for Python’s <code>sorted</code> function, <strong>making the “key func” concept really valuable in a number of situations you might face as a Python developer</strong>:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">nested_list</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> <span class="p">[[</span><span class="s1">'anaconda'</span><span class="p">,</span> <span class="mi">1360</span><span class="p">],</span> <span class="p">[</span><span class="s1">'apple'</span><span class="p">,</span> <span class="mi">100</span><span class="p">],</span> <span class="p">[</span><span class="s1">'cherry'</span><span class="p">,</span> <span class="mi">7</span><span class="p">]]</span> </pre></div> </section><footer><h3>Try it out yourself</h3> <p>I hope this post helped you out. What started out as a simple question ended up being a little more involved than you may have expected. But it’s often like that when you learn about new programming concepts.</p> <p>Feel free to drop me a line of Twitter or over email if you got stuck anywhere. I’d love to improve this tutorial over time :)</p> <div class="footnote"> <hr> <ol> <li id="fn:1"> <p>Sometimes you’ll see tuples used for the inner lists. Using tuples instead of lists doesn’t really make a difference for how <code>min</code> and <code>max</code> work, but in some cases it can bring a performance benefit. Nothing we’ll have to worry about for now. The code in this tutorial will work fine on a list of tuples. <a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text">↩</a></p> </li> <li id="fn:2"> <p>I actually googled these for you. Apparently the average cherry sold in a super market weighs 7 grams. I’m not 100 per cent sure about anacondas though. <a class="footnote-backref" href="#fnref:2" rev="footnote" title="Jump back to footnote 2 in the text">↩</a></p> </li> <li id="fn:3"> <p>Note that Python strings are also sequences so when you compare two strings they will be compared character by character. <a class="footnote-backref" href="#fnref:3" rev="footnote" title="Jump back to footnote 3 in the text">↩</a></p> </li> <li id="fn:4"> <p>Because this is such a common piece of functionality the Python standard library includes often-used key func implementations in the <a href="https://docs.python.org/3/library/operator.html" target="_blank"><code>operator</code> module</a>. You may want to check out <a href="https://docs.python.org/3/library/operator.html#operator.getitem" target="_blank"><code>operator.getitem</code></a>, <a href="https://docs.python.org/3/library/operator.html#operator.itemgetter" target="_blank"><code>operator.itemgetter</code></a>, and <a href="https://docs.python.org/3/library/operator.html#operator.attrgetter" target="_blank"><code>operator.attrgetter</code></a>. <a class="footnote-backref" href="#fnref:4" rev="footnote" title="Jump back to footnote 4 in the text">↩</a></p> </li> </ol> </div></footer></article>https://dbader.org/blog/python-min-max-and-nested-listsTue, 26 Jul 2016 00:00:00 GMTA better Python REPL: bpython vs pythonhttps://dbader.org/blog/bpython-a-better-python-repl<article><header><h1>A better Python REPL: bpython vs python</h1> <p>A quick video that demonstrates <strong>bpython</strong>, an awesome alternative Python interpreter.</p> </header><section><p>Compared to the vanilla Python interpreter bpython knows a few extra tricks like syntax highlighting, auto indent (yay!), and auto completion.</p> <p>Check it out, it’s a really great tool!</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/QITlSgYf8mc?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> <p>If you’d like to learn more about bpython, the following links should help you out:</p> <ul> <li><a href="http://bpython-interpreter.org/" target="_blank">bpython Homepage</a></li> <li><a href="http://docs.bpython-interpreter.org/" target="_blank">bpython Docs</a></li> <li><a href="https://docs.python.org/2/tutorial/interpreter.html" target="_blank">python REPL Docs</a></li> </ul></section><footer></footer></article>https://dbader.org/blog/bpython-a-better-python-replFri, 22 Jul 2016 00:00:00 GMT6 things you’re missing out on by never using classes in your Python codehttps://dbader.org/blog/6-things-youre-missing-out-on-by-never-using-classes-in-your-python-code<article><header><h1>6 things you’re missing out on by never using classes in your Python code</h1> <p>Maybe you’ve been using Python for a while now and you’re starting to feel like you’re getting the hang of it. But one day you catch yourself thinking: “What about classes?”</p> </header><section><figure><img alt="a motivating picture of a snake" src="/blog/figures/what-about-python-classes.jpg" width="800" height="533"></figure> <p>You see other developers at your company or online discussing the merits of object oriented programming. “<strong>Classes are incredibly useful and everyone should know about them,</strong>” they say.</p> <p>You’re beginning to get worried about writing your code the right way:</p> <blockquote> <p>“I’ve never defined a class… I mean I know how to get stuff done quickly. I use functions and modules for everything, often many small functions whose output I feed into another. I simply don’t use classes because I never felt the need for them.</p> <p>What am I really missing out on?</p> <p><strong>Am I shorting myself on something by never using classes in my Python code?</strong>“</p> </blockquote> <p>I noticed that this is a common question for Python developers with a (data) science or numerical computing background.</p> <p>You may be familiar with languages like R or Matlab and you haven’t really felt the need to use classes in your day to day work writing Python code. Most of your computations are sequential in nature and you found that a structured programming style worked just fine for the category of analytical problems you’re facing.</p> <p>But still … there’s this nagging feeling in the back of your head – <strong>“What am I really missing out on by not writing classes?”</strong></p> <p>Let me try and help you out there. In this post I’m going over 6 things you might be missing out on by never using classes in your Python code. I’m not an OOP zealot by any means but I figured a post like that would be helpful to some folks in the Python community.</p> <p>Alright, let’s dive in!</p> </section><section><h2>1. Easier collections of fields</h2> <p>If you categorically avoid the use of classes it’s easy to find yourself in a situation where you just re-invent your own “ad-hoc classes” using other built-in data structures, like lists or dicts.</p> <p>For example, you might end up with lots of lists or dicts that share the same keys to access different kinds of data associated with a single logical object:</p> <div class="codehilite"><pre><span></span><span class="n">car_colors</span><span class="p">[</span><span class="mi">23</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'yellow'</span> <span class="c1"># Color of Car 23</span> <span class="n">car_mileage</span><span class="p">[</span><span class="mi">23</span><span class="p">]</span> <span class="o">=</span> <span class="mf">38189.4</span> <span class="c1"># Mileage of Car 23</span> </pre></div> <p>By switching to classes you could have a single list of objects, each of which has several named fields on it to address the associated data:</p> <div class="codehilite"><pre><span></span><span class="n">cars</span><span class="p">[</span><span class="mi">23</span><span class="p">]</span><span class="o">.</span><span class="n">color</span> <span class="o">=</span> <span class="s1">'yellow'</span> <span class="n">cars</span><span class="p">[</span><span class="mi">23</span><span class="p">]</span><span class="o">.</span><span class="n">mileage</span> <span class="o">=</span> <span class="mf">38189.4</span> </pre></div> <p>Instead of using lists and dictionaries that happen to contain all your data you can keep all of it under one roof, which makes accessing and passing these objects around much more convenient.</p> <p>You’ll also no longer need to pass around big tuples of stuff from function to function.</p> <p>(<em>Side note</em>: You may want to go and learn more about Python’s <code>namedtuple</code> objects which reduce the amount of boilerplate class-code you need to write if you just want simple C-style records of fields. Notice that <code>namedtuple</code> objects are implemented as Python classes internally.)</p> </section><section><h2>2. Simpler domain models</h2> <p>It’s easier to think about a domain model with a number of classes that have some relation to each other, than it is to think about lists and dictionaries that are linked together by shared keys and indexes.</p> <p>Some domains lend themselves well to being modeled with classes, for example GUI controls. I found that with an object-oriented approach the domain model sometimes becomes easier to discuss and reason about with other developers. And that’s always a good thing.</p> </section><section><h2>3. The ability to chain objects together and let them interact in an expressive way</h2> <p>This is where object oriented programming really shines – when the objects you’re dealing with have behavior on them. For example, a button that can be clicked or a car that can accelerate and brake.</p> <p>In this case it helps if you <em>encapsulate</em> those behaviors by making methods on your Button and Car classes so that other objects can call those methods and change the Button’s or Car’s internal state without knowing how that operation is implemented.</p> <p>Especially when you have a lot of behavior on your objects it’s helpful to have it all in the same place and under one roof that is the object itself. That way you can chain objects together and let them interact in an expressive way that’s hard to emulate in a procedural coding style.</p> <p>For an example of this compare the following two code samples implementing the same sequence of interactions.</p> <p>The OOP version:</p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="ow">not</span> <span class="n">garage</span><span class="o">.</span><span class="n">is_full</span><span class="p">:</span> <span class="n">garage</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">my_car</span><span class="p">)</span> <span class="n">my_car</span><span class="o">.</span><span class="n">turn_off</span><span class="p">()</span> <span class="n">garage</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> </pre></div> <p>vs the non-OOP / procedural version:</p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="ow">not</span> <span class="n">is_garage_full</span><span class="p">(</span><span class="n">garage</span><span class="p">):</span> <span class="n">add_car_to_garage</span><span class="p">(</span><span class="n">my_car</span><span class="p">,</span> <span class="n">garage</span><span class="p">)</span> <span class="n">turn_off_car</span><span class="p">(</span><span class="n">my_car</span><span class="p">)</span> <span class="n">close_garage</span><span class="p">(</span><span class="n">garage</span><span class="p">)</span> </pre></div> </section><section><h2>4. Custom exceptions</h2> <p>If you want to define custom exception types in Python there isn’t really a way around using classes. Custom exception help communicate the programmer’s intent more clearly and they’re a great debugging aid.</p> <p>Here’s an example of a more specific exception type that’s based on one of Python’s built-in exceptions:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">ValueTooSmallError</span><span class="p">(</span><span class="ne">ValueError</span><span class="p">):</span> <span class="k">pass</span> </pre></div> <p>I’ve recorded a quick screencast tutorial that shows you how to use this technique to make your code more readable:</p> <div class="youtube-embed"> <iframe id="ytplayer" type="text/html" src="https://www.youtube.com/embed/hLLaw9BI-EE?autoplay=0&amp;modestbranding=1&amp;showinfo=0&amp;origin=https://dbader.org" frameborder="0" allowfullscreen></iframe> </div> <div class="youtube-subscribe"> <p>» <a href="/youtube/">Subscribe to the dbader.org YouTube Channel</a> for more Python tutorials.</p> </div> </section><section><h2>5. Fitting in with the coding style used by your colleagues</h2> <p>Another good reason for learning about OOP is that it is still a popular paradigm in the programming community. Therefore it is likely that you’ll be working with an object-oriented code base at some point in your career.</p> <p>It can be difficult to integrate and maintain code in a single code base that was written in a different style, let’s say functional vs object-oriented.</p> <p>If you’d rather avoid being the one engineer sticking out with an idiosyncratic coding style then it might be worth familiarizing yourself with OOP and classes.</p> </section><section><h2>6. Useful OOP design patterns</h2> <p>There’s a large body of <a href="https://en.wikipedia.org/wiki/Software_design_pattern" target="_blank">OOP design patterns</a> out there that can speed up development and improve readability. Design patterns are not a panacea and as with all things they can be overused – but in some situations being able to apply common design patterns will be helpful.</p> <p>I’m going to shamelessly plug one of my open-source Python modules, <a href="https://github.com/dbader/schedule" target="_blank">schedule</a>, as an example here. It uses the <a href="https://en.wikipedia.org/wiki/Builder_pattern" target="_blank">Builder pattern</a> to let you schedule periodic jobs with a developer-friendly API:</p> <div class="codehilite"><pre><span></span><span class="kn">import</span> <span class="nn">schedule</span> <span class="k">def</span> <span class="nf">job</span><span class="p">():</span> <span class="k">print</span><span class="p">(</span><span class="s2">"I'm working..."</span><span class="p">)</span> <span class="n">schedule</span><span class="o">.</span><span class="n">every</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span><span class="o">.</span><span class="n">minutes</span><span class="o">.</span><span class="n">do</span><span class="p">(</span><span class="n">job</span><span class="p">)</span> <span class="n">schedule</span><span class="o">.</span><span class="n">every</span><span class="p">()</span><span class="o">.</span><span class="n">hour</span><span class="o">.</span><span class="n">do</span><span class="p">(</span><span class="n">job</span><span class="p">)</span> <span class="n">schedule</span><span class="o">.</span><span class="n">every</span><span class="p">()</span><span class="o">.</span><span class="n">day</span><span class="o">.</span><span class="n">at</span><span class="p">(</span><span class="s2">"10:30"</span><span class="p">)</span><span class="o">.</span><span class="n">do</span><span class="p">(</span><span class="n">job</span><span class="p">)</span> </pre></div> </section><footer><h3>Final thoughts…</h3> <p>With all of this keep in mind that OOP isn’t the only programming paradigm in the world. You might be lucky and not miss out on much extra productivity even if you’re not well-versed in object oriented programming. If you don’t feel the need to use classes then there might not be a real reason to use them in your problem domain!</p> <p>However, to be a well-rounded engineer I believe it’s necessary to know the basics of OOP and object design.</p> <p>I like to think of classes and OOP as yet another tool in the toolbox. <strong>Sometimes it will be exactly what you need to get the job done – and sometimes it won’t.</strong></p> <p>And that’s okay.</p></footer></article>https://dbader.org/blog/6-things-youre-missing-out-on-by-never-using-classes-in-your-python-codeFri, 15 Jul 2016 00:00:00 GMTParsing ISO 8601 timestamps in plain Djangohttps://dbader.org/blog/django-parse-iso-8601-timestamp<article><header><h1>Parsing ISO 8601 timestamps in plain Django</h1> <p>“How do I parse an ISO 8601 formatted date in Django without bringing in extra dependencies?”</p> </header><section><p>If you do any web development with Python and Django then you’ll inevitable find yourself wanting to parse ISO 8601 timestamps into Python’s native <code>datetime.datetime</code> objects at some point. In other words, given a timestamp string like <code>'2016-12-11T09:27:24.895'</code> we want to convert it into a proper Python <code>datetime</code> object for further processing.</p> <p>If you search Google on how to do this you’ll often find people recommend the 3rd party <a href="https://pypi.python.org/pypi/python-dateutil" target="_blank">python-dateutil</a> module. Python-dateutil is a great choice – but in some cases it does more than you really need.</p> <p>If you’re already using Django you can parse ISO 8601 timestamps without bringing in another dependency using <a href="https://docs.djangoproject.com/en/1.9/ref/utils/#django.utils.dateparse.parse_datetime" target="_blank">django.utils.dateparse.parse_datetime</a>.</p> <p>Here’s how:</p> <div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">django.utils.dateparse</span> <span class="kn">import</span> <span class="n">parse_datetime</span> <span class="n">parsed</span> <span class="o">=</span> <span class="n">parse_datetime</span><span class="p">(</span><span class="s1">'2001-12-11T09:27:24.895551'</span><span class="p">)</span> <span class="k">assert</span> <span class="n">parsed</span> <span class="o">==</span> <span class="n">datetime</span><span class="p">(</span><span class="mi">2001</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">11</span><span class="p">,</span> <span class="mi">9</span><span class="p">,</span> <span class="mi">27</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">608645</span><span class="p">)</span> </pre></div> <p>Note that if you pass a malformed value to <code>parse_datetime</code> it can throw a <code>KeyError</code>, <code>ValueError</code>, or <code>TypeError</code> exception so you might want to be ready to handle those.</p> <p>Importantly, <code>parse_datetime</code> also understands timezone-aware timestamps and correctly sets the UTC offset on the resulting <code>datetime</code> object:</p> <div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">django.utils.dateparse</span> <span class="kn">import</span> <span class="n">parse_datetime</span> <span class="kn">from</span> <span class="nn">django.utils.timezone</span> <span class="kn">import</span> <span class="n">is_aware</span><span class="p">,</span> <span class="n">utc</span> <span class="n">expected</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2016</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">27</span><span class="p">,</span> <span class="mi">11</span><span class="p">,</span> <span class="mi">18</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="mi">303886</span><span class="p">,</span> <span class="n">tzinfo</span><span class="o">=</span><span class="n">utc</span><span class="p">)</span> <span class="n">parsed</span> <span class="o">=</span> <span class="n">parse_datetime</span><span class="p">(</span><span class="s1">'2016-04-27T11:18:42.303886+00:00'</span><span class="p">)</span> <span class="k">assert</span> <span class="n">parsed</span> <span class="o">==</span> <span class="n">expected</span> <span class="k">assert</span> <span class="n">is_aware</span><span class="p">(</span><span class="n">parsed</span><span class="p">)</span> </pre></div></section><footer></footer></article>https://dbader.org/blog/django-parse-iso-8601-timestampMon, 11 Jul 2016 00:00:00 GMTCatching bogus Python asserts on CIhttps://dbader.org/blog/catching-bogus-python-asserts<article><header><h1>Catching bogus Python asserts on CI</h1> <p>It’s easy to accidentally write Python assert statements that always evaluate to true. Here’s how to avoid this mistake and catch bad assertions as part of your continuous integration build.</p> </header><section><figure><img alt="Bad Python assert example" src="/blog/figures/bogus-python-asserts-header.png" width="717" height="185"></figure> </section><section><h2>Asserts that are always true</h2> <p>There’s an easy mistake to make with Python’s <code>assert</code>:</p> <p>When you pass it a <strong>tuple</strong> as the first argument, the assertion <strong>always evaluates as true</strong> and therefore <strong>never fails</strong>.</p> <p>To give you a simple example, this assertion will never fail:</p> <div class="codehilite"><pre><span></span><span class="k">assert</span><span class="p">(</span><span class="mi">1</span> <span class="o">==</span> <span class="mi">2</span><span class="p">,</span> <span class="s1">'This should fail'</span><span class="p">)</span> </pre></div> <p>Especially for developers new to Python this can be a surprising result.</p> <p>Let’s take a quick look at the <a href="https://docs.python.org/3/reference/simple_stmts.html#the-assert-statement" target="_blank">syntax for Python’s assert statement</a> to find out why this assertion is bogus and will never fail.</p> <p>Here’s the syntax for <code>assert</code> from the Python docs:</p> <div class="codehilite"><pre><span></span>assert_stmt ::= "assert" expression1 ["," expression2] </pre></div> <p><code>expression1</code> is the condition we test, and the optional <code>expression2</code> is an error message that’s displayed if the assertion fails.</p> <p>At execution time, the Python interpreter transforms each assert statement into the following:</p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="n">__debug__</span><span class="p">:</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">expression1</span><span class="p">:</span> <span class="k">raise</span> <span class="ne">AssertionError</span><span class="p">(</span><span class="n">expression2</span><span class="p">)</span> </pre></div> <p>Let’s take the broken example assertion and apply the transform.</p> <div class="codehilite"><pre><span></span><span class="k">assert</span><span class="p">(</span><span class="mi">1</span> <span class="o">==</span> <span class="mi">2</span><span class="p">,</span> <span class="s1">'This should fail'</span><span class="p">)</span> </pre></div> <p>becomes the following:</p> <div class="codehilite"><pre><span></span><span class="k">if</span> <span class="n">__debug__</span><span class="p">:</span> <span class="k">if</span> <span class="ow">not</span> <span class="p">(</span><span class="mi">1</span> <span class="o">==</span> <span class="mi">2</span><span class="p">,</span> <span class="s1">'This should fail'</span><span class="p">):</span> <span class="k">raise</span> <span class="ne">AssertionError</span><span class="p">()</span> </pre></div> <p>Now we can see where things go wrong.</p> <p>Because <code>assert</code> is a statement and not a function call, the parentheses lead to <code>expression1</code> containing the whole tuple <code>(1 == 2, 'This should fail')</code>.</p> <p><a href="https://docs.python.org/3/library/stdtypes.html#truth-value-testing" target="_blank">Non-empty tuples are always truthy in Python</a> and therefore the assertion will always evaluate to true, which is maybe not what we expected.</p> <p>This behavior can make writing multi-line asserts error-prone. Imagine we have the following assert statement somewhere in our test code:</p> <div class="codehilite"><pre><span></span><span class="k">assert</span> <span class="p">(</span> <span class="n">counter</span> <span class="o">==</span> <span class="mi">10</span><span class="p">,</span> <span class="s1">'It should have counted all the items'</span> <span class="p">)</span> </pre></div> <p>This test case would <strong>never catch an incorrect result</strong>. The assertion always evaluates to <code>True</code> regardless of the state of the <code>counter</code> variable.</p> <p><a href="http://pytest.org/" target="_blank">Pytest</a> encourages you to <a href="http://pytest.org/latest/assert.html#assert-with-the-assert-statement" target="_blank">use plain <code>assert</code> statements in unit tests</a> instead of the <code>assertEquals</code>, <code>assertTrue</code>, …, <code>assertXYZ</code> methods provided by the <code>unittest</code> module in the standard library.</p> <p>It’s relatively easy to accidentally write bad multi-line asserts this way. They can lead to broken test cases that give a falls sense of security in our test code.</p> </section><section><h2>Why isn’t this a warning in Python?</h2> <p>Well, it actually <strong>is a syntax warning in Python 2.6+</strong>:</p> <div class="codehilite"><pre><span></span>&gt;&gt;&gt; assert (1==2, 'This should fail') &lt;input&gt;:2: SyntaxWarning: assertion is always true, perhaps remove parentheses? </pre></div> <p>The trouble is that when you use the <code>py.test</code> test runner, these warnings are hidden:</p> <div class="codehilite"><pre><span></span>$ cat test.py def test_foo(): assert (1==2, 'This should fail') $ py.test -v test.py ======= test session starts ======= platform darwin -- Python 3.5.1, pytest-2.9.0, py-1.4.31, pluggy-0.3.1 rootdir: /Users/daniel/dev/, inifile: pytest.ini collected 1 items test.py::test_foo PASSED ======= 1 passed in 0.03 seconds ======= </pre></div> <p>This makes it easy to write a broken assert in a test case. The assertion will always pass and it’s hard to notice that anything is wrong because <code>py.test</code> hides the syntax warning.</p> </section><section><h2>The solution: Code linting</h2> <p>Luckily, the “bogus assert” issue is the kind of problem that can be <strong>easily caught with a good code linter</strong>.</p> <p><a href="https://pypi.python.org/pypi/pyflakes" target="_blank">pyflakes</a> is an excellent Python linter that strikes a nice balance between helping you catch bugs and avoiding false positives. I highly recommend it.</p> <p>Starting with <a href="https://github.com/pyflakes/pyflakes/releases/tag/1.1.0" target="_blank">pyflakes 1.1.0</a> asserts against a tuple cause a warning, which will help you find bad assert statements in your code.</p> <p>I also recommend to run the linter as part of the continuous integration build. <strong>The build should fail if the program isn’t lint free</strong>. This helps avoid issues when developers forget to run the linter locally before committing their code.</p> <p>Besides going with code linting as a purely technical solution, it’s also a good idea to adopt the following technique when writing tests:</p> <p><strong>When writing a new unit test, always make sure the test actually fails when the code under test is broken or delivers the wrong result.</strong></p> </section><section><h2>The interim solution</h2> <p>If you’re not using pyflakes but still want to be informed of faulty asserts you can use the following <code>grep</code> command as part of your build process:</p> <div class="codehilite"><pre><span></span><span class="o">(</span>egrep <span class="s1">'assert *\('</span> --include <span class="s1">'*.py'</span> --recursive my_app/ <span class="o">||</span> <span class="nb">exit</span> <span class="m">0</span> <span class="o">&amp;&amp;</span> <span class="nb">exit</span> <span class="m">1</span><span class="p">;</span><span class="o">)</span> </pre></div> <p>This will fail the build if there’s an assert statement followed by open parentheses. This is obviously <strong>not perfect and you should use pyflakes</strong>, but in a pinch it’s better than nothing.</p> <p>(Go with pyflakes if you can! 😃)</p></section><footer></footer></article>https://dbader.org/blog/catching-bogus-python-assertsWed, 30 Mar 2016 00:00:00 GMTGenerate and host your API documentation for free with open-source toolshttps://dbader.org/blog/open-source-tools-to-generate-and-host-your-api-documentation-for-free<article><header><h1>Generate and host your API documentation for free with open-source tools</h1> <p>How to generate documentation for a RESTful API as part of your continuous integration build and then automatically deploy it to a website. Includes a full example project on GitHub.</p> </header><section><p>For a recent project I worked on a <strong>distributed system that communicated internally through RESTful APIs</strong>. The implementation work was distributed around different teams working across several timezones.</p> <p>Early on we identified integrating the different systems as one of the likely challenges for the project. To <strong>reduce the communication risk</strong>, we decided to follow a shared set of API design guidelines across all systems.</p> <p>The goal was to have <strong>up-to-date API documentation for each system</strong> available at all times. We used a <strong>combination of tools and automated workflows</strong> that’s been working out well for the project.</p> <p>In this article I’d like to share the approach we used for <strong>auto-generating our API documentation</strong>. We used the apiDoc documentation generator, automated builds for the documentation on CI, and automated deploys of the generated documentation to a website.</p> <hr> <p><strong>Tl;dr:</strong> Here’s what we’re going to build in this tutorial: <a href="http://apidoc-example.surge.sh/" target="_blank">apidoc-example.surge.sh</a></p> <hr> </section><section><h2>Table of Contents</h2> <ol> <li><a href="#apidoc">Intro to apiDoc</a><ul> <li><a href="#apidoc-setup">Setting up apiDoc</a></li> </ul> </li> <li><a href="#apidoc-hosting">Hosting your API docs</a><ul> <li><a href="#apidoc-deploy">Deploying to Surge</a></li> </ul> </li> <li><a href="#auto-deploys">Setting up auto-deploys</a><ul> <li><a href="#npm-scripts">Defining npm scripts for our build</a></li> <li><a href="#travis-ci">Travis CI (build server)</a></li> <li><a href="#surge-credentials">Deploying to Surge from CI</a></li> <li><a href="#first-deploy">The first deploy</a></li> </ul> </li> <li><a href="#github-example">A full, live example on GitHub</a><ul> <li><a href="http://apidoc-example.surge.sh/" target="_blank">apidoc-example.surge.sh</a></li> <li><a href="https://github.com/dbader/apidoc-example" target="_blank">github.com/dbader/apidoc-example</a></li> </ul> </li> </ol> </section><section><h2><a class="anchor" name="apidoc"></a>1. Intro to apiDoc</h2> <p><a href="http://apidocjs.com/" target="_blank">apiDoc</a> is a command-line tool for generating API documentation directly from annotations in the source code of your app. Its syntax is similar to <a href="https://en.wikipedia.org/wiki/Javadoc" target="_blank">JavaDoc</a> and relatively easy to pick up.</p> <p>apiDoc works with most popular programming languages, which means you can use the same annotation syntax across multiple projects in a polyglot environment.</p> <p>To give you a taste of apiDoc’s syntax, here are some example annotations describing a simple endpoint on a RESTful API. I’m using Python as an example here but things would look similar for JavaScript or Ruby:</p> <div class="codehilite"><pre><span></span><span class="nd">@app.route</span><span class="p">(</span><span class="s1">'/api/v1/random/'</span><span class="p">,</span> <span class="n">method</span><span class="o">=</span><span class="p">[</span><span class="s1">'OPTIONS'</span><span class="p">,</span> <span class="s1">'GET'</span><span class="p">])</span> <span class="k">def</span> <span class="nf">get_random_number</span><span class="p">():</span> <span class="sd">"""</span> <span class="sd"> @api {GET} /v1/random/ Generate a random number</span> <span class="sd"> @apiName GetRandomNumber</span> <span class="sd"> @apiGroup Random</span> <span class="sd"> @apiDescription Generates a random number in the range `[0.0, 1.0)`.</span> <span class="sd"> @apiSuccess (Success 200) {UUID} request_id Unique id for the request</span> <span class="sd"> @apiSuccess (Success 200) {Number} results Random number in `[0.0, 1.0)`</span> <span class="sd"> @apiSampleRequest /v1/random/</span> <span class="sd"> @apiExample cURL example</span> <span class="sd"> $ curl https://apidoc-example.herokuapp.com/api/v1/random/</span> <span class="sd"> @apiSuccessExample {js} Success-Response:</span> <span class="sd"> HTTP/1.0 200 OK</span> <span class="sd"> {</span> <span class="sd"> "request_id": "ad506913-a073-4d23-9f95-388d1c1e2c46",</span> <span class="sd"> "result": 0.3606252123151169</span> <span class="sd"> }</span> <span class="sd"> """</span> <span class="c1"># ...</span> </pre></div> <p>apiDoc parses out these annotations and then generates a static website with API documentation meant for people to read. Here’s an example of what that website export looks like:</p> <figure><img alt="" src="/blog/figures/apidoc-example.png" width="832" height="718"></figure> <p>The standard website template that comes with apiDoc looks fairly clean and organized. You can fully customize the look and feel, for example, to use your company branding.</p> <p>The sites generated by apiDoc also include fancy features like the ability to send a sample request to your API directly from the website and inspect the returned result. Here’s what that looks like<sup id="fnref:1"><a class="footnote-ref" href="#fn:1" rel="footnote">1</a></sup>:</p> <figure><img alt="" src="/blog/figures/apidoc-sample-request.png" width="818" height="277"></figure> </section><footer><h3><a class="anchor" name="apidoc-setup"></a>Setting up apiDoc</h3> <p>Now I want to walk you through setting up apiDoc. Let’s install apiDoc and generate our first set of docs. The first thing you’ll need to do is install the <a href="https://www.npmjs.com/package/apidoc" target="_blank">apiDoc command-line client</a> from npm:</p> <div class="codehilite"><pre><span></span>$ npm install apidoc </pre></div> <p>Let’s assume the code for your web app that provides the API lives in a folder called <code>app/</code>. We’ll also asume there’s one source code file in the <code>app/</code> folder that includes the example apiDoc annotations for the <code>GetRandomNumber</code> endpoint we saw earlier.</p> <p>To generate the website-based API documentation and save it in the <code>apidocs/</code> folder you’ll need to run the following command:</p> <div class="codehilite"><pre><span></span>$ apidoc -i ./app/ -o ./apidocs/ </pre></div> <p>Once the apidoc command completes successfully you can open <code>apidocs/index.html</code> in a browser and check out your generated docs.</p> <p>Well, that was easy!</p> <h2><a class="anchor" name="apidoc-hosting"></a>2. Hosting your API docs</h2> <p>Next, we’re going to figure out how to deploy our API docs generated with apiDoc to a public website. My tool of choice for this job is <a href="https://surge.sh/" target="_blank">Surge</a>.</p> <p>Surge’s pitch is “simple, single-command web publishing” and that’s pretty accurate. It’s like a nice command-line interface for Amazon S3 and they offer a free pricing tier that I found perfect for hosting API docs<sup id="fnref:2"><a class="footnote-ref" href="#fn:2" rel="footnote">2</a></sup>.</p> <p>Surge also provides you with a free <code>(your-project).surge.sh</code> subdomain which is great for referencing the latest version of your API documentation from another website or document.</p> <h3><a class="anchor" name="apidoc-deploy"></a>Deploying to Surge</h3> <p>Now that you’re hopefully sold on Surge<sup id="fnref:3"><a class="footnote-ref" href="#fn:3" rel="footnote">3</a></sup>, let’s move right on and set up a deploy command that takes the generated HTML docs and puts them on a website.</p> <p>First, install the <a href="https://www.npmjs.com/package/surge" target="_blank">Surge command-line client</a> from npm:</p> <div class="codehilite"><pre><span></span>$ npm install surge </pre></div> <p>At this point you should already have your apiDoc website export sitting in the <code>apidocs/</code> directory. Now we’ll do the first deploy through Surge. As part of this step the Surge client let’s you create a free Surge account when you run the following command:</p> <div class="codehilite"><pre><span></span>$ surge apidocs/ </pre></div> <p>When the Surge client asks for a domain, feel free to change the randomly generated subdomain to something more suitable, for example <code>myproject-api.surge.sh</code>. If the subdomain is available it will be associated with your Surge account and you can re-use it in subsequent deploys (no one else will be able to steal the domain from you).</p> <p>As the <code>surge apidocs/</code> command executes it informs you of its progress in the terminal:</p> <div class="codehilite"><pre><span></span>Welcome to Surge! (surge.sh) Please login or create an account by entering your email and password: email: youremail@example.com password: ************* project path: apidocs size: 68 files, 598.9 KB domain: myproject-api.surge.sh upload: [====================] 100%, eta: 0.0s propagate on CDN: [====================] 100% plan: Free users: youremail@example.com IP address: 123.124.125.126 Success! Project is published and running at myproject-api.surge.sh </pre></div> <p>Once the Surge deploy is complete you can access the API docs with a web browser at <code>myproject-api.surge.sh</code> (or whatever other subdomain you picked in the previous step).</p> <p>If all we need are manual deploys then we’re done at this point.</p> <p>We can simply refresh the deployed docs by running the apiDoc and Surge commands again. However we might be able to <strong>save a lot of time by automating that process</strong> and putting it on our continuous integration server.</p> <p>Keep reading if you want to learn how that works.</p> <h2><a class="anchor" name="auto-deploys"></a>3. Setting up auto-deploys</h2> <p>I’m a fan of continuous deploy workflows. Regenerating API documentation from source code annotations is something that <strong>can and should be automated</strong>.</p> <p>Let me walk you through the (mostly automated) workflow I use to export API docs with apiDoc to a public website.</p> <p>The general idea is to:</p> <ul> <li><strong>build the docs on a continuous integration server</strong> using the apiDoc command-line tool; and</li> <li>then to <strong>publish them</strong> somewhere, so everyone on your team can access the docs with a web browser.</li> </ul> <p>For the continuous integration service I’m going to go with <a href="https://travis-ci.org/" target="_blank">Travis CI</a> in this example. They offer hosted build servers that are free for use in open-source projects.</p> <h3><a class="anchor" name="npm-scripts"></a>Defining npm scripts for our build</h3> <p>Before we charge ahead and configure Travis CI, let’s take 3 minutes to clean up our build process a little.</p> <p>We’ll define two <a href="https://docs.npmjs.com/misc/scripts" target="_blank">npm scripts</a> in the npm <code>package.json</code> for our project. This is generally a good practice so we don’t have to remember the specifics of how we’re calling the <code>apidoc</code> or <code>surge</code> command-line tools<sup id="fnref:4"><a class="footnote-ref" href="#fn:4" rel="footnote">4</a></sup>.</p> <p>This is what our <code>package.json</code> is going to look like:</p> <div class="codehilite"><pre><span></span><span class="p">{</span> <span class="nt">"devDependencies"</span><span class="p">:</span> <span class="p">{</span> <span class="nt">"apidoc"</span><span class="p">:</span> <span class="s2">"0.15.1"</span><span class="p">,</span> <span class="nt">"surge"</span><span class="p">:</span> <span class="s2">"0.17.7"</span> <span class="p">},</span> <span class="nt">"scripts"</span><span class="p">:</span> <span class="p">{</span> <span class="nt">"apidocs"</span><span class="p">:</span> <span class="s2">"apidoc -i ./app/ -o ./apidocs/"</span><span class="p">,</span> <span class="nt">"apidocs-deploy"</span><span class="p">:</span> <span class="s2">"surge -p ./apidocs/ -d apidoc-example.surge.sh"</span> <span class="p">}</span> <span class="p">}</span> </pre></div> <p>The npm scripts defined here simply contain the commands we went through earlier:</p> <ul> <li><code>npm run apidocs</code> builds the static HTML for the docs; and</li> <li><code>npm run apidocs-deploy</code> deploys the static HTML to Surge with a custom subdomain.</li> </ul> <h3><a class="anchor" name="travis-ci"></a>Travis CI (build server)</h3> <p>Now that we streamlined our build process a little, let’s set up continuous integration to automate these steps.</p> <p><a href="https://travis-ci.org/" target="_blank">Travis CI</a> offers hosted build servers that are free for use in open-source projects. That makes them perfect for use in this example<sup id="fnref:5"><a class="footnote-ref" href="#fn:5" rel="footnote">5</a></sup>.</p> <p>Travis CI is pretty easy to configure for a project hosted on GitHub. The focus of this tutorial is on setting up apiDoc, so for a detailed explanation of how to set up Travis CI you’ll want to check out the <a href="https://docs.travis-ci.com/" target="_blank">Travis CI documentation</a>.</p> <figure><img alt="" src="/blog/figures/apidoc-travis.png" width="879" height="298"></figure> <p>The most important thing to know is that Travis is configured through a config file in the root folder of your project called <code>.travis.yml</code>. This is what a <code>.travis.yml</code> for a simple project might look like:</p> <div class="codehilite"><pre><span></span><span class="l l-Scalar l-Scalar-Plain">language</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">python</span> <span class="l l-Scalar l-Scalar-Plain">python</span><span class="p p-Indicator">:</span> <span class="p p-Indicator">-</span> <span class="s">"2.7"</span> <span class="l l-Scalar l-Scalar-Plain">install</span><span class="p p-Indicator">:</span> <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">pip install -r requirements.txt</span> <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">npm install</span> <span class="l l-Scalar l-Scalar-Plain">script</span><span class="p p-Indicator">:</span> <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">run_tests.sh</span> <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">npm run apidocs</span> <span class="l l-Scalar l-Scalar-Plain">deploy</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">provider</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">script</span> <span class="l l-Scalar l-Scalar-Plain">script</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">npm run apidocs-deploy</span> <span class="l l-Scalar l-Scalar-Plain">skip_cleanup</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">true</span> <span class="l l-Scalar l-Scalar-Plain">on</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">branch</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">master</span> </pre></div> <p>You can see that we’re using the <code>npm run apidocs</code> and <code>npm run apidocs-deploy</code> scripts we set up earlier. We’re also configuring a <a href="https://docs.travis-ci.com/user/deployment/#Conditional-Releases-with-on%3A" target="_blank">Travis CI deploy script</a> that will only execute when Travis builds the <code>master</code> branch.</p> <p>At this point we should be able to build the docs Travis, but to deploy them to Surge you need to provide the <strong>credentials for your Surge account</strong>.</p> <h3><a class="anchor" name="surge-credentials"></a>Deploying to Surge from CI</h3> <p>Because we don’t want to put our Surge username and password into a script inside our repo, we need to configure the Surge client to take the credentials from environment variables.</p> <p>To do that we first need to grab the <em>Surge token</em> for our account. Run the following command to get the token:</p> <div class="codehilite"><pre><span></span>$ surge token Surge - surge.sh email: youremail@example.com token: 13371e4c9b93f3f0682250b6cf8331b </pre></div> <p>Next, we’ll take the email address and token and configure the following two <a href="https://docs.travis-ci.com/user/environment-variables/#Defining-Variables-in-Repository-Settings" target="_blank">environment variables on Travis CI</a>:</p> <div class="codehilite"><pre><span></span>SURGE_LOGIN=youremail@example.com SURGE_TOKEN=13371e4c9b93f3f0682250b6cf8331b </pre></div> <p>If you want to verify that the token works correctly you can run the following command on your machine. It should successfully deploy the API docs like in the previous part of the tutorial:</p> <div class="codehilite"><pre><span></span>$ <span class="nv">SURGE_LOGIN</span><span class="o">=</span>... <span class="nv">SURGE_TOKEN</span><span class="o">=</span>... npm run apidocs-deploy </pre></div> <p>Let’s stop for a second and talk about the <strong>security of those keys</strong>.</p> <p>You might wonder what happens if someone submits a pull-request against that repository. Wouldn’t they be able to just print out the keys and steal them?</p> <p>This is what the Travis CI docs say:</p> <blockquote> <p>”[…] we do not provide these values to untrusted builds, triggered by pull requests from another repository.” – <a href="https://docs.travis-ci.com/user/environment-variables/#Defining-Variables-in-Repository-Settings" target="_blank">Travis CI Docs</a></p> </blockquote> <p>That’s good to know. Unless we accidentally merge a “malicious” pull-request no-one should be able to steal our keys<sup id="fnref:6"><a class="footnote-ref" href="#fn:6" rel="footnote">6</a></sup>.</p> <h3><a class="anchor" name="first-deploy"></a>The first deploy</h3> <p>Once you’ve set up Travis CI and made the necessary configuration changes and pushed them up to GitHub, <strong>you should see the first build complete on CI</strong>.</p> <figure><img alt="" src="/blog/figures/apidoc-deploy.png" width="873" height="139"></figure> <p>It should auto-deploy your docs. This means from now on you should never have to manually deploy the docs again. You just merge to master and CI will refresh the docs and deploy them.</p> <p>Hopefully that’ll save you and your team quite a bit of time in the long run!</p> <h2><a class="anchor" name="github-example"></a>4. A full, live example on GitHub</h2> <p>Here’s a full example of the above setup that you can use as a blueprint for your own API documentation efforts. It generates API docs for a simple Flask app with the process I laid out in this article. You can see the resulting API docs at <a href="http://apidoc-example.surge.sh/" target="_blank">apidoc-example.surge.sh</a>.</p> <p>Go check out the project here: <strong><a href="https://github.com/dbader/apidoc-example" target="_blank">github.com/dbader/apidoc-example</a></strong></p> <div class="footnote"> <hr> <ol> <li id="fn:1"> <p>This is a bit of a gimmick really, but if your API is public and doesn’t require authentication it can be a nice addition and make the docs a little more lively. <a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text">↩</a></p> </li> <li id="fn:2"> <p>If you’re okay with the fact that your docs are publicly available on the internet. Surge provides a paid plan with password protection that’s certainly worth the money if you need that feature. <a class="footnote-backref" href="#fnref:2" rev="footnote" title="Jump back to footnote 2 in the text">↩</a></p> </li> <li id="fn:3"> <p>Did I tell you they’re a local company from Vancouver run by a group of excellent and smart designers and engineers? 😃 <a class="footnote-backref" href="#fnref:3" rev="footnote" title="Jump back to footnote 3 in the text">↩</a></p> </li> <li id="fn:4"> <p>Alternatively you could use a Makefile for this job or just a simple shell script. <a class="footnote-backref" href="#fnref:4" rev="footnote" title="Jump back to footnote 4 in the text">↩</a></p> </li> <li id="fn:5"> <p>Continuous integration for the project I worked on for my client was hosted on <a href="https://circleci.com/" target="_blank">CircleCI</a> which is a similar service that I can also recommend. <a class="footnote-backref" href="#fnref:5" rev="footnote" title="Jump back to footnote 5 in the text">↩</a></p> </li> <li id="fn:6"> <p>It’s probably still a good idea to use a separate Surge account for each project. <a class="footnote-backref" href="#fnref:6" rev="footnote" title="Jump back to footnote 6 in the text">↩</a></p> </li> </ol> </div></footer></article>https://dbader.org/blog/open-source-tools-to-generate-and-host-your-api-documentation-for-freeThu, 17 Mar 2016 00:00:00 GMTHow to stop Django Rest Framework from leaking docstrings into OPTIONS responseshttps://dbader.org/blog/django-rest-framework-options-response<article><header><h1>How to stop Django Rest Framework from leaking docstrings into OPTIONS responses</h1> <p>When you make an HTTP OPTIONS request against an endpoint in a Django Rest Framework app you might be surprised about what you’ll find in the response to that request.</p> </header><section><p>In its default configuration <a href="http://www.django-rest-framework.org/" target="_blank">Rest Framework</a> returns a bunch of metadata that you might not want to return as part of the response. Here’s an example:</p> <div class="codehilite"><pre><span></span>$ http OPTIONS localhost:8000/api/v1/test/ </pre></div> <div class="codehilite"><pre><span></span>HTTP/1.0 200 OK Allow: POST, OPTIONS Content-Type: application/json Date: Tue, 02 Mar 2016 8:23:00 GMT Server: WSGIServer/0.2 CPython/3.5.1 Vary: Cookie </pre></div> <div class="codehilite"><pre><span></span><span class="p">{</span> <span class="s2">"description"</span><span class="o">:</span> <span class="s2">"This is the docstring of the view handling the</span> <span class="s2"> request\nThis might contain information you don't want to leak</span> <span class="s2"> out in an OPTIONS request.\n"</span><span class="p">,</span> <span class="s2">"name"</span><span class="o">:</span> <span class="s2">"Test Endpoint"</span><span class="p">,</span> <span class="s2">"parses"</span><span class="o">:</span> <span class="p">[</span> <span class="s2">"application/x-www-form-urlencoded"</span><span class="p">,</span> <span class="s2">"multipart/form-data"</span><span class="p">,</span> <span class="s2">"application/json"</span> <span class="p">],</span> <span class="s2">"renders"</span><span class="o">:</span> <span class="p">[</span> <span class="s2">"application/json"</span> <span class="p">]</span> <span class="p">}</span> </pre></div> <p>As you can see, by default the response includes the full docstring for the view as part of the <code>description</code> field. If that’s not what you want you can configure the metadata returned by Django Rest Framework through the <a href="http://www.django-rest-framework.org/api-guide/metadata/" target="_blank">metadata scheme</a> mechanism.</p> <p>Here’s a null metadata scheme that configures <code>OPTIONS</code> responses to be empty:</p> <div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">rest_framework.metadata</span> <span class="kn">import</span> <span class="n">BaseMetadata</span> <span class="k">class</span> <span class="nc">NoMetaData</span><span class="p">(</span><span class="n">BaseMetadata</span><span class="p">):</span> <span class="k">def</span> <span class="nf">determine_metadata</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">view</span><span class="p">):</span> <span class="k">return</span> <span class="bp">None</span> </pre></div> <p>To set that metadata class globally we can use the <code>DEFAULT_METADATA_CLASS</code> setting in Rest Framework:</p> <div class="codehilite"><pre><span></span><span class="n">REST_FRAMEWORK</span> <span class="o">=</span> <span class="p">{</span> <span class="s1">'DEFAULT_METADATA_CLASS'</span><span class="p">:</span> <span class="s1">'yourapp.metadata.NoMetaData'</span> <span class="p">}</span> </pre></div> <p>When we make the same <code>OPTIONS</code> request now we get the empty response we wanted:</p> <div class="codehilite"><pre><span></span>$ http OPTIONS localhost:8000/api/v1/test/ </pre></div> <div class="codehilite"><pre><span></span>HTTP/1.0 200 OK Allow: POST, OPTIONS Content-Type: application/json Date: Tue, 02 Mar 2016 8:42:00 GMT Server: WSGIServer/0.2 CPython/3.5.1 Vary: Cookie </pre></div></section><footer></footer></article>https://dbader.org/blog/django-rest-framework-options-responseThu, 03 Mar 2016 00:00:00 GMTDebugging memory usage in a live Python web apphttps://dbader.org/blog/debugging-python-memory-usage<article><header><h1>Debugging memory usage in a live Python web app</h1> <p>I worked on a Python web app a while ago that was struggling with using too much memory in production. A helpful technique for debugging this issue was adding a simple API endpoint that exposed memory stats while the app was running.</p> </header><section><figure><img alt="" src="/blog/figures/python-memory-header.png" width="796" height="537"></figure> </section><section><h2>Enter Pympler</h2> <p>There’s a great module called <a href="https://pythonhosted.org/Pympler/" target="_blank">Pympler</a> for debugging memory stats in CPython. It walks your process heap and reports the object types, number of objects, and their size in bytes for all allocated Python objects.</p> <p>The following function generates a memory summary using Pympler and returns it as a string:</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">memory_summary</span><span class="p">():</span> <span class="c1"># Only import Pympler when we need it. We don't want it to</span> <span class="c1"># affect our process if we never call memory_summary.</span> <span class="kn">from</span> <span class="nn">pympler</span> <span class="kn">import</span> <span class="n">summary</span><span class="p">,</span> <span class="n">muppy</span> <span class="n">mem_summary</span> <span class="o">=</span> <span class="n">summary</span><span class="o">.</span><span class="n">summarize</span><span class="p">(</span><span class="n">muppy</span><span class="o">.</span><span class="n">get_objects</span><span class="p">())</span> <span class="n">rows</span> <span class="o">=</span> <span class="n">summary</span><span class="o">.</span><span class="n">format_</span><span class="p">(</span><span class="n">mem_summary</span><span class="p">)</span> <span class="k">return</span> <span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">rows</span><span class="p">)</span> </pre></div> <p>Let’s plug this into an example app that allocates some memory and then calls <code>memory_summary</code>:</p> <div class="codehilite"><pre><span></span><span class="sd">"""</span> <span class="sd">Don't forget to $ pip install pympler.</span> <span class="sd">"""</span> <span class="kn">import</span> <span class="nn">sys</span> <span class="kn">from</span> <span class="nn">StringIO</span> <span class="kn">import</span> <span class="n">StringIO</span> <span class="k">def</span> <span class="nf">memory_summary</span><span class="p">():</span> <span class="c1"># ... (see above)</span> <span class="c1"># Allocate some memory</span> <span class="n">my_str</span> <span class="o">=</span> <span class="s1">'a'</span> <span class="o">*</span> <span class="mi">2</span><span class="o">**</span><span class="mi">26</span> <span class="k">class</span> <span class="nc">MyObject</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">memory</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="nb">id</span><span class="p">(</span><span class="bp">self</span><span class="p">))</span> <span class="o">*</span> <span class="mi">2</span><span class="o">**</span><span class="mi">10</span> <span class="n">my_objs</span> <span class="o">=</span> <span class="p">[</span><span class="n">MyObject</span><span class="p">()</span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="mi">2</span><span class="o">**</span><span class="mi">16</span><span class="p">)]</span> <span class="k">print</span><span class="p">(</span><span class="n">memory_summary</span><span class="p">())</span> </pre></div> <p>Running this example will result in a printout like the one below, which should give you a rough idea which objects are taking up the most space in your app:</p> <div class="codehilite"><pre><span></span> types | # objects | total size ============================ | =========== | ============ str | 6727 | 64.61 MB &lt;class '__main__.MyObject | 65536 | 4.00 MB dict | 596 | 950.84 KB list | 251 | 601.54 KB code | 1872 | 234.00 KB wrapper_descriptor | 1094 | 85.47 KB type | 96 | 85.45 KB builtin_function_or_method | 726 | 51.05 KB method_descriptor | 586 | 41.20 KB set | 135 | 36.59 KB weakref | 386 | 33.17 KB tuple | 384 | 28.27 KB _sre.SRE_Pattern | 42 | 19.31 KB &lt;class 'abc.ABCMeta | 20 | 17.66 KB member_descriptor | 231 | 16.24 KB </pre></div> <p>For example, we see that the <code>str</code> objects we allocated take up the biggest chunk of memory at around 65 MB. And as expected, there are also 2^16 = 65536 <code>MyObject</code> instances, taking up 4 MB of space in total.</p> </section><section><h2>But how can we access this information in a production web app?</h2> <p>I ended up just exposing the output of <code>memory_summary()</code> as a <code>/debug/memory</code> plaintext endpoint secured with HTTP basic auth. This allowed us to access the allocation stats for the app while it was running in production.</p> <p>A more advanced way to track these stats in a production web app would be to feed them into a <a href="/blog/monitoring-your-nodejs-app-with-datadog">service like DataDog</a> to plot and track them over time. However, in many cases a simple solution like printing the stats to the application log might suffice.</p> <p>Please also note that these stats are <em>per interpreter process</em>. If you’re running your web app as multiple CPython processes behind a load balancer (like you should) then you’ve got to be sure to take that into account when making sense of these memory stats.</p> <p>Still, I found that just getting a rough sample of which objects are taking up the most space gave me a better idea of the memory usage pattern of the app and helped reduce memory consumption with some follow up work.</p></section><footer></footer></article>https://dbader.org/blog/debugging-python-memory-usageTue, 09 Feb 2016 00:00:00 GMTHow to store photos in the cloud and avoid vendor lock-inhttps://dbader.org/blog/how-to-store-photos-in-the-cloud-and-avoid-vendor-lock-in<article><header><h1>How to store photos in the cloud and avoid vendor lock-in</h1> <p>I’ve been burned by relying 100% on a cloud service before. Some time ago a photographer friend convinced me to sign up for an awesome photo storage service called <a href="http://www.theverge.com/2013/11/5/5039216/everpix-life-and-death-inside-the-worlds-best-photo-startup" target="_blank">Everpix</a>.</p> </header><section><figure><img alt="Header Image" src="/blog/figures/cloud-photo-storage.jpg" width="1200" height="800"></figure> <p>My internet connection ran red hot for a couple of days and nights until my whole photo library was finally transferred to Everpix’s cloud. And I loved the service. It was fast and had great UX. Finally, a cloud photo storage solution that worked well for me.</p> <p>It was simply a joy to use.</p> <p>At least for a few days—then they sent me an email telling me they <strong>ran out of money and had to shut down the company</strong>.</p> <p>There was a grace period where Everpix let you download your photos in their original quality as a giant zip archive. So at least people didn’t lose any data if they acted quickly enough.</p> <p>A few weeks later Everpix was finally gone and I felt frustrated<sup id="fnref:1"><a class="footnote-ref" href="#fn:1" rel="footnote">1</a></sup>.</p> <p>I had really enjoyed being able to access all of my photos from any device I owned. I had liked the fact that I didn’t have to worry about manual backups as much.</p> <p>I decided that I wasn’t going to be tied to a single cloud service ever again and set out to build my own photo storage solution. It’s not as fancy as Everpix was but it get’s the job done and feels much more future proof.</p> <p>Let me give you a quick overview of how it works.</p> </section><section><h2>One folder structure to rule them all</h2> <p>Instead of using a proprietary storage format like Apple’s <a href="https://www.apple.com/osx/photos/" target="_blank">Photos.app</a> or Everpix, all of my photos simply go into a nested folder structure based on their timestamp.</p> <p>I give each photo a path and filename based on the time it was taken and then I sort it into the following folder structure:</p> <div class="codehilite"><pre><span></span>├── 2014 │   ├── 2014-01 │ │   ├── 2014-01-05 13.24.45.jpg │ │   ├── 2014-01-05 21.28.48.jpg │ │   ├── 2014-01-05 21.28.48-1.jpg │ │   ├── 2014-01-06 21.14.38.jpg │   ├── 2014-02 │ | ├─ ... │   ├── ... │   └── 2014-12 ├── 2015 │   ├── 2015-01 │   ├── 2015-02 │   ├── ... │   └── 2015-12 ├── ... </pre></div> <p>This is a dead simple scheme that I’ll be able to keep using as long as there are hierarchical file systems. And the good news is that all of this sorting and structuring can happen automatically based on <a href="https://en.wikipedia.org/wiki/Exchangeable_image_file_format" target="_blank">EXIF</a> timestamps or file creation dates.</p> <p>I found that a simple folder structure is a perfect fit for my photo storage needs. I sometimes create “albums” by moving some photos into a separate folder, for example:</p> <div class="codehilite"><pre><span></span>├── 2015 │   ├── 2015-01 │   ├── 2015-02 │   ├── 2015-02 My Album │   ├── 2015-03 ├── ... </pre></div> <p>This let’s me keep the year-month sort order in the yearly folders and provides enough structure to find important events quickly. Occasionally I also create “virtual” albums in Carousel to share with friends and family, but more on that in a minute.</p> </section><section><h2>Dropbox &amp; Photosorter</h2> <p>The setup I use now is built around <a href="https://www.dropbox.com/" target="_blank">Dropbox</a> for cloud storage and my open-source <a href="https://github.com/dbader/photosorter" target="_blank">photosorter</a> tool. The complete workflow is fully automated and looks like this:</p> <ol> <li><strong>New photos go into the <em>Camera Uploads</em> folder on my Dropbox.</strong> This either happens by me manually copying them off an SD card into the Camera Uploads folder or the Dropbox iOS app automatically uploads new photos when my phone has a Wi-Fi connection.</li> <li><strong>Photosorter runs on my home server and watches <em>Camera Uploads</em> for new photos.</strong> It then takes them and moves them into the appropriate place in my <em>Photos</em> folder which also lives in my Dropbox. Photosorter detects and ignores duplicates through their SHA1 hash. Photos taken in the same instant are deduplicated by adding a suffix (<code>-1</code>, <code>-2</code>, etc) to the filename.</li> <li><strong>Dropbox picks up the new files in my <em>Photos</em> folder and distributes them to all my devices.</strong> Once the photos are in Dropbox I can also access them from anywhere using the Dropbox website.</li> </ol> <p>This setup has the nice side-effect that I have a physical backup of my photos in several places, like my home server and my Mac. This works because my photo library is only about 100 GB in size. For a larger library I’m either going to just buy more storage or keep a full backup on my home server and disable syncing on my Mac.</p> <p>If want to give photosorter a try there’s documentation and a deployment example on its <a href="https://github.com/dbader/photosorter" target="_blank">GitHub page</a>.</p> </section><section><h2>Carousel</h2> <div class="update-box"> <h2>Update: Dropbox is going to kill Carousel, meh.</h2> <p>Since I wrote this article Dropbox announced that they will <a href="https://blogs.dropbox.com/dropbox/2015/12/saying-goodbye-to-carousel-and-mailbox/" target="_blank">shut down Carousel on March 31st, 2016</a>. This is a bit of a bummer because parts of the workflow I’m describing here worked really well with Carousel.</p> <p><strong>However</strong>, they said that they’ll port most of Carousel’s functionality back into the Dropbox app and website. That’s fine by me and would work well with my photos workflow. I don’t really care <em>which</em> app I need to launch to look at my photos (that’s also kind of the whole point of this article). Once Carousel’s gone I’ll update the article with new recommendations for tools.</p> </div> <p><a href="https://carousel.dropbox.com" target="_blank">Carousel</a> is Dropbox’s new product for managing photos in your Dropbox account. I really like the Carousel app and website. It’s a super convenient way to browse through my photos from anywhere I want. I also frequently use it to share photos with friends and family by creating ad-hoc albums on Carousel.</p> <p>Their iOS app let’s me access all of my photos while not taking up much space on my phone. This is thanks to Carousel’s smart caching system that only keeps high-quality version of photos you viewed recently on your phone. It’s similar to iCloud photos on iOS 9, works well and usually requires zero babysitting.</p> <p>Carousel also has a cool <em>flashbacks</em> feature that shows you photos that you took in the same week one or more years ago. Everpix had that too and it’s a neat way to enjoy older photos from my library.</p> <p>Like I said before I also use the iOS app to automatically upload new photos from my iPhone when I’m on Wi-Fi. This pretty much guarantees that I won’t lose photos while I’m traveling. It also helps keep enough free space on my phone so I can continue taking photos.</p> </section><section><h2>The future</h2> <p>I’m currently running <a href="https://github.com/dbader/photosorter" target="_blank">photosorter</a> on my home server. At some point I might replace it with a virtual machine on S3 or Digital Ocean which will provide cheaper storage and better fault tolerance. I don’t really trust that little Toshiba notebook drive spinning 24/7.</p> <p>This setup has served me well over the past two years. Obviously setting this up is more involved than just using a turnkey solution. But I also feel like it’s more future proof than using an off-the-shelf service like Apple’s <a href="https://www.apple.com/icloud/photos/" target="_blank">iCloud Photo Library</a> or <a href="https://photos.google.com/" target="_blank">Google Photos</a>.</p> <p>I’ve been burned by Everpix’s sudden disappearance and if Dropbox goes away I’ll just use a different filesystem-based sync service like <a href="https://www.getsync.com/" target="_blank">BitTorrent Sync</a>. If you’re worried about <a href="http://www.androidauthority.com/google-photos-worried-privacy-616339/" target="_blank">privacy</a> then running your own photo storage solution might be appealing, too.</p> <div class="footnote"> <hr> <ol> <li id="fn:1"> <p>I think I’m going to feel frustrated pretty soon again when <a href="https://www.rdio.com/" target="_blank">Rdio</a> shuts down… <a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text">↩</a></p> </li> </ol> </div></section><footer></footer></article>https://dbader.org/blog/how-to-store-photos-in-the-cloud-and-avoid-vendor-lock-inSat, 28 Nov 2015 00:00:00 GMTOS X notifications for your pytest runshttps://dbader.org/blog/osx-notifications-for-your-pytest-runs<article><header><h1>OS X notifications for your pytest runs</h1> <p>This article shows you how to use the pytest-osxnotify, a plugin for pytest that adds native Mac OS X notifications to the pytest terminal runner.</p> </header><section><figure><img alt="pytest-osxnotify demo gif" src="/blog/figures/pytest-osxnotify-demo.gif" width="418" height="162"></figure> </section><section><h2>pytest + OS X notifications = happy developers</h2> <p><a href="https://github.com/dbader/pytest-osxnotify" target="_blank">pytest-osxnotify</a> is a plugin for the <a href="http://pytest.org/" target="_blank">pytest</a> testing tool. It adds OS X notifications to your test runs so you know when a test run completes and whether it failed or succeeded without looking at your terminal window.</p> <p>This is especially useful when you re-run your tests automatically every time a source file was modified.</p> </section><section><h2>A quick example</h2> <p>Installing pytest-osxnotify is easy. Let’s set up a simple example that shows you how to use pytest so that it watches your source files for modifications and re-runs the tests as necessary.</p> <p>We start by installing pytest, pytest-xdist and pytest-osxnotify<sup id="fnref:1"><a class="footnote-ref" href="#fn:1" rel="footnote">1</a></sup>.</p> <div class="codehilite"><pre><span></span>$ pip install pytest pytest-xdist pytest-osxnotify </pre></div> <p>Let’s also create a simple test file for us to run. Save the following as <code>example_test.py</code> in the current folder.</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">test_example1</span><span class="p">():</span> <span class="k">assert</span> <span class="bp">True</span> <span class="k">def</span> <span class="nf">test_example2</span><span class="p">():</span> <span class="k">assert</span> <span class="bp">True</span> <span class="k">def</span> <span class="nf">test_example3</span><span class="p">():</span> <span class="k">assert</span> <span class="bp">True</span> </pre></div> <p>Now we start the pytest <em>watcher</em> that monitors our source file for modifications and re-runs the tests when necessary.</p> <div class="codehilite"><pre><span></span>$ py.test -f example_test.py </pre></div> <p>That’s it. We can now move our terminal to the background and hack away in our <a href="setting-up-sublime-text-for-python-development">favourite editor</a> knowing that we’ll stay informed about the results of our test runs.</p> <figure><img alt="pytest-osxnotify example" src="/blog/figures/pytest-osxnotify-example.jpg" width="728" height="504"></figure> <div class="footnote"> <hr> <ol> <li id="fn:1"> <p>You’ll typically want to install your dependencies into a Python <a href="https://virtualenv.pypa.io/en/latest/" target="_blank">virtualenv</a> so that they don’t pollute your system install. Look <a href="http://docs.python-guide.org/en/latest/dev/virtualenvs/" target="_blank">here</a> for a good tutorial on using virtualenv. <a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text">↩</a></p> </li> </ol> </div></section><footer></footer></article>https://dbader.org/blog/osx-notifications-for-your-pytest-runsMon, 18 May 2015 00:00:00 GMTSoftware engineer reading list: My favourite books about programminghttps://dbader.org/blog/my-favourite-books-about-programming<article><header><h1>Software engineer reading list: My favourite books about programming</h1> <p>Reading books is one of the best ways to improve your craftsmanship and to become a better software developer. This is a continuously updated list with my favourite programming books, sorted by topic. I link to the ebook version where possible but most books should be available made from dead trees as well.</p> </header><section><figure><img alt="" src="/blog/figures/favourite-books.jpg" width="1400" height="788"></figure> </section><section><h2>Architecture &amp; System Design</h2> <p>How to build reliable software that works well.</p> <ul> <li><a href="http://amzn.to/1oeMB5a" target="_blank">Release It!</a> by Michael T. Nygard</li> <li><a href="http://amzn.to/1nlVqbb" target="_blank">The Architecture of Open Source Applications</a> by Amy Brown</li> <li><a href="http://amzn.to/1nJmT54" target="_blank">The Architecture of Open Source Applications, Volume II</a> by Amy Brown</li> <li><a href="http://amzn.to/1nJmXlg" target="_blank">The Performance of Open Source Applications</a> by Tavish Armstrong</li> </ul> </section><section><h2>Craftsmanship</h2> <p>Books about best practices, code quality and professionalism. Every single one of these books is fantastic and I got so much out of them. If you don’t know which area to focus on first then start here.</p> <ul> <li><a href="http://amzn.to/1lCUuLv" target="_blank">Clean Code</a> by Robert C. Martin</li> <li><a href="http://amzn.to/1oeMw1f" target="_blank">Team Geek</a> by Brian W. Fitzpatrick</li> <li><a href="http://amzn.to/1lCUyL8" target="_blank">The Clean Coder</a> by Robert C. Martin</li> <li><a href="http://amzn.to/1giLvm4" target="_blank">The Passionate Programmer</a> by Chad Fowler</li> <li><a href="http://amzn.to/1lJIms1" target="_blank">The Zen Programmer</a> by Christian Grobmeier</li> <li><a href="http://amzn.to/1zFUGmH" target="_blank">HBR’s 10 Must Reads on Managing Yourself</a></li> <li><a href="http://amzn.to/13JS6Bu" target="_blank">Better: A surgeon’s notes on performance</a> by Atul Gawande</li> <li><a href="http://amzn.to/2kP64wQ" target="_blank">Code Complete</a> by Steve McConnell</li> </ul> </section><section><h2>Programming Languages</h2> <p>Books about specific programming languages that I enjoyed. There’s often free resources available online but sometimes it’s nice to just buy a book that takes you through many aspects of a language. Some of these books are great reads even if you’re not interested in the language specifically, as they teach you important universal concepts.</p> <p><strong>Haskell</strong></p> <ul> <li><a href="http://learnyouahaskell.com/" target="_blank">Learn You a Haskell for Great Good!</a> by Miran Lipovača</li> <li><a href="http://amzn.to/1nm5n8C" target="_blank">Parallel and Concurrent Programming in Haskell</a> by Simon Marlow</li> </ul> <p><strong>JavaScript</strong></p> <ul> <li><a href="http://amzn.to/1jfd4rB" target="_blank">Effective JavaScript</a> by David Herman</li> <li><a href="http://amzn.to/1oeMRRJ" target="_blank">JavaScript: The Good Parts</a> by Douglas Crockford</li> </ul> <p><strong>Python</strong></p> <ul> <li><a href="http://amzn.to/1vzKjPK" target="_blank">Writing Idiomatic Python</a> by Jeff Knupp</li> <li><a href="http://amzn.to/1zFdnuK" target="_blank">Effective Python</a> by Bret Slatkin</li> <li><a href="http://amzn.to/2cfgOB2" target="_blank">Python Cookbook, 3rd Ed.</a> by David Beazley and Brian Jones</li> <li><a href="http://amzn.to/2cfgphX" target="_blank">Two Scoops of Django</a> by Daniel and Audrey Roy Greenfeld</li> <li><a href="http://amzn.to/2bVLSXx" target="_blank">Fluent Python</a> by Luciano Ramalho</li> <li><a href="http://amzn.to/2c08vsg" target="_blank">Automate the Boring Stuff with Python</a> by Al Sweigart</li> </ul> <p><strong>Scala</strong></p> <ul> <li><a href="http://amzn.to/Rox4kI" target="_blank">Programming in Scala</a> by Martin Odersky</li> </ul> </section><section><h2>Interviews &amp; Hiring</h2> <p>These books work both ways. If you’re trying to be hired as an engineer or hiring others then you can learn a lot from them.</p> <ul> <li><a href="http://amzn.to/1dEbYKS" target="_blank">Elements of Programming Interviews</a> by Aziz, Lee and Prakash</li> <li><a href="http://amzn.to/1oeOjn9" target="_blank">Cracking the Coding Interview</a> by Gayle Laakmann McDowell</li> <li><a href="http://amzn.to/1mh9527" target="_blank">Programming Interviews Exposed</a> by John Morgan</li> </ul> </section><section><h2>Leadership &amp; Managing developers</h2> <p>These are useful even if you’re not in a leadership position. They’ll help you understand your manager better and will make you a more effective communicator.</p> <ul> <li><a href="http://amzn.to/1x5pOxW" target="_blank">Managing Humans</a> by Michael Lopp</li> <li><a href="http://leadingsnowflakes.com/" target="_blank">Leading Snowflakes</a> by Oren Ellenbogen</li> <li><a href="http://amzn.to/1x5rjML" target="_blank">How to Win Friends &amp; Influence People</a> by Dale Carnegie</li> <li><a href="http://amzn.to/1v8XpRp" target="_blank">It’s Not All About Me</a> by Robin Dreeke</li> </ul> </section><section><h2>CompSci fundamentals, algorithms, and math</h2> <p>This stuff is important. Languages and frameworks come and go but the foundations remain largely static. Re-visit these every once in a while.</p> <ul> <li><a href="http://amzn.to/1oeOte3" target="_blank">The Algorithm Design Manual</a> by Steven S. Skiena</li> <li><a href="http://amzn.to/1dEc45e" target="_blank">Algorithms</a> by Dasgupta, Papadimitriou, and Vazirani</li> <li><a href="http://amzn.to/1mh7ERq" target="_blank">Introduction to Algorithms</a> by Thomas H. Cormen</li> <li><a href="http://amzn.to/1oeOAqe" target="_blank">Concrete Mathematics</a> by Ronald L. Graham</li> </ul> </section><section><h2>Postmortems</h2> <p>The best software engineering war stories around. I get inspired by reading about successful or failed software projects that others have worked on. These books let you learn from the experiences and careers of some of the best people in the field.</p> <ul> <li><a href="http://amzn.to/1vzJpCU" target="_blank">Coders at Work</a> by Peter Seibel</li> <li><a href="http://amzn.to/1tfp1VH" target="_blank">FoxTales</a> by Kerry Nietz</li> <li><a href="http://amzn.to/1idgTxK" target="_blank">Masters of Doom</a> by David Kushner</li> <li><a href="http://amzn.to/1oeNwTc" target="_blank">Postmortems from Game Developer</a> by Austin Grossman</li> <li><a href="http://amzn.to/1lCUs6a" target="_blank">Showstopper</a> by G. Pascal Zachary</li> <li><a href="http://amzn.to/1oeNjzk" target="_blank">The Future Was Here: The Commodore Amiga</a> by Jimmy Maher</li> <li><a href="http://amzn.to/1oeNkTW" target="_blank">The Making of Karateka</a> by Jordan Mechner</li> <li><a href="http://amzn.to/1oeNpqP" target="_blank">The Making of Prince of Persia</a> by Jordan Mechner</li> </ul> </section><section><h2>Writing</h2> <p>Being able to communicate succinctly in writing is often more important than raw technical ability. Especially if you want to convince others. These books have helped me to structure my thinking and improved my English. Especially if English is your second language like it is for me this is an area you should focus on.</p> <ul> <li><a href="http://amzn.to/1k5HIUp" target="_blank">On Writing Well</a> by William Zinsser</li> <li><a href="http://amzn.to/1oh8HnB" target="_blank">Oxford Guide to Plain English</a> by Martin Cutts</li> <li><a href="http://amzn.to/1oeNI4J" target="_blank">Writing for Computer Science </a> by Justin Zobel</li> </ul></section><footer></footer></article>https://dbader.org/blog/my-favourite-books-about-programmingSat, 24 May 2014 00:00:00 GMTAbstract Base Classes in Pythonhttps://dbader.org/blog/abstract-base-classes-in-python<article><header><h1>Abstract Base Classes in Python</h1> <p>Abstract Base Classes (ABCs) enforce that derived classes implement particular methods from the base class. In this chapter you’ll learn about the benefits of abstract base classes and how to define them with Python’s built-in “abc” module.</p> </header><section><figure><img alt="" src="/blog/figures/python-abcs-header.png" width="1280" height="719"></figure> </section><section><h2>What the ABC?</h2> <p>What are Abstract Base Classes good for? A while ago I had a discussion about which pattern to use for implementing a maintainable class hierarchy in Python. More specifically, the goal was to define a simple class hierarchy for a service backend in the most programmer-friendly and maintainable way.</p> <p>There was a <code>BaseService</code> that defines a common interface and several concrete implementations that do different things but all provide the same interface (<code>MockService</code>, <code>RealService</code>, and so on). To make this relationship explicit the concrete implementations all subclass <code>BaseService</code>.</p> <p>To be as maintainable and programmer-friendly as possible the idea was to make sure that:</p> <ul> <li>instantiating the base class is impossible; and</li> <li>forgetting to implement interface methods in one of the subclasses raises an error as early as possible.</li> </ul> </section><section><h2>When to Use Python’s <code>abc</code> Module</h2> <p>This raises the question why you’d want to use Python’s <code>abc</code> module. The above design problem is pretty common in more complex systems. To enforce that a derived class implements a number of methods from the base class we can use something like this Python idiom:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Base</span><span class="p">:</span> <span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">()</span> <span class="k">def</span> <span class="nf">bar</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">()</span> <span class="k">class</span> <span class="nc">Concrete</span><span class="p">(</span><span class="n">Base</span><span class="p">):</span> <span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="s1">'foo() called'</span> <span class="c1"># Oh no, we forgot to override bar()...</span> <span class="c1"># def bar(self):</span> <span class="c1"># return "bar() called"</span> </pre></div> <p>So, what do we get from this first implementation? Calling methods on an instance of <code>Base</code> correctly raises <code>NotImplementedError</code> exceptions:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">b</span> <span class="o">=</span> <span class="n">Base</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">b</span><span class="o">.</span><span class="n">foo</span><span class="p">()</span> <span class="ne">NotImplementedError</span> </pre></div> <p>Furthermore, instantiating and using <code>Concrete</code> works as expected—and, if we call an unimplemented method on it like <code>bar()</code> this also raises an exception:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">c</span> <span class="o">=</span> <span class="n">Concrete</span><span class="p">()</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">c</span><span class="o">.</span><span class="n">foo</span><span class="p">()</span> <span class="s1">'foo() called'</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">c</span><span class="o">.</span><span class="n">bar</span><span class="p">()</span> <span class="ne">NotImplementedError</span> </pre></div> <p>This first implementation is decent but it isn’t perfect yet. The downsides here are that we can still:</p> <ul> <li>instantiate <code>Base</code> just fine without getting an error; and</li> <li>provide incomplete subclasses—instantiating <code>Concrete</code> will not raise an error until we call the missing method <code>bar()</code>.</li> </ul> <p>With Python’s <code>abc</code> module was <a href="http://docs.python.org/2/library/abc.html" target="_blank">added in Python 2.6</a>, we can do quite a bit better and solve these outstanding issues. Here’s an updated implementation using an Abstract Base Class defined with the <code>abc</code> module:</p> <div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">abc</span> <span class="kn">import</span> <span class="n">ABCMeta</span><span class="p">,</span> <span class="n">abstractmethod</span> <span class="k">class</span> <span class="nc">Base</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">ABCMeta</span><span class="p">):</span> <span class="nd">@abstractmethod</span> <span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">pass</span> <span class="nd">@abstractmethod</span> <span class="k">def</span> <span class="nf">bar</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">pass</span> </pre></div> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Concrete</span><span class="p">(</span><span class="n">Base</span><span class="p">):</span> <span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">pass</span> <span class="c1"># We forget to declare bar() again...</span> </pre></div> <p>This still behaves as expected and creates the correct class hierarchy:</p> <div class="codehilite"><pre><span></span><span class="k">assert</span> <span class="nb">issubclass</span><span class="p">(</span><span class="n">Concrete</span><span class="p">,</span> <span class="n">Base</span><span class="p">)</span> </pre></div> <p>Yet, we do get something awesome here. Subclasses of <code>Base</code> raise a <code>TypeError</code> <em>at instantiation time</em> whenever we forget to implement any abstract methods. The raised exception tells us which method or methods we’re missing:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">c</span> <span class="o">=</span> <span class="n">Concrete</span><span class="p">()</span> <span class="ne">TypeError</span><span class="p">:</span> <span class="s2">"Can't instantiate abstract class Concrete with abstract methods bar"</span> </pre></div> <p>Without <code>abc</code> we’d only get a <code>NotImplementedError</code> if a missing method is actually called. Being notified about missing methods at instantiation time is a great advantage. It makes it more difficult to write invalid subclasses. This might not be a big deal writing new code, but a few weeks or months down the line I promise it’ll be helpful.</p> <p>This pattern is not a full replacement for static-typing, of course. But in some situation it can make the resulting code more robust and more readily maintainable. Also, using <code>abc</code> states the code’s intent more clearly. I’d encourage you to read the <code>abc</code> module documentation and to start applying this pattern where it makes sense.</p></section><footer></footer></article>https://dbader.org/blog/abstract-base-classes-in-pythonMon, 21 Oct 2013 00:00:00 GMTFunctional linked lists in Pythonhttps://dbader.org/blog/functional-linked-lists-in-python<article><header><h1>Functional linked lists in Python</h1> <p>Linked lists are fundamental data structures that every programmer should know. This article explains how to implement a simple linked list data type in Python using a functional programming style.</p> </header><section></section><section><h2>Inspiration</h2> <p>The excellent book <a href="http://amzn.to/Rox4kI" target="_blank">Programming in Scala</a> inspired me to play with functional programming concepts in Python. I ended up implementing a basic <a href="https://en.wikipedia.org/wiki/Linked_list" target="_blank">linked list</a> data structure using a Lisp-like functional style that I want to share with you.</p> <p>I wrote most of this using <a href="http://omz-software.com/pythonista/" target="_blank">Pythonista</a> on my iPad. Pythonista is a Python IDE-slash-scratchpad and surprisingly fun to work with. It’s great when you’re stuck without a laptop and want to explore some CS fundamentals :)</p> <p>So without further ado, let’s dig into the implementation.</p> </section><section><h2>Constructing linked lists</h2> <p>Our linked list data structure consists of two fundamental building blocks: <code>Nil</code> and <code>cons</code>. <code>Nil</code> represents the empty list and serves as a sentinel for longer lists. The <code>cons</code> operation extends a list at the front by inserting a new value.</p> <p>The lists we construct using this method consist of nested 2-tuples. For example, the list <code>[1, 2, 3]</code> is represented by the expression <code>cons(1, cons(2, cons(3, Nil)))</code> which evaluates to the nested tuples <code>(1, (2, (3, Nil)))</code>.</p> <div class="codehilite"><pre><span></span><span class="n">Nil</span> <span class="o">=</span> <span class="bp">None</span> <span class="k">def</span> <span class="nf">cons</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">xs</span><span class="o">=</span><span class="n">Nil</span><span class="p">):</span> <span class="k">return</span> <span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">xs</span><span class="p">)</span> <span class="k">assert</span> <span class="n">cons</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">Nil</span><span class="p">)</span> <span class="k">assert</span> <span class="n">cons</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">Nil</span><span class="p">)))</span> <span class="o">==</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">Nil</span><span class="p">)))</span> </pre></div> <p>Why should we use this structure?</p> <p>First, the <a href="http://en.wikipedia.org/wiki/Cons" target="_blank">cons operation</a> is deeply rooted in the history of functional programming. From Lisp’s <em>cons cells</em> to ML’s and Scala’s <code>::</code> operator, cons is everywhere – you can even use it as a verb.</p> <p>Second, tuples are a convenient way to define simple data structures. For something as simple as our list building blocks, we don’t necessarily have to define a proper class. Also, it keeps this introduction short and sweet.</p> <p>Third, tuples are <em>immutable</em> in Python which means their state cannot be modified after creation. <a href="http://en.wikipedia.org/wiki/Immutable_object" target="_blank">Immutability</a> is often a desired property because it helps you write simpler and more thread-safe code. I like <a href="http://www.altdevblogaday.com/2012/04/26/functional-programming-in-c/" target="_blank">this article</a> by John Carmack where he shares his views on functional programming and immutability.</p> <p>Abstracting away the tuple construction using the <code>cons</code> function gives us a lot of flexibility on how lists are represented internally as Python objects. For example, instead of using 2-tuples we could store our elements in a chain of anonymous functions with Python’s <code>lambda</code> keyword.</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">cons</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">xs</span><span class="o">=</span><span class="n">Nil</span><span class="p">):</span> <span class="k">return</span> <span class="k">lambda</span> <span class="n">i</span><span class="p">:</span> <span class="n">x</span> <span class="k">if</span> <span class="n">i</span> <span class="o">==</span> <span class="mi">0</span> <span class="k">else</span> <span class="n">xs</span> </pre></div> <p>To write simpler tests for more complex list operations we’ll introduce the helper function <code>lst</code>. It allows us to define list instances using a more convenient syntax and without deeply nested <code>cons</code> calls.</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">lst</span><span class="p">(</span><span class="o">*</span><span class="n">xs</span><span class="p">):</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">xs</span><span class="p">:</span> <span class="k">return</span> <span class="n">Nil</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="n">cons</span><span class="p">(</span><span class="n">xs</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">lst</span><span class="p">(</span><span class="o">*</span><span class="n">xs</span><span class="p">[</span><span class="mi">1</span><span class="p">:]))</span> <span class="k">assert</span> <span class="n">lst</span><span class="p">()</span> <span class="o">==</span> <span class="n">Nil</span> <span class="k">assert</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">==</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">Nil</span><span class="p">)</span> <span class="k">assert</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span> <span class="o">==</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="n">Nil</span><span class="p">))))</span> </pre></div> </section><section><h2>Basic operations</h2> <p>All operations on linked lists can be expressed in terms of the three fundamental operations <code>head</code>, <code>tail</code>, and <code>is_empty</code>.</p> <ul> <li><code>head</code> returns the first element of a list.</li> <li><code>tail</code> returns a list containing all elements except the first.</li> <li><code>is_empty</code> returns <code>True</code> if the list contains zero elements.</li> </ul> <p>You’ll see later that these three operations are enough to implement a simple sorting algorithm like <em>insertion sort</em>.</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">head</span><span class="p">(</span><span class="n">xs</span><span class="p">):</span> <span class="k">return</span> <span class="n">xs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">assert</span> <span class="n">head</span><span class="p">(</span><span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span> <span class="o">==</span> <span class="mi">1</span> </pre></div> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">tail</span><span class="p">(</span><span class="n">xs</span><span class="p">):</span> <span class="k">return</span> <span class="n">xs</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="k">assert</span> <span class="n">tail</span><span class="p">(</span><span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="o">==</span> <span class="n">lst</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span> </pre></div> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">is_empty</span><span class="p">(</span><span class="n">xs</span><span class="p">):</span> <span class="k">return</span> <span class="n">xs</span> <span class="ow">is</span> <span class="n">Nil</span> <span class="k">assert</span> <span class="n">is_empty</span><span class="p">(</span><span class="n">Nil</span><span class="p">)</span> <span class="k">assert</span> <span class="ow">not</span> <span class="n">is_empty</span><span class="p">(</span><span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span> </pre></div> </section><section><h2>Length and concatenation</h2> <p>The <code>length</code> operation returns the number of elements in a given list. To find the length of a list we need to scan all of its <em>n</em> elements. Therefore this operation has a time complexity of <em>O(n)</em>.</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">length</span><span class="p">(</span><span class="n">xs</span><span class="p">):</span> <span class="k">if</span> <span class="n">is_empty</span><span class="p">(</span><span class="n">xs</span><span class="p">):</span> <span class="k">return</span> <span class="mi">0</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="mi">1</span> <span class="o">+</span> <span class="n">length</span><span class="p">(</span><span class="n">tail</span><span class="p">(</span><span class="n">xs</span><span class="p">))</span> <span class="k">assert</span> <span class="n">length</span><span class="p">(</span><span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="o">==</span> <span class="mi">4</span> <span class="k">assert</span> <span class="n">length</span><span class="p">(</span><span class="n">Nil</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span> </pre></div> <p><code>concat</code> takes two lists as arguments and concatenates them. The result of <code>concat(xs, ys)</code> is a new list that contains all elements in <code>xs</code> followed by all elements in <code>ys</code>. We implement the function with a simple divide and conquer algorithm.</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">concat</span><span class="p">(</span><span class="n">xs</span><span class="p">,</span> <span class="n">ys</span><span class="p">):</span> <span class="k">if</span> <span class="n">is_empty</span><span class="p">(</span><span class="n">xs</span><span class="p">):</span> <span class="k">return</span> <span class="n">ys</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="n">cons</span><span class="p">(</span><span class="n">head</span><span class="p">(</span><span class="n">xs</span><span class="p">),</span> <span class="n">concat</span><span class="p">(</span><span class="n">tail</span><span class="p">(</span><span class="n">xs</span><span class="p">),</span> <span class="n">ys</span><span class="p">))</span> <span class="k">assert</span> <span class="n">concat</span><span class="p">(</span><span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="n">lst</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="o">==</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span> </pre></div> </section><section><h2>Last, init, and list reversal</h2> <p>The basic operations <code>head</code> and <code>tail</code> have corresponding operations <code>last</code> and <code>init</code>. <code>last</code> returns the last element of a non-empty list and <code>init</code> returns all elements except the last one (the <em>initial</em> elements).</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">last</span><span class="p">(</span><span class="n">xs</span><span class="p">):</span> <span class="k">if</span> <span class="n">is_empty</span><span class="p">(</span><span class="n">tail</span><span class="p">(</span><span class="n">xs</span><span class="p">)):</span> <span class="k">return</span> <span class="n">head</span><span class="p">(</span><span class="n">xs</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="n">last</span><span class="p">(</span><span class="n">tail</span><span class="p">(</span><span class="n">xs</span><span class="p">))</span> <span class="k">assert</span> <span class="n">last</span><span class="p">(</span><span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="o">==</span> <span class="mi">4</span> </pre></div> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">init</span><span class="p">(</span><span class="n">xs</span><span class="p">):</span> <span class="k">if</span> <span class="n">is_empty</span><span class="p">(</span><span class="n">tail</span><span class="p">(</span><span class="n">tail</span><span class="p">(</span><span class="n">xs</span><span class="p">))):</span> <span class="k">return</span> <span class="n">cons</span><span class="p">(</span><span class="n">head</span><span class="p">(</span><span class="n">xs</span><span class="p">))</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="n">cons</span><span class="p">(</span><span class="n">head</span><span class="p">(</span><span class="n">xs</span><span class="p">),</span> <span class="n">init</span><span class="p">(</span><span class="n">tail</span><span class="p">(</span><span class="n">xs</span><span class="p">)))</span> <span class="k">assert</span> <span class="n">init</span><span class="p">(</span><span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="o">==</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> </pre></div> <p>Both operations need <em>O(n)</em> time to compute their result. Therefore it’s a good idea to reverse a list if you frequently use <code>last</code> or <code>init</code> to access its elements. The <code>reverse</code> function below implements list reversal, but in a slow way that takes <em>O(n²)</em> time.</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">reverse</span><span class="p">(</span><span class="n">xs</span><span class="p">):</span> <span class="k">if</span> <span class="n">is_empty</span><span class="p">(</span><span class="n">xs</span><span class="p">):</span> <span class="k">return</span> <span class="n">xs</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="n">concat</span><span class="p">(</span><span class="n">reverse</span><span class="p">(</span><span class="n">tail</span><span class="p">(</span><span class="n">xs</span><span class="p">)),</span> <span class="n">cons</span><span class="p">(</span><span class="n">head</span><span class="p">(</span><span class="n">xs</span><span class="p">),</span> <span class="n">Nil</span><span class="p">))</span> <span class="k">assert</span> <span class="n">reverse</span><span class="p">(</span><span class="n">Nil</span><span class="p">)</span> <span class="o">==</span> <span class="n">Nil</span> <span class="k">assert</span> <span class="n">reverse</span><span class="p">(</span><span class="n">cons</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">Nil</span><span class="p">))</span> <span class="o">==</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">Nil</span><span class="p">)</span> <span class="k">assert</span> <span class="n">reverse</span><span class="p">(</span><span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="o">==</span> <span class="n">lst</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="k">assert</span> <span class="n">reverse</span><span class="p">(</span><span class="n">reverse</span><span class="p">(</span><span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">)))</span> <span class="o">==</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span> </pre></div> </section><section><h2>Prefixes and suffixes</h2> <p>The following operations <code>take</code> and <code>drop</code> generalize <code>head</code> and <code>tail</code> by returning arbitrary prefixes and suffixes of a list. For example, <code>take(2, xs)</code> returns the first two elements of the list <code>xs</code> whereas <code>drop(3, xs)</code> returns everything except the last three elements in <code>xs</code>.</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">take</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">xs</span><span class="p">):</span> <span class="k">if</span> <span class="n">n</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="k">return</span> <span class="n">Nil</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="n">cons</span><span class="p">(</span><span class="n">head</span><span class="p">(</span><span class="n">xs</span><span class="p">),</span> <span class="n">take</span><span class="p">(</span><span class="n">n</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">tail</span><span class="p">(</span><span class="n">xs</span><span class="p">)))</span> <span class="k">assert</span> <span class="n">take</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="o">==</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> </pre></div> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">drop</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">xs</span><span class="p">):</span> <span class="k">if</span> <span class="n">n</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="k">return</span> <span class="n">xs</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="n">drop</span><span class="p">(</span><span class="n">n</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">tail</span><span class="p">(</span><span class="n">xs</span><span class="p">))</span> <span class="k">assert</span> <span class="n">drop</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span> <span class="o">==</span> <span class="n">lst</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="k">assert</span> <span class="n">drop</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="o">==</span> <span class="n">lst</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span> </pre></div> </section><section><h2>Element selection</h2> <p>Random element selection on linked lists doesn’t really make sense in terms of time complexity – accessing an element at index <em>n</em> requires <em>O(n)</em> time. However, the element access operation <code>apply</code> is simple to implement using <code>head</code> and <code>drop</code>.</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">apply</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">xs</span><span class="p">):</span> <span class="k">return</span> <span class="n">head</span><span class="p">(</span><span class="n">drop</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">xs</span><span class="p">))</span> <span class="k">assert</span> <span class="nb">apply</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="o">==</span> <span class="mi">1</span> <span class="k">assert</span> <span class="nb">apply</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="o">==</span> <span class="mi">3</span> </pre></div> </section><section><h2>More complex examples</h2> <p>The three basic operations <code>head</code>, <code>tail</code>, and <code>is_empty</code> are all we need to implement a simple (and slow) sorting algorithm like <a href="http://en.wikipedia.org/wiki/Insertion_sort" target="_blank">insertion sort</a>.</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">insert</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">xs</span><span class="p">):</span> <span class="k">if</span> <span class="n">is_empty</span><span class="p">(</span><span class="n">xs</span><span class="p">)</span> <span class="ow">or</span> <span class="n">x</span> <span class="o">&lt;=</span> <span class="n">head</span><span class="p">(</span><span class="n">xs</span><span class="p">):</span> <span class="k">return</span> <span class="n">cons</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">xs</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="n">cons</span><span class="p">(</span><span class="n">head</span><span class="p">(</span><span class="n">xs</span><span class="p">),</span> <span class="n">insert</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">tail</span><span class="p">(</span><span class="n">xs</span><span class="p">)))</span> <span class="k">assert</span> <span class="n">insert</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="o">==</span> <span class="n">lst</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span> <span class="k">assert</span> <span class="n">insert</span><span class="p">(</span><span class="mi">99</span><span class="p">,</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="o">==</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">99</span><span class="p">)</span> <span class="k">assert</span> <span class="n">insert</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="o">==</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span> <span class="k">def</span> <span class="nf">isort</span><span class="p">(</span><span class="n">xs</span><span class="p">):</span> <span class="k">if</span> <span class="n">is_empty</span><span class="p">(</span><span class="n">xs</span><span class="p">):</span> <span class="k">return</span> <span class="n">xs</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="n">insert</span><span class="p">(</span><span class="n">head</span><span class="p">(</span><span class="n">xs</span><span class="p">),</span> <span class="n">isort</span><span class="p">(</span><span class="n">tail</span><span class="p">(</span><span class="n">xs</span><span class="p">)))</span> <span class="k">assert</span> <span class="n">isort</span><span class="p">(</span><span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="o">==</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span> <span class="k">assert</span> <span class="n">isort</span><span class="p">(</span><span class="n">lst</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="o">==</span> <span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span> </pre></div> <p>The following <code>to_string</code> operation flattens the recursive structure of a given list and returns a Python-style string representation of its elements. This is useful for debugging and makes for a nice little programming exercise.</p> <div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">to_string</span><span class="p">(</span><span class="n">xs</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="s2">"["</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s2">", "</span><span class="p">,</span> <span class="n">postfix</span><span class="o">=</span><span class="s2">"]"</span><span class="p">):</span> <span class="k">def</span> <span class="nf">_to_string</span><span class="p">(</span><span class="n">xs</span><span class="p">):</span> <span class="k">if</span> <span class="n">is_empty</span><span class="p">(</span><span class="n">xs</span><span class="p">):</span> <span class="k">return</span> <span class="s2">""</span> <span class="k">elif</span> <span class="n">is_empty</span><span class="p">(</span><span class="n">tail</span><span class="p">(</span><span class="n">xs</span><span class="p">)):</span> <span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">head</span><span class="p">(</span><span class="n">xs</span><span class="p">))</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">head</span><span class="p">(</span><span class="n">xs</span><span class="p">))</span> <span class="o">+</span> <span class="n">sep</span> <span class="o">+</span> <span class="n">_to_string</span><span class="p">(</span><span class="n">tail</span><span class="p">(</span><span class="n">xs</span><span class="p">))</span> <span class="k">return</span> <span class="n">prefix</span> <span class="o">+</span> <span class="n">_to_string</span><span class="p">(</span><span class="n">xs</span><span class="p">)</span> <span class="o">+</span> <span class="n">postfix</span> <span class="k">assert</span> <span class="n">to_string</span><span class="p">(</span><span class="n">lst</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="o">==</span> <span class="s2">"[1, 2, 3, 4]"</span> </pre></div> </section><section><h2>Where to go from here</h2> <p>This article is more of a thought experiment than a guide on how to implement a useful linked list in Python. Keep in mind that the above code has severe restrictions and is not fit for real life use. For example, if you use this linked list implementation with larger example lists you’ll quickly hit recursion depth limits (CPython doesn’t optimize tail recursion).</p> <p>I spent a few fun hours playing with functional programming concepts in Python and I hope I inspired you to do the same. If you want to explore functional programming in ‘real world’ Python check out the following resources:</p> <ul> <li><a href="http://docs.python.org/3.3/howto/functional.html" target="_blank">The Python Functional Programming HOWTO</a></li> <li><a href="http://www.ibm.com/developerworks/linux/library/l-prog/index.html" target="_blank">Charming Python: Functional programming in Python</a></li> <li><a href="http://pyvideo.org/video/1799/functional-programming-with-python" target="_blank">Mike Müller’s PyCon talk: Functional programming with Python</a></li> </ul></section><footer></footer></article>https://dbader.org/blog/functional-linked-lists-in-pythonMon, 05 Aug 2013 00:00:00 GMTSetting up Sublime Text for Python developmenthttps://dbader.org/blog/setting-up-sublime-text-for-python-development<article><header><h1>Setting up Sublime Text for Python development</h1> <p>I recently started using Sublime Text 2 more and more as my main editor for Python development. This article explains my setup and some tweaks that make Python programmers happy.</p> </header><section><figure><img alt="My Sublime Text setup" src="/blog/figures/sublime-title.jpg" width="1400" height="572"></figure> </section><section><h2>Why Sublime Text?</h2> <p>I’ve been an avid user of <a href="https://github.com/textmate/textmate" target="_blank">TextMate</a> for a long time. It’s light-weight, open-source, and as a native OS X application it feels very Mac-esque. While TextMate is a great editor it seems very bare bones sometimes.</p> <p>For some projects I used the more beefy <a href="http://confluence.jetbrains.com/display/PYH/PyCharm+IDE+and+Python+Plugin+for+IntelliJ+IDEA" target="_blank">IntelliJ IDEA with the Python plug-in</a>. I especially like its debugger and test runner. Yet, often a full-blown IDE like IntelliJ is overkill when working on small to medium-sized projects.</p> <p>Over the last few weeks I began using <a href="http://www.sublimetext.com" target="_blank">Sublime Text</a> more and more. Once I took the time to set it up I felt very much at home. It’s really fast, receives steady updates, and – as a big bonus – fully cross-platform. What finally won me over compared to TextMate was Sublime’s great plug-in ecosystem. There are several plug-ins available that make Python development very smooth and enjoyable.</p> <p>I’m still switching editors on a per project basis now. But I noticed that for me Sublime Text seems to hit the sweet spot between a bare bones editor and a full-blown IDE for Python development.</p> <div class="update-box"> <h3>Update: Is Sublime Text still the best choice for Python devs?</h3> <p>Since I wrote this article quite a few things have changed in the world of Python editors and IDEs. If you’re wondering whether Sublime Text is still the right choice for you then this review article I wrote may be helpful:</p> <p>» <a href="sublime-text-for-python-development-2016-review">Sublime Text for Python development — My 2016 review</a> «</p> </div> </section><section><h2>Font choice</h2> <p><a href="http://font.ubuntu.com" target="_blank">Ubuntu Mono</a> is a great font. I’ve switched from primarily using <a href="http://en.wikipedia.org/wiki/Menlo_(typeface)" target="_blank">Menlo</a> a few days ago and I’m not regretting it so far.</p> <p>With Ubuntu Mono, I find font size 16 very comfortable to read on my 15-inch MacBook. At 1680 × 1050 the sidebar plus two editor views (wrapped at 80 characters) fit nicely next to each other.</p> <p>If you want to go nuclear on making the ideal font choice, this <a href="http://www.slant.co/topics/67/~what-are-the-best-programming-fonts" target="_blank">topic on slant.co</a> gives a good overview. It includes screenshots and download links for popular programming fonts.</p> </section><section><h2>Installed plug-ins</h2> <p>As mentioned before, Sublime has a very extensive plug-in ecosystem. I’m currently using the following plug-ins:</p> <ul> <li> <p><a href="http://wbond.net/sublime_packages/package_control" target="_blank">Package Control</a> A package manager for installing additional plug-ins directly from within Sublime. <strong>This should be the only package you have to install manually.</strong> All other packages listed here can be installed via Package Control. It’s also possible to update installed packages with Package Control. Simply think of it as the <code>apt-get</code> of Sublime packages.</p> </li> <li> <p><a href="https://github.com/theymaybecoders/sublime-tomorrow-theme" target="_blank">Color Scheme - Tomorrow Night</a> <em>Color schemes</em> determine the font colors used for syntax highlighting in the editor view. <em>Tomorrow</em> is a nice dark color scheme.</p> </li> <li> <p><a href="http://buymeasoda.github.io/soda-theme/" target="_blank">Theme - Soda Dark</a> <em>Themes</em> change the color and style of Sublime’s UI elements. This one fits perfectly with the Tomorrow color scheme.</p> </li> <li> <p><a href="https://github.com/titoBouzout/SideBarEnhancements" target="_blank">SideBarEnhancements</a> This plug-in provides additional context menu options in the sidebar, such as “New file” or “New Folder”. These should be in there by default, but they are not.</p> </li> <li> <p><a href="https://github.com/alienhard/SublimeAllAutocomplete" target="_blank">All Autocomplete</a> Sublime’s default autocomplete only considers words found in the current file. This plug-in extends the autocomplete word list to find matches across all open files.</p> </li> <li> <p><a href="https://github.com/Kronuz/SublimeCodeIntel" target="_blank">SublimeCodeIntel</a> Enhances autocomplete for some languages including Python. The plug-in also lets you jump to symbol definitions across files by pressing <code>alt</code> and then clicking on a symbol. Very handy.</p> </li> <li> <p><a href="https://github.com/wuub/SublimeREPL" target="_blank">SublimeREPL</a> Allows you to run a Python interpreter session in an editor view. I tend to use <a href="http://bpython-interpreter.org" target="_blank">bpython</a> in a separate terminal window but sometimes SublimeREPL is helpful.</p> </li> <li> <p><a href="sublime-text-gitgutter-review">GitGutter</a> Adds little icons to the editor’s gutter area indicating whether a line has been inserted, modified, or deleted according to Git. To get colored icons update your color scheme file as instructed in the GitGutter readme.</p> </li> <li> <p><a href="https://github.com/biermeester/Pylinter" target="_blank">Pylinter</a> This plug-in provides the best <a href="http://www.pylint.org" target="_blank">pylint</a> editor integration I’ve seen so far. It automatically lints <code>.py</code> files whenever they’re saved and displays pylint violations directly in the editor view. It also has a convenient shortcut that locally disables a pylint check by inserting a <code>#pylint: disable</code> comment. This plug-in sealed the deal for me.</p> </li> </ul> </section><section><h2>Preferences files</h2> <p>One of the nice things about Sublime Text is that it can be completely configured using simple JSON-based <a href="http://docs.sublimetext.info/en/latest/customization/settings.html" target="_blank">preferences files</a>. This allows you to easily transfer your settings to another system. I’ve also seen people use Dropbox to automatically synchronize their settings on every computer they’re using.</p> <p><strong><code>Preferences.sublime-settings</code></strong> configures Sublime’s look-and-feel and its built-in behavior. You can open the prefs file for editing within Sublime via <em>Preferences &gt; Settings – User</em>. I’m using the following settings:</p> <div class="codehilite"><pre><span></span><span class="p">{</span> <span class="c1">// Colors</span> <span class="s2">"color_scheme"</span><span class="o">:</span> <span class="s2">"Packages/Tomorrow Color Schemes/Tomorrow-Night.tmTheme"</span><span class="p">,</span> <span class="s2">"theme"</span><span class="o">:</span> <span class="s2">"Soda Dark.sublime-theme"</span><span class="p">,</span> <span class="c1">// Font</span> <span class="s2">"font_face"</span><span class="o">:</span> <span class="s2">"Ubuntu Mono"</span><span class="p">,</span> <span class="s2">"font_size"</span><span class="o">:</span> <span class="mf">16.0</span><span class="p">,</span> <span class="s2">"font_options"</span><span class="o">:</span> <span class="p">[</span><span class="s2">"subpixel_antialias"</span><span class="p">,</span> <span class="s2">"no_bold"</span><span class="p">],</span> <span class="s2">"line_padding_bottom"</span><span class="o">:</span> <span class="mi">0</span><span class="p">,</span> <span class="s2">"line_padding_top"</span><span class="o">:</span> <span class="mi">0</span><span class="p">,</span> <span class="c1">// Cursor style - no blinking and slightly wider than default</span> <span class="s2">"caret_style"</span><span class="o">:</span> <span class="s2">"solid"</span><span class="p">,</span> <span class="s2">"wide_caret"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="c1">// Editor view look-and-feel</span> <span class="s2">"draw_white_space"</span><span class="o">:</span> <span class="s2">"all"</span><span class="p">,</span> <span class="s2">"fold_buttons"</span><span class="o">:</span> <span class="kc">false</span><span class="p">,</span> <span class="s2">"highlight_line"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="s2">"auto_complete"</span><span class="o">:</span> <span class="kc">false</span><span class="p">,</span> <span class="s2">"show_minimap"</span><span class="o">:</span> <span class="kc">false</span><span class="p">,</span> <span class="s2">"show_full_path"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="c1">// Editor behavior</span> <span class="s2">"scroll_past_end"</span><span class="o">:</span> <span class="kc">false</span><span class="p">,</span> <span class="s2">"highlight_modified_tabs"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="s2">"find_selected_text"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="c1">// Word wrapping - follow PEP 8 recommendations</span> <span class="s2">"rulers"</span><span class="o">:</span> <span class="p">[</span> <span class="mi">72</span><span class="p">,</span> <span class="mi">79</span> <span class="p">],</span> <span class="s2">"word_wrap"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="s2">"wrap_width"</span><span class="o">:</span> <span class="mi">80</span><span class="p">,</span> <span class="c1">// Whitespace - no tabs, trimming, end files with \n</span> <span class="s2">"tab_size"</span><span class="o">:</span> <span class="mi">4</span><span class="p">,</span> <span class="s2">"translate_tabs_to_spaces"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="s2">"trim_trailing_white_space_on_save"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="s2">"ensure_newline_at_eof_on_save"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="c1">// Sidebar - exclude distracting files and folders</span> <span class="s2">"file_exclude_patterns"</span><span class="o">:</span> <span class="p">[</span> <span class="s2">".DS_Store"</span><span class="p">,</span> <span class="s2">"*.pid"</span><span class="p">,</span> <span class="s2">"*.pyc"</span> <span class="p">],</span> <span class="s2">"folder_exclude_patterns"</span><span class="o">:</span> <span class="p">[</span> <span class="s2">".git"</span><span class="p">,</span> <span class="s2">"__pycache__"</span><span class="p">,</span> <span class="s2">"env"</span><span class="p">,</span> <span class="s2">"env3"</span> <span class="p">]</span> <span class="p">}</span> </pre></div> <p><strong><code>Pylinter.sublime-settings</code></strong> configures the pylinter plug-in. I use the following settings to lint Python files automatically on save and to display graphical icons for lint violations:</p> <div class="codehilite"><pre><span></span><span class="p">{</span> <span class="c1">// Configure pylint's behavior</span> <span class="s2">"pylint_rc"</span><span class="o">:</span> <span class="s2">"/Users/daniel/dev/pylintrc"</span><span class="p">,</span> <span class="c1">// Show different icons for errors, warnings, etc.</span> <span class="s2">"use_icons"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="c1">// Automatically run Pylinter when saving a Python document</span> <span class="s2">"run_on_save"</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="c1">// Don't hide pylint messages when moving the cursor</span> <span class="s2">"message_stay"</span><span class="o">:</span> <span class="kc">true</span> <span class="p">}</span> </pre></div> </section><section><h2>Key bindings</h2> <p>Sublime’s key bindings are also fully user-configurable via JSON-based <code>sublime-keymap</code> preferences files. I’ve made a few changes to the default bindings to better serve my existing TextMate/IntelliJ muscle memory. You may not need to make changes to the key bindings at all. But if you want to, modifying them is very easy and transferable across platforms. I use the following additional key bindings:</p> <div class="codehilite"><pre><span></span><span class="p">[</span> <span class="c1">// Rebind "go to file" to cmd+shift+O</span> <span class="p">{</span> <span class="s2">"keys"</span><span class="o">:</span> <span class="p">[</span><span class="s2">"super+shift+o"</span><span class="p">],</span> <span class="s2">"command"</span><span class="o">:</span> <span class="s2">"show_overlay"</span><span class="p">,</span> <span class="s2">"args"</span><span class="o">:</span> <span class="p">{</span> <span class="s2">"overlay"</span><span class="o">:</span> <span class="s2">"goto"</span><span class="p">,</span> <span class="s2">"show_files"</span><span class="o">:</span> <span class="kc">true</span> <span class="p">}},</span> <span class="c1">// Rebind swap line up/down to cmd+shift+up/down</span> <span class="p">{</span> <span class="s2">"keys"</span><span class="o">:</span> <span class="p">[</span><span class="s2">"super+shift+up"</span><span class="p">],</span> <span class="s2">"command"</span><span class="o">:</span> <span class="s2">"swap_line_up"</span> <span class="p">},</span> <span class="p">{</span> <span class="s2">"keys"</span><span class="o">:</span> <span class="p">[</span><span class="s2">"super+shift+down"</span><span class="p">],</span> <span class="s2">"command"</span><span class="o">:</span> <span class="s2">"swap_line_down"</span> <span class="p">},</span> <span class="c1">// Delete a line with cmd+delete</span> <span class="p">{</span> <span class="s2">"keys"</span><span class="o">:</span> <span class="p">[</span><span class="s2">"super+backspace"</span><span class="p">],</span> <span class="s2">"command"</span><span class="o">:</span> <span class="s2">"run_macro_file"</span><span class="p">,</span> <span class="s2">"args"</span><span class="o">:</span> <span class="p">{</span> <span class="s2">"file"</span><span class="o">:</span> <span class="s2">"Packages/Default/Delete Line.sublime-macro"</span> <span class="p">}},</span> <span class="c1">// Reindent selection with cmd+alt+L</span> <span class="p">{</span> <span class="s2">"keys"</span><span class="o">:</span> <span class="p">[</span><span class="s2">"super+alt+l"</span><span class="p">],</span> <span class="s2">"command"</span><span class="o">:</span> <span class="s2">"reindent"</span><span class="p">}</span> <span class="p">]</span> </pre></div> </section><section><h2>Command line tools</h2> <p>Similarly to TextMate’s <code>mate</code>, Sublime Text includes a command line tool that allows you to open the editor from the shell. The tool called <code>subl</code> is not enabled by default. To make it available from any shell do the following:</p> <div class="codehilite"><pre><span></span>ln -s /Applications/Sublime<span class="se">\ </span>Text<span class="se">\ </span><span class="m">2</span>.app/Contents/SharedSupport/bin/subl /usr/local/bin/subl </pre></div> <p>To use Sublime as the default editor for interactive Git commands, for example when composing commit messages, add the following line to your <code>~/.profile</code>:</p> <div class="codehilite"><pre><span></span><span class="nb">export</span> <span class="nv">GIT_EDITOR</span><span class="o">=</span><span class="s2">"subl --wait --new-window"</span> </pre></div> <p>I’ve recorded a quick screencast that shows you how to do to this in some more detail: » <a href="using-sublime-text-as-your-git-editor">Using Sublime Text as your Git editor</a> «</p> </section><section><h2>Further inspiration</h2> <p>I hope this little guide was helpful to you. If you’ve got any comments or suggested improvements, please feel free to drop me a line on Twitter or send an email. I’d like to thank the following authors for their articles on setting up Sublime. They inspired my setup and may teach you some more tricks as well:</p> <ul> <li><a href="https://medium.com/kr-projects/4f06541322a8" target="_blank">Kenneth Reitz: Sublime Text 2 Love</a></li> <li><a href="http://drewbarontini.com/setup/sublime-text/" target="_blank">Drew Barontini: Sublime (2)</a></li> <li><a href="http://outofmemoryblog.blogspot.co.uk/2012/08/python-development-with-sublime-text-2.html" target="_blank">Filippo Pacifici: Python development with Sublime Text 2 tips and tricks</a></li> <li><a href="http://opensourcehacker.com/2012/05/11/sublime-text-2-tips-for-python-and-web-developers/" target="_blank">opensourcehacker.org: Sublime Text 2 tips for Python and web developers</a></li> <li><a href="/products/sublime-python-guide/">My Sublime Text 3 setup guide for Python developers</a></li> </ul></section><footer></footer></article>https://dbader.org/blog/setting-up-sublime-text-for-python-developmentSun, 12 May 2013 00:00:00 GMTMonochrome font rendering with FreeType and Pythonhttps://dbader.org/blog/monochrome-font-rendering-with-freetype-and-python<article><header><h1>Monochrome font rendering with FreeType and Python</h1> <p>For my Raspberry Pi internet radio project I needed a way to render text suitable for a low resolution monochrome LCD. This article describes how to render 1-bit text using FreeType and Python.</p> </header><section><figure><img alt="Hello, World." src="/blog/figures/monofont-helloworld.png" width="700" height="243"></figure> </section><section><h2>What we’re going to do</h2> <p>I’ve structured this tutorial into four main sections. First, there’ll be a brief introduction to the FreeType font rendering library. Second, we’ll attempt to render bitmap images of single characters. Third, we expand the previous functionality to render strings of multiple characters. Fourth, you’ll learn how to add support for kerning in order to improve the visual quality of your font rendering. The image above shows what results to expect from this tutorial.</p> <p>At the end of the article you’ll also find the full <a href="#example-code">example code for download</a>.</p> </section><section><h2>Update: What it looks like on a real display</h2> <figure><img alt="Font rendering on the Raspberry Pi" src="/blog/figures/piradio-board.jpg" width="700" height="393"></figure> <p>Some people have asked for images of the font rendering code being used with a real LCD. The above picture shows an earlier version of the code running on a Raspberry Pi Model B connected to the “Raspi-LCD” board by <a href="http://www.emsystech.de/produkt/raspi-lcd/" target="_blank">Emsystech Engineering</a>. The board contains a backlit 128 × 64 pixels display and five buttons. It comes with a C library that I use from Python with the <code>ctypes</code> module. The board is high quality and the haptics of the buttons are very good as well (they’re very clicky). I recommend it very much.</p> </section><section><h2>The FreeType library</h2> <p><a href="http://www.freetype.org" target="_blank">FreeType</a> is a popular open source C library for rendering fonts. Apparently more than a billion consumer devices with graphical display use FreeType to display text. The widespread use and high-quality output make the library an ideal choice for rendering text. FreeType works with the most common font formats like <a href="http://en.wikipedia.org/wiki/TrueType" target="_blank">TrueType</a> (.ttf files) and <a href="http://en.wikipedia.org/wiki/OpenType" target="_blank">OpenType</a> (.otf files).</p> <p>For using FreeType with Python I recommend <a href="http://code.google.com/p/freetype-py/" target="_blank">freetype-py</a> by <a href="http://www.loria.fr/~rougier/" target="_blank">Nicolas Rougier</a> which provides Python bindings for FreeType 2.</p> </section><section><h2>Rendering single characters</h2> <p>The first thing we want to achieve is to render monochromatic images for single characters. Once we can do that it’ll be reasonably simple to extend our code to display strings with multiple characters. To generate a bitmap image representation for a single character (<em>glyph</em>) with FreeType we need to do the following:</p> <ul> <li>Load the font file.</li> <li>Get the glyph bitmap for the given character.</li> <li>Unpack the glyph bitmap into a more convenient format.</li> </ul> <p>After this we’re able to render monochrome bitmaps for single characters. For example, the character <em>e</em> would look like this:</p> <figure><img alt='"e" glyph' src="/blog/figures/monofont-eglyph.png" width="683" height="182"></figure> <p>We’re going to work on this list from top to bottom and start by defining a class <code>Font</code> that represents a fixed-size font as loaded from a file on disk:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Font</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">size</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">face</span> <span class="o">=</span> <span class="n">freetype</span><span class="o">.</span><span class="n">Face</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">face</span><span class="o">.</span><span class="n">set_pixel_sizes</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">size</span><span class="p">)</span> <span class="k">def</span> <span class="nf">glyph_for_character</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">char</span><span class="p">):</span> <span class="c1"># Let FreeType load the glyph for the given character and</span> <span class="c1"># tell it to render a monochromatic bitmap representation.</span> <span class="bp">self</span><span class="o">.</span><span class="n">face</span><span class="o">.</span><span class="n">load_char</span><span class="p">(</span><span class="n">char</span><span class="p">,</span> <span class="n">freetype</span><span class="o">.</span><span class="n">FT_LOAD_RENDER</span> <span class="o">|</span> <span class="n">freetype</span><span class="o">.</span><span class="n">FT_LOAD_TARGET_MONO</span><span class="p">)</span> <span class="k">return</span> <span class="n">Glyph</span><span class="o">.</span><span class="n">from_glyphslot</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">face</span><span class="o">.</span><span class="n">glyph</span><span class="p">)</span> <span class="k">def</span> <span class="nf">render_character</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">char</span><span class="p">):</span> <span class="n">glyph</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">glyph_for_character</span><span class="p">(</span><span class="n">char</span><span class="p">)</span> <span class="k">return</span> <span class="n">glyph</span><span class="o">.</span><span class="n">bitmap</span> </pre></div> <p>We’ve used a yet undefined class called <code>Glyph</code> in the <code>glyph_for_character()</code> method. The <code>Glyph</code> class is our wrapper around FreeType’s glyph representations and primarily helps with unpacking FreeType’s bitmap format for monochrome glyphs. FreeType stores monochrome bitmaps in a packed format where multiple pixels are encoded within a single byte. This format is slightly inconvenient to use because it involves some bit-fiddling.</p> <p>To give an example on how to access individual pixels in this format we’re going to unpack the glyph bitmap into a Python <code>bytearray</code>. In this unpacked format each pixel is represented by a single byte. A value of <code>0</code> means that the pixel is <em>off</em> and any other value means that it is <em>on</em>. The <code>Glyph</code> class with the bitmap unpacking code looks as follows:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Glyph</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pixels</span><span class="p">,</span> <span class="n">width</span><span class="p">,</span> <span class="n">height</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">bitmap</span> <span class="o">=</span> <span class="n">Bitmap</span><span class="p">(</span><span class="n">width</span><span class="p">,</span> <span class="n">height</span><span class="p">,</span> <span class="n">pixels</span><span class="p">)</span> <span class="nd">@staticmethod</span> <span class="k">def</span> <span class="nf">from_glyphslot</span><span class="p">(</span><span class="n">slot</span><span class="p">):</span> <span class="sd">"""Construct and return a Glyph object from a FreeType GlyphSlot."""</span> <span class="n">pixels</span> <span class="o">=</span> <span class="n">Glyph</span><span class="o">.</span><span class="n">unpack_mono_bitmap</span><span class="p">(</span><span class="n">slot</span><span class="o">.</span><span class="n">bitmap</span><span class="p">)</span> <span class="n">width</span><span class="p">,</span> <span class="n">height</span> <span class="o">=</span> <span class="n">slot</span><span class="o">.</span><span class="n">bitmap</span><span class="o">.</span><span class="n">width</span><span class="p">,</span> <span class="n">slot</span><span class="o">.</span><span class="n">bitmap</span><span class="o">.</span><span class="n">rows</span> <span class="k">return</span> <span class="n">Glyph</span><span class="p">(</span><span class="n">pixels</span><span class="p">,</span> <span class="n">width</span><span class="p">,</span> <span class="n">height</span><span class="p">)</span> <span class="nd">@staticmethod</span> <span class="k">def</span> <span class="nf">unpack_mono_bitmap</span><span class="p">(</span><span class="n">bitmap</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Unpack a freetype FT_LOAD_TARGET_MONO glyph bitmap into a bytearray where</span> <span class="sd"> each pixel is represented by a single byte.</span> <span class="sd"> """</span> <span class="c1"># Allocate a bytearray of sufficient size to hold the glyph bitmap.</span> <span class="n">data</span> <span class="o">=</span> <span class="nb">bytearray</span><span class="p">(</span><span class="n">bitmap</span><span class="o">.</span><span class="n">rows</span> <span class="o">*</span> <span class="n">bitmap</span><span class="o">.</span><span class="n">width</span><span class="p">)</span> <span class="c1"># Iterate over every byte in the glyph bitmap. Note that we're not</span> <span class="c1"># iterating over every pixel in the resulting unpacked bitmap --</span> <span class="c1"># we're iterating over the packed bytes in the input bitmap.</span> <span class="k">for</span> <span class="n">y</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">bitmap</span><span class="o">.</span><span class="n">rows</span><span class="p">):</span> <span class="k">for</span> <span class="n">byte_index</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">bitmap</span><span class="o">.</span><span class="n">pitch</span><span class="p">):</span> <span class="c1"># Read the byte that contains the packed pixel data.</span> <span class="n">byte_value</span> <span class="o">=</span> <span class="n">bitmap</span><span class="o">.</span><span class="n">buffer</span><span class="p">[</span><span class="n">y</span> <span class="o">*</span> <span class="n">bitmap</span><span class="o">.</span><span class="n">pitch</span> <span class="o">+</span> <span class="n">byte_index</span><span class="p">]</span> <span class="c1"># We've processed this many bits (=pixels) so far. This determines</span> <span class="c1"># where we'll read the next batch of pixels from.</span> <span class="n">num_bits_done</span> <span class="o">=</span> <span class="n">byte_index</span> <span class="o">*</span> <span class="mi">8</span> <span class="c1"># Pre-compute where to write the pixels that we're going</span> <span class="c1"># to unpack from the current byte in the glyph bitmap.</span> <span class="n">rowstart</span> <span class="o">=</span> <span class="n">y</span> <span class="o">*</span> <span class="n">bitmap</span><span class="o">.</span><span class="n">width</span> <span class="o">+</span> <span class="n">byte_index</span> <span class="o">*</span> <span class="mi">8</span> <span class="c1"># Iterate over every bit (=pixel) that's still a part of the</span> <span class="c1"># output bitmap. Sometimes we're only unpacking a fraction of a byte</span> <span class="c1"># because glyphs may not always fit on a byte boundary. So we make sure</span> <span class="c1"># to stop if we unpack past the current row of pixels.</span> <span class="k">for</span> <span class="n">bit_index</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">min</span><span class="p">(</span><span class="mi">8</span><span class="p">,</span> <span class="n">bitmap</span><span class="o">.</span><span class="n">width</span> <span class="o">-</span> <span class="n">num_bits_done</span><span class="p">)):</span> <span class="c1"># Unpack the next pixel from the current glyph byte.</span> <span class="n">bit</span> <span class="o">=</span> <span class="n">byte_value</span> <span class="o">&amp;</span> <span class="p">(</span><span class="mi">1</span> <span class="o">&lt;&lt;</span> <span class="p">(</span><span class="mi">7</span> <span class="o">-</span> <span class="n">bit_index</span><span class="p">))</span> <span class="c1"># Write the pixel to the output bytearray. We ensure that `off`</span> <span class="c1"># pixels have a value of 0 and `on` pixels have a value of 1.</span> <span class="n">data</span><span class="p">[</span><span class="n">rowstart</span> <span class="o">+</span> <span class="n">bit_index</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span> <span class="k">if</span> <span class="n">bit</span> <span class="k">else</span> <span class="mi">0</span> <span class="k">return</span> <span class="n">data</span> </pre></div> <p>Clearly, the most important parts of <code>Glyph</code> class are in the bitmap unpacking code. Once we’re rendering multi-character strings we’ll extend the class with additional metadata, such as the <em>advance width</em> that tells us the horizontal distance between glyphs.</p> <p>The final part that’s missing is the <code>Bitmap</code> class. It’s a simple helper class for working with <code>bytearray</code>-based bitmaps:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Bitmap</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> A 2D bitmap image represented as a list of byte values. Each byte indicates</span> <span class="sd"> the state of a single pixel in the bitmap. A value of 0 indicates that</span> <span class="sd"> the pixel is `off` and any other value indicates that it is `on`.</span> <span class="sd"> """</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">width</span><span class="p">,</span> <span class="n">height</span><span class="p">,</span> <span class="n">pixels</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">width</span> <span class="o">=</span> <span class="n">width</span> <span class="bp">self</span><span class="o">.</span><span class="n">height</span> <span class="o">=</span> <span class="n">height</span> <span class="bp">self</span><span class="o">.</span><span class="n">pixels</span> <span class="o">=</span> <span class="n">pixels</span> <span class="ow">or</span> <span class="nb">bytearray</span><span class="p">(</span><span class="n">width</span> <span class="o">*</span> <span class="n">height</span><span class="p">)</span> <span class="k">def</span> <span class="fm">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="sd">"""Return a string representation of the bitmap's pixels."""</span> <span class="n">rows</span> <span class="o">=</span> <span class="s1">''</span> <span class="k">for</span> <span class="n">y</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">height</span><span class="p">):</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">width</span><span class="p">):</span> <span class="n">rows</span> <span class="o">+=</span> <span class="s1">'*'</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">pixels</span><span class="p">[</span><span class="n">y</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">width</span> <span class="o">+</span> <span class="n">x</span><span class="p">]</span> <span class="k">else</span> <span class="s1">' '</span> <span class="n">rows</span> <span class="o">+=</span> <span class="s1">'</span><span class="se">\n</span><span class="s1">'</span> <span class="k">return</span> <span class="n">rows</span> </pre></div> <p>The class allows us to quickly experiment with font rendering in the Python REPL. Calling <code>repr()</code> on a <code>Bitmap</code> object returns a textual representation of the 2D image encoded in the bitmap. This is going to be very helpful when we start debugging our font rendering code. Next, let’s actually try to render a single glyph bitmap:</p> <div class="codehilite"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">fnt</span> <span class="o">=</span> <span class="n">Font</span><span class="p">(</span><span class="s2">"helvetica.ttf"</span><span class="p">,</span> <span class="mi">24</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">ch</span> <span class="o">=</span> <span class="n">fnt</span><span class="o">.</span><span class="n">render_character</span><span class="p">(</span><span class="s2">"e"</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="nb">repr</span><span class="p">(</span><span class="n">ch</span><span class="p">)</span> <span class="o">*****</span> <span class="o">*******</span> <span class="o">***</span> <span class="o">***</span> <span class="o">***</span> <span class="o">**</span> <span class="o">**</span> <span class="o">**</span> <span class="o">***********</span> <span class="o">***********</span> <span class="o">**</span> <span class="o">**</span> <span class="o">**</span> <span class="o">**</span> <span class="o">**</span> <span class="o">********</span> <span class="o">*****</span> </pre></div> <p>Great, that means our glyph rendering code works. The most complicated thing here was the bitmap unpacking code. We now continue with rendering strings with multiple characters.</p> </section><section><h2>Rendering multiple characters</h2> <p>Now that we know how to render single character glyphs we’re going to extend that functionality into rendering strings with several characters. The critical part here is glyph placement, that is, ensuring that all characters line up correctly. To render multi-character strings we make the following changes to the existing code:</p> <ul> <li>Extend the <code>Glyph</code> class with additional metadata that tells us how characters are placed next to each other (<em>advance width</em>, <em>top-side bearing</em>, <em>ascent</em>, and <em>descent</em>).</li> <li>Implement a two pass algorithm for rendering strings:<ul> <li>Pass 1: Compute the dimensions of the bitmap for a given string.</li> <li>Pass 2: Successively draw the glyph for each character into an output bitmap.</li> </ul> </li> </ul> <p>Once we’ve completed these steps we’ll be able to render strings such as this one:</p> <figure><img alt='"hello"' src="/blog/figures/monofont-hello.png" width="700" height="263"></figure> <p>We start with extending the <code>Glyph</code> class with fields for the glyph’s advance width, top-side bearing, ascent, and descent. I’ll briefly explain the purpose of these fields before we continue. If you want to learn more about these glyph metrics take a look at the <a href="http://www.freetype.org/freetype2/docs/glyphs/glyphs-3.html" target="_blank">FreeType documentation</a>.</p> <p>The <em>advance width</em> tells us where to place the next character horizontally, that is, how many pixels we move to the right (or to the left) to draw the next glyph.</p> <p>The <em>ascent</em>, <em>descent</em>, and the <em>top-side bearing</em> determine the vertical placement of the glyph. To understand vertical glyph placement the concept of the <em>baseline</em> is very important. The baseline is defined to be the line upon which most letters sit. The <em>ascent</em> and <em>descent</em> determine how the glyph should be placed relative to the baseline.</p> <p>In western typography most letters extend above the baseline. We say that they have a positive ascent. Some letters, such as <em>g</em>, extend below the baseline. This means that both their ascent <em>and</em> descent are positive. Of course, other mixtures are also possible, for example, there may be letters with an ascent of zero but a positive descent, and so on.</p> <p>The <em>top-side bearing</em> is the vertical distance from the glyph’s baseline to its bitmap’s top-most scanline. We need this value to compute the glyph’s ascent and descent.</p> <p>While these glyph metrics seem straightforward to compute, it took me a few tries and some pencil drawing to get them right. The updated version of the <code>Glyph</code> class with added metrics looks like this:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Glyph</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pixels</span><span class="p">,</span> <span class="n">width</span><span class="p">,</span> <span class="n">height</span><span class="p">,</span> <span class="n">top</span><span class="p">,</span> <span class="n">advance_width</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">bitmap</span> <span class="o">=</span> <span class="n">Bitmap</span><span class="p">(</span><span class="n">width</span><span class="p">,</span> <span class="n">height</span><span class="p">,</span> <span class="n">pixels</span><span class="p">)</span> <span class="c1"># The glyph bitmap's top-side bearing, i.e. the vertical distance from the</span> <span class="c1"># baseline to the bitmap's top-most scanline.</span> <span class="bp">self</span><span class="o">.</span><span class="n">top</span> <span class="o">=</span> <span class="n">top</span> <span class="c1"># Ascent and descent determine how many pixels the glyph extends</span> <span class="c1"># above or below the baseline.</span> <span class="bp">self</span><span class="o">.</span><span class="n">descent</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">height</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">top</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">ascent</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="nb">max</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">top</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">height</span><span class="p">)</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">descent</span><span class="p">)</span> <span class="c1"># The advance width determines where to place the next character</span> <span class="c1"># horizontally, that is, how many pixels we move to the right</span> <span class="c1"># to draw the next glyph.</span> <span class="bp">self</span><span class="o">.</span><span class="n">advance_width</span> <span class="o">=</span> <span class="n">advance_width</span> <span class="nd">@property</span> <span class="k">def</span> <span class="nf">width</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">bitmap</span><span class="o">.</span><span class="n">width</span> <span class="nd">@property</span> <span class="k">def</span> <span class="nf">height</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">bitmap</span><span class="o">.</span><span class="n">height</span> </pre></div> <p>Next, we’re going to work on the <code>Font</code> class and extend it with a two-pass algorithm for rendering multi-character strings.</p> <p>The <strong>first pass</strong> computes the space occupied by the given string, that is, the dimensions of the given text as if it were rendered into a bitmap. Besides the width and height of the resulting bitmap in pixels, we also need to know the position of the baseline for correct vertical glyph placement.</p> <p>We compute the overall width by summing up the advance widths for all glyphs. The overall height is determined by the maximum ascent and descent. The baseline of a multi-character string equals the maximum descent of all glyphs within<sup id="fnref:1"><a class="footnote-ref" href="#fn:1" rel="footnote">1</a></sup> the string.</p> <p>The resulting function <code>text_dimensions()</code> looks as follows:</p> <div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Font</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="k">def</span> <span class="nf">text_dimensions</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">text</span><span class="p">):</span> <span class="sd">"""</span> <span class="sd"> Return (width, height, baseline) of `text` rendered in the current font.</span> <span class="sd"> """</span> <span class="n">width</span> <span class="o">=</span> <span class="mi">0</span> <span class="n">max_ascent</span> <span class="o">=</span> <span class="mi">0</span> <span class="n">max_descent</span> <span class="o">=</span> <span class="mi">0</span> <span class="n">previous_char</span> <span class="o">=</span> <span class="bp">None</span> <span class="c1"># For each character in the text string we get the glyph</span> <span class="c1"># and update the overall dimensions of the resulting bitmap.</span> <span class="k">for</span> <span class="n">char</span> <span class="ow">in</span> <span class="n">text</span><span class="p">:</span> <span class="n">glyph</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">glyph_for_character</span><span class="p">(</span><span class="n">char</span><span class="p">)</span> <span class="n">max_ascent</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="n">max_ascent</span><span class="p">,</span> <span class="n">glyph</span><span class="o">.</span><span class="n">ascent</span><span class="p">)</span> <span class="n">max_descent</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="n">max_descent</span><span class="p">,</span> <span class="n">glyph</span><span class="o">.</span><span class="n">descent</span><span class="p">)</span> <span class="n">width</span> <span class="o">+=</span> <span class="n">glyph</span><span class="o">.</span><span class="n">advance_width</span> <span class="n">previous_char</span> <span class="o">=</span> <span class="n">char</span> <span class="n">height</span> <span class="o">=</span> <span class="n">max_ascent</span> <span class="o">+</span> <span class="n">max_descent</span> <span class="k">return</span> <span class="p">(</span><span class="n">width</span>&l