Dan Bader

Catching bogus Python asserts on CI

It’s easy to accidentally write Python assert statements that always evaluate to true. Here’s how to avoid this mistake and catch bad assertions as part of your continuous integration build.

Bad Python assert example

Asserts that are always true

There’s an easy mistake to make with Python’s assert:

When you pass it a tuple as the first argument, the assertion always evaluates as true and therefore never fails.

To give you a simple example, this assertion will never fail:

assert(1 == 2, 'This should fail')

Especially for developers new to Python this can be a surprising result.

Let’s take a quick look at the syntax for Python’s assert statement to find out why this assertion is bogus and will never fail.

Here’s the syntax for assert from the Python docs:

assert_stmt ::=  "assert" expression1 ["," expression2]

expression1 is the condition we test, and the optional expression2 is an error message that’s displayed if the assertion fails.

At execution time, the Python interpreter transforms each assert statement into the following:

if __debug__:
   if not expression1:
        raise AssertionError(expression2)

Let’s take the broken example assertion and apply the transform.

assert(1 == 2, 'This should fail')

becomes the following:

if __debug__:
    if not (1 == 2, 'This should fail'):
        raise AssertionError()

Now we can see where things go wrong.

Because assert is a statement and not a function call, the parentheses lead to expression1 containing the whole tuple (1 == 2, 'This should fail').

Non-empty tuples are always truthy in Python and therefore the assertion will always evaluate to true, which is maybe not what we expected.

This behavior can make writing multi-line asserts error-prone. Imagine we have the following assert statement somewhere in our test code:

assert (
    counter == 10,
    'It should have counted all the items'
)

This test case would never catch an incorrect result. The assertion always evaluates to True regardless of the state of the counter variable.

Pytest encourages you to use plain assert statements in unit tests instead of the assertEquals, assertTrue, …, assertXYZ methods provided by the unittest module in the standard library.

It’s relatively easy to accidentally write bad multi-line asserts this way. They can lead to broken test cases that give a falls sense of security in our test code.

Why isn’t this a warning in Python?

Well, it actually is a syntax warning in Python 2.6+:

>>> assert (1==2, 'This should fail')
<input>:2: SyntaxWarning: assertion is always true, perhaps remove parentheses?

The trouble is that when you use the py.test test runner, these warnings are hidden:

$ cat test.py
def test_foo():
    assert (1==2, 'This should fail')

$ py.test -v test.py
======= test session starts =======
platform darwin -- Python 3.5.1, pytest-2.9.0,
py-1.4.31, pluggy-0.3.1
rootdir: /Users/daniel/dev/, inifile: pytest.ini
collected 1 items

test.py::test_foo PASSED

======= 1 passed in 0.03 seconds =======

This makes it easy to write a broken assert in a test case. The assertion will always pass and it’s hard to notice that anything is wrong because py.test hides the syntax warning.

The solution: Code linting

Luckily, the “bogus assert” issue is the kind of problem that can be easily caught with a good code linter.

pyflakes is an excellent Python linter that strikes a nice balance between helping you catch bugs and avoiding false positives. I highly recommend it.

Starting with pyflakes 1.1.0 asserts against a tuple cause a warning, which will help you find bad assert statements in your code.

I also recommend to run the linter as part of the continuous integration build. The build should fail if the program isn’t lint free. This helps avoid issues when developers forget to run the linter locally before committing their code.

Besides going with code linting as a purely technical solution, it’s also a good idea to adopt the following technique when writing tests:

When writing a new unit test, always make sure the test actually fails when the code under test is broken or delivers the wrong result.

The interim solution

If you’re not using pyflakes but still want to be informed of faulty asserts you can use the following grep command as part of your build process:

(egrep 'assert *\(' --include '*.py' --recursive my_app/ || exit 0 && exit 1;)

This will fail the build if there’s an assert statement followed by open parentheses. This is obviously not perfect and you should use pyflakes, but in a pinch it’s better than nothing.

(Go with pyflakes if you can! 😃)

💓🐍 Love Python? Show It With Some Unique Python Swag Every Pythonista needs a great coffee (or tea!) mug. That’s why my wife Anja and I started an online store with unique mugs for Python devs: » Check it out at nerdlettering.com

Improve Your Python with a fresh 🐍 Python Trick 💌 every couple of days

🔒 No spam ever. Unsubscribe any time.

This article was filed under: programming, python, tdd, and testing.

Related Articles:
  • OS X notifications for your pytest runs – This article shows you how to use the pytest-osxnotify, a plugin for pytest that adds native Mac OS X notifications to the pytest terminal runner.
  • A countdown timer extension for Alfred – I wrote a countdown timer extension for the Alfred application launcher for OS X. The extension is open-source, written in Python and uses Mountain Lion’s user notifications.
  • Monochrome font rendering with FreeType and Python – For my Raspberry Pi internet radio project I needed a way to render text suitable for a low resolution monochrome LCD. This article describes how to render 1-bit text using FreeType and Python.
  • Setting up Sublime Text for Python development – I recently started using Sublime Text 2 more and more as my main editor for Python development. This article explains my setup and some tweaks that make Python programmers happy.
  • Functional linked lists in Python – Linked lists are fundamental data structures that every programmer should know. This article explains how to implement a simple linked list data type in Python using a functional programming style.
Latest Articles:

Peer-to-Peer Learning for Pythonistas PythonistaCafe is an invite-only, online community of Python and software development enthusiasts helping each other succeed and grow: » Learn more at pythonistacafe.com

← Browse All Articles