Catching bogus Python asserts on CI

Asserts that are always true

There’s an easy mistake to make with Python’s assert:

When you pass it a tuple as the first argument, the assertion always evaluates as true and therefore never fails.

To give you a simple example, this assertion will never fail:

assert(1 == 2, 'This should fail')

Especially for developers new to Python this can be a surprising result.

Let’s take a quick look at the syntax for Python’s assert statement to find out why this assertion is bogus and will never fail.

Here’s the syntax for assert from the Python docs:

assert_stmt ::=  "assert" expression1 ["," expression2]

expression1 is the condition we test, and the optional expression2 is an error message that’s displayed if the assertion fails.

At execution time, the Python interpreter transforms each assert statement into the following:

if __debug__:
   if not expression1:
        raise AssertionError(expression2)

Let’s take the broken example assertion and apply the transform.

assert(1 == 2, 'This should fail')

becomes the following:

if __debug__:
    if not (1 == 2, 'This should fail'):
        raise AssertionError()

Now we can see where things go wrong.

Because assert is a statement and not a function call, the parentheses lead to expression1 containing the whole tuple (1 == 2, 'This should fail').

Non-empty tuples are always truthy in Python and therefore the assertion will always evaluate to true, which is maybe not what we expected.

This behavior can make writing multi-line asserts error-prone. Imagine we have the following assert statement somewhere in our test code:

assert (
    counter == 10,
    'It should have counted all the items'
)

This test case would never catch an incorrect result. The assertion always evaluates to True regardless of the state of the counter variable.

Pytest encourages you to use plain assert statements in unit tests instead of the assertEquals, assertTrue, …, assertXYZ methods provided by the unittest module in the standard library.

It’s relatively easy to accidentally write bad multi-line asserts this way. They can lead to broken test cases that give a falls sense of security in our test code.

Why isn’t this a warning in Python?

Well, it actually is a syntax warning in Python 2.6+:

>>> assert (1==2, 'This should fail')
<input>:2: SyntaxWarning: assertion is always true, perhaps remove parentheses?

The trouble is that when you use the py.test test runner, these warnings are hidden:

$ cat test.py
def test_foo():
    assert (1==2, 'This should fail')

$ py.test -v test.py
======= test session starts =======
platform darwin -- Python 3.5.1, pytest-2.9.0,
py-1.4.31, pluggy-0.3.1
rootdir: /Users/daniel/dev/, inifile: pytest.ini
collected 1 items

test.py::test_foo PASSED

======= 1 passed in 0.03 seconds =======

This makes it easy to write a broken assert in a test case. The assertion will always pass and it’s hard to notice that anything is wrong because py.test hides the syntax warning.

The solution: Code linting

Luckily, the “bogus assert” issue is the kind of problem that can be easily caught with a good code linter.

pyflakes is an excellent Python linter that strikes a nice balance between helping you catch bugs and avoiding false positives. I highly recommend it.

Starting with pyflakes 1.1.0 asserts against a tuple cause a warning, which will help you find bad assert statements in your code.

I also recommend to run the linter as part of the continuous integration build. The build should fail if the program isn’t lint free. This helps avoid issues when developers forget to run the linter locally before committing their code.

Besides going with code linting as a purely technical solution, it’s also a good idea to adopt the following technique when writing tests:

When writing a new unit test, always make sure the test actually fails when the code under test is broken or delivers the wrong result.

The interim solution

If you’re not using pyflakes but still want to be informed of faulty asserts you can use the following grep command as part of your build process:

(egrep 'assert *\(' --include '*.py' --recursive my_app/ || exit 0 && exit 1;)

This will fail the build if there’s an assert statement followed by open parentheses. This is obviously not perfect and you should use pyflakes, but in a pinch it’s better than nothing.

(Go with pyflakes if you can! 😃)