If you enjoyed this book considering buying a copy

Chapter 3: Introduction to Pytest #

Alfredo Deza

I remember when I started writing tests to be confused on how to run them. At the time, a project called nose was what everyone used, and it is what I ended up picking up. It still required one to use Python’s unit test library, which forces one to use classes and inheritance for tests. I was starting with Python and didn’t have a good grasp on classes, let alone on inheritance! The experience wasn’t great; I dreaded writing tests.

At some point, I found myself having some issue in a test that forced me to try out the Pytest project to check if it was a problem in my code or a problem with the test runner. Pytest didn’t have any issues, and I found the output and experience so lovely that I kept using it. A few years into that, the Nose project stopped being actively maintained, and Pytest shot in popularity and interest.

The community and plugin ecosystem for Pytest is tremendous. There are plugins for everything, the ease of use of the test runner is excellent, and more than anything, the tool is still straightforward to use while at the same time it offers lots of advanced features with its framework. Not requiring the user to use any of its advanced configurations is a great thing!

For this chapter, make sure to create a new virtual environment (virtualenv) and install pytest after activating it:

$ python3 -m venv venv
$ source venv/bin/activate
$ pip install "pytest==5.3.2"

After installing, the pytest version should look

$ pytest --version
This is pytest version 5.3.2, imported from venv/lib/python3.6/site-packages/pytest/__init__.py

The most simple test possible #

Pytest allows you to write test functions, which is a major feature if you are getting started and haven’t worked with classes before. In some cases, I write test functions even though I am very familiar with classes. The most simple test possible with pytest is then a function one:

def test_simplest():
    assert True

A function that doesn’t do anything except asserting the True value. The example is silly, no code is getting executed other than the test, and the test is going to pass unless Python is completely broken and True starts meaning something else. But it is still valid to demonstrate how simple a test can be. Compare it to how it would look with the unittest library:

from unittest import TestCase

class TestSimple(TestCase):

    def test_simple(self):
        self.assertTrue(True)

The simplicity award goes to pytest.

For a more realistic example, I’m going to write tests for a small function in one of my projects. It intends to check if a data structure is empty or not, returning a boolean:

def is_empty(value):
    return len(value) == 0

Several things can happen here, and I want to ensure that this small function is going to be able to handle several different inputs, even some that I know are unexpected. Normally, the function would deal with data structures like lists, dictionaries, and tuples. When writing tests, it makes you think about what would happen if you force a function like this to behave with odd inputs like a string or a boolean. My first tests are going to be for the expected inputs that are all empty:

def test_empty_list():
    assert is_empty([]) is True

def test_empty_dict():
    assert is_empty({}) is True

Run pytest to see them pass:

============================= test session starts ==============================
platform darwin -- Python 3.6.2, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
collected 2 items

test_is_empty.py::test_empty_list PASSED                                 [ 50%]
test_is_empty.py::test_empty_dict PASSED                                 [100%]

======================= 2 passed, 2 deselected in 0.01s ========================

Now add some tests to check if the utility function returns a False when the data structures have some items:

def test_list_is_not_empty()
    assert is_empty([1,2,3]) is False

def test_dict_is_not_empty():
    assert is_empty({"item": 1}) is False

Now run pytest again, to see everything execute:

============================= test session starts ==============================
platform darwin -- Python 3.6.2, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
collected 4 items

test_is_empty.py::test_empty_list PASSED                                 [ 25%]
test_is_empty.py::test_empty_dict PASSED                                 [ 50%]
test_is_empty.py::test_list_is_not_empty PASSED                          [ 75%]
test_is_empty.py::test_dict_is_not_empty PASSED                          [100%]

============================== 4 passed in 0.02s ===============================

When writing tests, it makes you think about what would happen if you force a function like this to behave with odd inputs like a string or a boolean. The first tests were about correctness in expected inputs, but I want a function that can default to False if the inputs are garbage. Add a test to force breakage to demonstrate the failure:

def test_integer_is_false():
    assert is_empty(1) is False

Run pytest to see the failure, which is expected because the helper is not guarding against it:

============================= test session starts ==============================
platform darwin -- Python 3.6.2, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
collected 1 item

test_is_empty_robust.py::test_integer_is_false FAILED                    [100%]

=================================== FAILURES ===================================
____________________________ test_integer_is_false _____________________________

    def test_integer_is_false():
>       assert is_empty(1) is False

test_is_empty_robust.py:22:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

value = 1

    def is_empty(value):
>       return len(value) == 0
E       TypeError: object of type 'int' has no len()

test_is_empty_robust.py:2: TypeError
======================= 1 failed, 9 deselected in 0.09s ========================

Excellent! This is the first failure reported with pytest. The report has lots of details, and this book goes into those later. First, lets concentrate in the fact that there is a failure and the implementation needs to be updated to get the test passing.

def is_empty(value):
    try:
        return len(value) == 0
    except TypeError:
        return False

With these changes, the utility is now trying to return the value by default, unless a TypeError exception is raised, in which case it falls back to returning False.

To ensure everything works if the function receives an invalid input, execute pytest in the terminal once more, all tests should be passing. another test:

============================= test session starts ==============================
platform darwin -- Python 3.6.2, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
collected 5 items

test_is_empty_robust.py::test_empty_list PASSED                          [ 20%]
test_is_empty_robust.py::test_empty_dict PASSED                          [ 40%]
test_is_empty_robust.py::test_list_is_not_empty PASSED                   [ 60%]
test_is_empty_robust.py::test_dict_is_not_empty PASSED                   [ 80%]
test_is_empty_robust.py::test_integer_is_false PASSED                    [100%]

============================== 5 passed in 0.02s ===============================

This example is enough to demonstrate the fix. The real scenario, however, where an actual project uses the utility, has several other invalid inputs that are tested, like floats and even booleans!

Why is Pytest important? #

Decades ago, in the late 1960s, a new style for the High Jump was introduced: the Fosbury Flop. This new style was revolutionary and very different from other techniques at the time, like the straddle. It quickly became a controversial issue for coaches to prefer one over the other.

My dad tells this story about going through a course in Track and Field, and the professor asked him which style was better: the straddle or the Fosbury flop? His answer was the Fosbury flop. The professor dug further and asked why: “It is easier to teach!" my dad answered. Although the techniques were (seemingly) comparable, there is tremendous value in something easier to teach. This is similar to how I feel about pytest.

By being easy to grasp with simple tests, developers learn faster how to test, and are enabled to test more - a critical pilar for robust software!

Good tooling gets out of the way and enables you to achieve objectives. Great tooling takes it a step further by offering everything you may need when looking to step up. This is very hard to achieve, and I still find it incredible how pytest manages to appear so simple, and yet it has so much to offer.

Even though this book strongly advocates for pytest, the Python ecosystem is better by having choices. Contrary to my own beliefs, when I started getting involved using Python, when comparing something that is not part of the standard library (pytest) vs. something that is (unittest) doesn’t mean one is better than the other. Libraries that aren’t part of Python’s core can move faster, provide many updates, while the core moves at a glacier pace (it has to!).

When the Pytest project started, it was because the Python community saw there was an opportunity to take something like unittest and make it better, with a test runner and a framework. Taking advantage of the many years that have been put into the Pytest project is a fantastic gift.

The power of `assert` #

Have you ever seen a failed assertion in Python? They are not very helpful. Programmers with a background in languages like C or C++ tend to add assertions in lots of places in production code, as it is often done in those languages. Using bare asserts in Python is unusual, and it is usually discouraged. One reason for this is that Python can run in an optimized way which, among other things, removes assertions. Might as well rely on other flow control and error mechanisms!

This is a small function that uses assert for flow control, if the value is acceptable, it returns a string:

def assert_flow_control(value):
    """
    An example on flow control with bare asserts
    """
    assert value
    return "got a value of expected type!"

Start a new Python shell and make it break by passing False as its input argument:

>>> import bare_asserts
>>> bare_asserts.assert_flow_control(False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/alfredo/python/testing-in-python/chapter3/bare_asserts.py", line 6, in assert_flow_control
    assert value
AssertionError

If you didn’t write the function that executed, and this is a production environment, it is impossible what this assertion means. This is awful. Some may argue that it is possible to add a message to the assertion so that when it fails, a better context is given. This is fine, but I prefer software that takes care of repetitive tasks for me. Further, pytest usage of assert goes beyond providing a nice failure message.

When I give an introduction talk on pytest I usually include a slide with all the assertions that unittest provides. In comparison, pytest only has one assertion (assert). This is the complete list from unittest including some that are aliases and others that are being deprecated:

self.assertEqual(a, b)
self.assertNotEqual(a, b)
self.assertTrue(x)
self.assertFalse(x)
self.assertIs(a, b)
self.assertIsNot(a, b)
self.assertIsNone(x)
self.assertIsNotNone(x)
self.assertIn(a, b)
self.assertNotIn(a, b)
self.assertIsInstance(a, b)
self.assertNotIsInstance(a, b)
self.assertRaises(exc, fun, *args, **kwds)
self.assertRaisesRegex(exc, r, fun, *args, **kwds)
self.assertWarns(warn, fun, *args, **kwds)
self.assertWarnsRegex(warn, r, fun, *args, **kwds)
self.assertLogs(logger, level)
self.assertMultiLineEqual(a, b)
self.assertSequenceEqual(a, b)
self.assertListEqual(a, b)
self.assertTupleEqual(a, b)
self.assertSetEqual(a, b)
self.assertDictEqual(a, b)
self.assertAlmostEqual(a, b)
self.assertNotAlmostEqual(a, b)
self.assertGreater(a, b)
self.assertGreaterEqual(a, b)
self.assertLess(a, b)
self.assertLessEqual(a, b)
self.assertRegex(s, r)
self.assertNotRegex(s, r)
self.assertCountEqual(a, b)

From memory, I can usually name four or five. The bare assert that it is frowned upon in most Python projects is one of the core strengths of Pytest: it can compound all those different assertions from unittest into a single call while still producing useful output.

This is a test that compares two long strings, can you catch the differences by seeing them side by side?

def test_long_strings():
    string = ("This is a very, very, long string that has some differences"
            " that are hard to catch")
    expected = ("This is a very, very long string that hes some differences"
            " that are hard to catch")
    assert string == expected

It is pretty difficult to tell right away. In that example, I am cheating by having both the input and the expectation right there in the test. In real test scenarios, the test input will probably come from somewhere else, so all you have is the test output. The test fails when running pytest against that file. This is the output:

E       AssertionError: assert 'This is a ve...hard to catch' == 'This is a ve.
..hard to catch'
E         - This is a very, very, long string that has some differences that ar
e hard to catch
E         ?                     -                   ^
E         + This is a very, very long string that hes some differences that are
 hard to catch
E         ?

The test runner engine produces a diff on the two strings and displays a - character for missing items, and ^ for different ones. The expected value has a missing comma, and it needs to replace hes to has as indicated by the markers in the output. This is a tiny corner of brilliance that demonstrates how powerful it is. The diff engine is not only applicable for strings, but also for other data structures like dictionaries and lists.

Have you ever had to compare giant nested dictionaries? It is horrible to go through each key to find that some key is missing or has a typo. Pytest makes this a trivial task, which doesn’t require anything special (just like comparing long strings). This is the test for comparing the two dictionaries:

def test_nested_dictionaries():
    result = {'first': 12, 'second': 13,
            'others': {'success': True, 'msg': 'A success message!'}}
    expected = {'first': 12, 'second': 13,
            'others': {'success': True, 'msg': 'A sucess message!'}}
    assert result == expected

Run pytest against the example test, and expect failure. In this case, the failure is happening in the nested dictionary. Pytest understands the data structure and extracts the nested dictionary to offer a comparison omitting everything that it knows is identical (no need to include that!):

E   AssertionError: assert {'first': 12,... 'second': 13} ==  \
E                              {'first': 12,... 'second': 13}
E     Omitting 2 identical items, use -vv to show
E     Differing items:
E     {'others': {'msg': 'A success message!', 'success': True}} != \
E         {'others': {'msg': 'A sucess message!', 'success': True}}
E     Use -v to get the full diff

The report is useful, but it isn’t giving you what you need: where exactly is the issue to fix it? This happens because, by default, Pytest tries to be conservative in its output, hoping that the failure is simple enough to catch. In this case, it isn’t, so run again with the -vv flag to get a more thorough overview:

E     Full diff:
E       {
E        'first': 12,
E     -  'others': {'msg': 'A success message!', 'success': True},
E     ?                         -
E     +  'others': {'msg': 'A sucess message!', 'success': True},
E        'second': 13,
E       }

The - character indicates that I introduced a typo in the msg value (the success word). Pytest has support for all native data structures like these examples for dictionaries and strings. Knowing that it can handle them without any customization at all is a great advantage. There is no need to memorize the many different assertions that unittest provides.

The bare assert in Pytest is essential because, like other parts of the Pytest framework, it allows you to concentrate on writing useful tests, not in figuring out a framework. All the examples in this section used functions, further proving its simplicity.

Pytest vs. Unittest #

This chapter has already shown a few different reasons you should concentrate on Pytest, and this section emphasizes the differences. It is important to know why a tool is preferred vs. the alternative.

Tests can be functions: Pytest has support for functions as well as classes, unittest only works with classes
No need to know class inheritance: Pytest doesn’t require to inherit from a class, although it supports unittest.TestCase test classes just fine!
Plugins: There are many plugins by the community for Pytest, and it is not difficult to create your own. It is not possible with unittest.
Extensible reporting: All reporting in Pytest can be modified or extended. It is impossible with unittest
Fully featured test runner: The pytest command-line tool offers a lot right out of the box, no need to add additional code in tests to discover files. With unittest there is some effort with python -m unittest, but options are limited and not extensible.

If you find yourself working in a large project that is using unittest.TestCase tests, then you can use pytest to run them. This is a nice feature that allows you to get a feel for the tool and start tinkering with adding more straightforward tests at the same time. A lot of the time, I get asked if there is a way to “go back” from Pytest-style tests as if the transformation was complicated like switching database schemas… I’ve never seen any need to go back, and to the contrary, I’ve put more work in production code to port towards Pytest! I’ve never seen any project go back.

Lastly, Pytest is not only a test runner. It is a whole framework which also includes a test runner. By being a framework, it allows extending and modifying it so that it fits best whatever type of testing you are doing. I hope that by now, you are fully convinced that this is the right way to go and want to learn more about it!

Chapter 3: Introduction to Pytest #

The most simple test possible #

Why is Pytest important? #

The power of assert #

Pytest vs. Unittest #

The power of `assert` #