- Explain how pytest is used to ensure our code works as expected.
- Configure an application to run pytest.
- Describe the structure of a pytest test.
- Execute pytest with flags to ensure its output is what we need.
- Interpret pytest output to find the sources of errors.
- Unit Testing: a development process where the smallest testable parts of an application are independently tested for proper functionality.
- Test-Driven Development: a development process where tests are written to meet expectations for an application before code is written to meet those expectations.
- Assertion: a statement that determines if a wrapped statement produces a falsy value or exception. In either case, the assertion fails and the execution of code stops.
- Flag: a method of providing options to modify commands from the command
line. Flags begin with a dash
-.
pytest is a testing framework in Python that makes it easy to write short, easy-to-parse tests for applications ranging from a single function to huge libraries.
pytest is primarily used for a task called unit testing. Unit testing is a process in which the smallest parts of an application- no matter how large the application- are looked at individually and tested to make sure they operate as intended. Testing is usually carried out by developers themselves, but sometimes by teams of quality assurance (QA) engineers as well.
Unit tests will typically run on many functions at once with many different types of input. The number of failures with their descriptions will be returned to the tester so that they can update any non-functional code. Tests can be run quickly and as many times as desired. They can also be run on code that doesn't exist yet- this allows for a process called test-driven development (TDD), where failures are used to inform what code to write next.
Let's say you want to write a function that interpolates into a string: given a name, it should return "Welcome, name!"
This is fairly simple to write, but even easier to test. Let's start by writing an assertion that will raise an Exception if the return value of our function is incorrect. Open up the Python shell and enter the following:
>>> def interpolate_welcome(name):
... pass
...
>>> assert interpolate_welcome('Guido') == 'Welcome, Guido!'You should see the following output:
# => Traceback (most recent call last):
# => File "<stdin>", line 1, in <module>
# => AssertionErrorThis tells us that in the most recent block of code (the assertion), there was
an error on the first line. As this is an assertion, the error is easy enough
to parse out: interpolate_welcome() doesn't interpolate the name we pass in!
Let's use that assertion to drive the development of some working code:
>>> def interpolate_welcome(name):
... return f'Welcome, {name}!'
...
>>> assert interpolate_welcome('Guido') == 'Welcome, Guido!'You should see no output from this assertion- that means that it worked!
Simple assertions are enough for us to do TDD, but they don't give us many details. We'll see how pytest improves upon this process in just a bit.
While you've worked with several pytest tests up to this point, you may not have dug into the test files themselves. Let's do that now.
The first thing that we need to do to create an application environment that
supports pytest is include either pytest.ini or setup.py. pytest generates
its own paths when it is run, and can often struggle to find the files that you
wish to test. The inclusion of these files allows us to specify where pytest
should start building paths. We give it two options: ., the root directory,
and lib, the application directory.
Once this is set up, we should create a directory to house our tests. This can
be named any valid Python package name (no dashes!), but we recommend making it
clear that it contains test. Ours is contained in lib/ and is simply called
testing/.
We've put a couple of tests inside of testing: test_string.py and
subdirectory/bool_test.py. pytest files must be named either starting with
"test_" or ending with "_test". pytest will look in the current directory and
every subdirectory for any files that match this naming pattern and execute any
tests within.
The test themselves have to be named fairly strictly: test_{name} for
functions and Test{Group} for classes. We'll explain what this means a bit
more later on in Phase 3- just know that pytest classes contain groups of tests
and pytest functions are single tests.
pytest can be executed from the command line using the command pytest. This
will run every test in the current directory and any subdirectories, with paths
to separate files being determined by pytest.ini. So long as you execute
pytest from one of the directories specified there, you shouldn't have any
issues getting your tests to run.
Let's start off by simply running pytest. Enter your virtual environment from
the project root directory with pipenv install; pipenv shell and enter the
command pytest:
$ pytest
====== test session starts ======
platform darwin -- Python 3.8.13, pytest-7.2.1, pluggy-1.0.0
rootdir: python-p3-pytest, configfile: pytest.ini
collected 3 items
string_functions.py contains a function "return_string()" that returns a variable of type str. . [ 33%]
string_functions.py contains a function "interpolate_string()" that takes a string and inserts it into another string. . [ 66%]
string_functions.py contains a function "return_true" that returns True. F [100%]
====== FAILURES ======
______ test_return_true ______
def test_return_true(self):
'''contains a function "return_true" that returns True.'''
> assert return_true() == True
E assert False == True
E + where False = return_true()
lib/testing/subdirectory/bool_test.py:10: AssertionError
====== short test summary info ======
FAILED string_functions.py contains a function "return_true" that returns True. - assert False == True
====== 1 failed, 2 passed in 0.05s ======There's a lot to parse here! Rather than jumping right into this output, let's first figure out how to tailor our output to our needs. We only failed one test, after all- how can we run the one that failed?
pytest provides a number of different options to specify which tests to run. We can run all tests, all tests in a directory, all tests in a file, all tests in a class, and only specified tests.
As we saw before, running all tests just requires us to run pytest from the
project root directory. If all tests are in a subdirectory (such as lib/ or
testing/ in this repo), we can also run pytest from there.
An easier way to run all tests in a directory is to specify the directory after
the pytest command:
$ pytest lib/testing
(python-p3-pytest) python-p3-pytest % pytest lib/testing
====== test session starts ======
...
module in bool_functions, function "return_true" returns True. - assert False == True
====== 1 failed, 2 passed in 0.05s ======We can see above that only one test is failing: test_return_true(). This is
contained in bool_test.py. We can specify to run the tests in this file
with the same syntax as above, noting that bool_test.py is contained in a
subdirectory of testing/: pytest lib/testing/subdirectory/bool_test.py.
$ pytest lib/testing/subdirectory/bool_test.py
====== test session starts ======
...
====== short test summary info ======
FAILED module in bool_functions, function "return_true" returns True. - assert False == True
====== 1 failed in 0.04s ======Getting bored of the same syntax? Good news! Navigating the contents of a file
has nothing to do with Unix directory structure, so we won't be adding any more
forward slashes /.
pytest uses a double colon :: to navigate from the file itself down to classes
and functions. Since we don't have any classes in our tests, we can run
pytest lib/testing/bool_test.py::test_return_true to look into this error
specifically.
$ pytest lib/testing/subdirectory/bool_test.py::test_return_true
====== test session starts ======
...
====== short test summary info ======
FAILED module in bool_functions, function "return_true" returns True. - assert False == True
====== 1 failed in 0.04s ======Now that we can run the tests that we want, let's start parsing useful information out of the results.
Let's go back and look at the full output from the pytest command:
====== test session starts ======
platform darwin -- Python 3.8.13, pytest-7.2.1, pluggy-1.0.0
rootdir: python-p3-pytest, configfile: pytest.ini
collected 3 items
string_functions.py contains a function "return_string()" that returns a variable of type str. . [ 33%]
string_functions.py contains a function "interpolate_string()" that takes a string and inserts it into another string. . [ 66%]
string_functions.py contains a function "return_true" that returns True. F [100%]
====== FAILURES ======
______ test_return_true ______
def test_return_true():
'''in bool_functions, function "return_true" returns True.'''
> assert return_true() == True
E assert False == True
E + where False = return_true()
lib/testing/subdirectory/bool_test.py:7: AssertionError
====== short test summary info ======
FAILED string_functions.py contains a function "return_true" that returns True. - assert False == True
====== 1 failed, 2 passed in 0.05s ======There's a lot to look through here- let's go line by line.
====== test session starts ======
platform darwin -- Python 3.8.13, pytest-7.2.1, pluggy-1.0.0
rootdir: python-p3-pytest, configfile: pytest.ini
collected 3 itemsWe begin with a message that the test session has started, with some equals
signs = building a border. This is to help you find the test results if
you need to scroll back at any point. The next two lines simply describe your
configuration for your system and pytest- the Python version, the pytest
version, the root directory, configuration file, and so on. "collected 3 items"
tells us that pytest has found three tests.
string_functions.py contains a function "return_string()" that returns a variable of type str. . [ 33%]
string_functions.py contains a function "interpolate_string()" that takes a string and inserts it into another string. . [ 66%]
string_functions.py contains a function "return_true" that returns True. F [100%]Next, pytest shows us the progress of testing and whether each test passed or
failed. . after the test description denotes a pass, F denotes a failure.
The percentages in brackets inform us of how far we have progressed in testing-
this can be useful if certain functions in your application take a long time to
run.
====== FAILURES ======
______ test_return_true ______
def test_return_true():
'''in bool_functions, function "return_true" returns True.'''
> assert return_true() == True
E assert False == True
E + where False = return_true()
lib/testing/subdirectory/bool_test.py:7: AssertionErrorAfter looking at an overview of the tests, we dig deeper into the failures.
pytest begins with the name of the test that failed, then shows the code from
which the error arose. Along the left, look for an arrow (greater than) symbol
>. This shows the specific line of code that produced an error when
interpreted.
Underneath the line of code that produced the error, we see lines that begin
with an "E". The first "E" line will show the values of the interpreted line
of code- here, we can see that False is being asserted to equal True.
Naturally, this is cause for an error. The next line will show us where this
erroneous value came from- False is the return value of return_true().
Finally, pytest provides the path to the line of the error from the location
the test was run. lib/testing/subdirectory/bool_test.py on line 7 produced
an AssertionError.
====== short test summary info ======
FAILED string_functions.py contains a function "return_true" that returns True. - assert False == True
====== 1 failed, 2 passed in 0.05s ======At the end of every testing session, pytest provides an overview of the important takeaways for developers and QA engineers. Inside of this summary, the description of the test is included alongside a note that it failed and the interpreted code that caused the failure. We then get the number of failures, number of passed tests, and the amount of time it took to run all tests.
Note: While we won't be looking at the times during our tests, these are going to be important to pay attention to when building larger applications. Slower applications mean fewer users- you don't want that!
When running commands from the command line, you often have the option to
include flags. Flags are a way to modify your commands- they begin with
a dash -, and they're tailored to each command they're used with. For example,
cp (copy) has a -r flag that specifies that the copy operation should be
recursive- you need to use this to copy a directory and its contents. tree
has a -L flag that allows you to specify the depth of subdirectories you
want to include in its output.
pytest has many flags, but we're just going to focus on the few that will be most helpful to you.
-xis pytest's "exit" flag. This executes tests until one fails, then stops executing tests. This is very helpful for test-driven development, as you'll want to focus on developing to one test at a time.--pdbopens the Python debugger when a test fails. It does not open the prettier, improvedipdb, but its basic functions are very similar.-stells pytest to show the full output for failed tests (i.e.print()statements).-q(for quiet) shortens pytest's output. Running with the-qflag will only show a single line for the summary of the testing session and details of the failures.-hwill help you figure out where to place arguments for thepytestcommand and provide a long list of flags and configurations for use with pytest.
This is a test-driven lab. Run pytest -x to execute tests until the first
fails, then use your new knowledge of pytest and assertions to get all tests
passing.
When you're done with this first task, write a test in
testing/test_not_none.py to help get the function in not_none_functions.py
to pass.
Submit your work with git when all tasks are complete.
You’ve seen that pytest is a valuable tool with the Python labs you’ve been working on. Using pytest, we are able to construct tests that help ensure that you understand the concepts and can use them successfully. Well-written tests also help you understand more clearly what the expectations are for what your code will do and how it will function.
But this is not the only reason to learn how to use pytest - it will also be valuable to you in your career as a Python software engineer. Testing packages like pytest are used for test-driven development (TDD), which is considered to be the one of the best processes for writing robust, well-designed, error-free code. Because tests focus on the uses of software rather than the details of its implementation, TDD leads to increased attention to usability and functionality, which in turn results in more stable, usable applications. Finally, having test coverage can make it easy to detect bugs that are introduced during the development process, as well as bugs in existing code.
Think of this lesson as a reference. You should have an understanding of what pytest is and what it does, but you do not need to be a pytest wizard before moving on. As you proceed through the curriculum, if you’re ever struggling with figuring out a lab, this is a great place to return to help you get the most out of pytest.