Skip to content

Commit

Permalink
DOC: TEST.rst: add example with pytest.mark.parametrize
Browse files Browse the repository at this point in the history
The "Creating many similar tests" section of TEST.rst did not mention the use
of pytest.mark.parametrize, which is a very common and convenient way to
run several similar tests. I also noticed the following about the example:

- All instructions of `test_single` and `test_double` were identical aside
  from the dtype. This did not seem consistent with the intent of the
  example.
- The example showed use of `np.matrix`, which is no longer recommended.
- The example used a function `imply`, which was not defined.

This commit removes the "Creating many similar tests" section and expands the
parametric tests section with an example that uses pytest.mark.parametrize.
It also mentions NumPy's assert_equal, assert_allclose, and assert_less
functions and makes other minor adjustments.

[skip actions] [skip azp] [skip cirrus]
  • Loading branch information
mdhaber committed Sep 7, 2023
1 parent da8cdcf commit 2a64c50
Showing 1 changed file with 39 additions and 53 deletions.
92 changes: 39 additions & 53 deletions doc/TESTS.rst
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,8 @@ Run tests using your favourite IDE such as `vscode`_ or `pycharm`_
Writing your own tests
''''''''''''''''''''''

If you are writing a package that you'd like to become part of NumPy,
please write the tests as you develop the package.
If you are writing code that you'd like to become part of NumPy,
please write the tests as you develop your code.
Every Python module, extension module, or subpackage in the NumPy
package directory should have a corresponding ``test_<name>.py`` file.
Pytest examines these files for test methods (named ``test*``) and test
Expand Down Expand Up @@ -114,27 +114,41 @@ a test class::
with pytest.raises(ValueError, match='.*some matching regex.*'):
...

Within these test methods, ``assert`` and related functions are used to test
whether a certain assumption is valid. If the assertion fails, the test fails.
Within these test methods, the ``assert`` statement or a specialized assertion
function is used to test whether a certain assumption is valid. If the
assertion fails, the test fails. Common assertion functions include:

- :func:`numpy.testing.assert_equal` for testing exact elementwise equality
between a result array and a reference,
- :func:`numpy.testing.assert_allclose` for testing near elementwise equality
between a result array and a reference (i.e. with specified relative and
absolute tolerances), and
- :func:`numpy.testing.assert_array_less` for testing (strict) elementwise
ordering between a result array and a reference.

Note that these assertion functions only compare the numerical vales of the
arrays. Consider adding separate ``assert`` statements regarding the array
dtype and shape (when the reference is a scalar). Note that
``pytest`` internally rewrites the ``assert`` statement to give informative
output when it fails, so should be preferred over the legacy variant
output when it fails, so it should be preferred over the legacy variant
``numpy.testing.assert_``. Whereas plain ``assert`` statements are ignored
when running Python in optimized mode with ``-O``, this is not an issue when
running tests with pytest.

Similarly, the pytest functions :func:`pytest.raises` and :func:`pytest.warns`
should be preferred over their legacy counterparts
:func:`numpy.testing.assert_raises` and :func:`numpy.testing.assert_warns`,
since the pytest variants are more broadly used and allow more explicit
targeting of warnings and errors when used with the ``match`` regex.

which are more broadly used. These versions also accept a ``match``
parameter, which should always be used to precisely target the intended
warning or error.

Note that ``test_`` functions or methods should not have a docstring, because
that makes it hard to identify the test from the output of running the test
suite with ``verbose=2`` (or similar verbosity setting). Use plain comments
(``#``) if necessary.
(``#``) to describe the intent of the test and help the unfamiliar reader to
interpret the code.

Also since much of NumPy is legacy code that was
Also, since much of NumPy is legacy code that was
originally written without unit tests, there are still several modules
that don't have tests yet. Please feel free to choose one of these
modules and develop tests for it.
Expand Down Expand Up @@ -209,9 +223,21 @@ automatically via special arguments. For example, the special argument name
Parametric tests
----------------

One very nice feature of testing is allowing easy testing across a range
of parameters - a nasty problem for standard unit tests. Use the
``pytest.mark.parametrize`` decorator.
One very nice feature of ``pytest`` is the ease of testing across a range
of parameter values using the ``pytest.mark.parametrize`` decorator. For example,
suppose you wish to test ``linalg.solve`` for all combinations of three
array sizes and two data types::

@pytest.mark.parametrize('dimensionality', [3, 10, 25])
@pytest.mark.parametrize('dtype', [np.float32, np.float64])
def test_solve(dimensionality, dtype):
np.random.seed(842523)
A = np.random.random(size=(dimensionality, dimensionality)).astype(dtype)
b = np.random.random(size=dimensionality).astype(dtype)
x = np.linalg.solve(A, b)
eps = np.finfo(dtype).eps
assert_allclose(A @ x, b, rtol=eps*1e2, atol=0)
assert x.dtype == np.dtype(dtype)

Doctests
--------
Expand Down Expand Up @@ -294,46 +320,6 @@ found and run::
Tips & Tricks
'''''''''''''

Creating many similar tests
---------------------------

If you have a collection of tests that must be run multiple times with
minor variations, it can be helpful to create a base class containing
all the common tests, and then create a subclass for each variation.
Several examples of this technique exist in NumPy; below are excerpts
from one in `numpy/linalg/tests/test_linalg.py
<https://github.com/numpy/numpy/blob/main/numpy/linalg/tests/test_linalg.py>`__::

class LinalgTestCase:
def test_single(self):
a = array([[1., 2.], [3., 4.]], dtype=single)
b = array([2., 1.], dtype=single)
self.do(a, b)

def test_double(self):
a = array([[1., 2.], [3., 4.]], dtype=double)
b = array([2., 1.], dtype=double)
self.do(a, b)

...

class TestSolve(LinalgTestCase):
def do(self, a, b):
x = linalg.solve(a, b)
assert_allclose(b, dot(a, x))
assert imply(isinstance(b, matrix), isinstance(x, matrix))

class TestInv(LinalgTestCase):
def do(self, a, b):
a_inv = linalg.inv(a)
assert_allclose(dot(a, a_inv), identity(asarray(a).shape[0]))
assert imply(isinstance(a, matrix), isinstance(a_inv, matrix))

In this case, we wanted to test solving a linear algebra problem using
matrices of several data types, using ``linalg.solve`` and
``linalg.inv``. The common test cases (for single-precision,
double-precision, etc. matrices) are collected in ``LinalgTestCase``.

Known failures & skipping tests
-------------------------------

Expand Down

0 comments on commit 2a64c50

Please sign in to comment.