TESTS.txt

.. -*- rest -*-

NumPy/SciPy Testing Guidelines
==============================

.. contents::

Introduction
''''''''''''

SciPy uses the `Nose testing system
<http://www.somethingaboutorange.com/mrl/projects/nose>`__, with some
minor convenience features added.  Nose is an extension of the unit
testing framework offered by `unittest.py
<http://docs.python.org/lib/module-unittest.html>`__. Our goal is that
every module and package in SciPy should have a thorough set of unit
tests. These tests should exercise the full functionality of a given
routine as well as its robustness to erroneous or unexpected input
arguments. Long experience has shown that by far the best time to
write the tests is before you write or change the code - this is
`test-driven development
<http://en.wikipedia.org/wiki/Test-driven_development>`__.  The
arguments for this can sound rather abstract, but we can assure you
that you will find that writing the tests first leads to more robust
and better designed code. Well-designed tests with good coverage make
an enormous difference to the ease of refactoring. Whenever a new bug
is found in a routine, you should write a new test for that specific
case and add it to the test suite to prevent that bug from creeping
back in unnoticed.

To run SciPy's full test suite, use the following::

  >>> import scipy
  >>> scipy.test()

SciPy uses the testing framework from NumPy (specifically
``numpy.testing``), so all the SciPy examples shown here are also
applicable to NumPy.  So NumPy's full test suite can be run as
follows::

  >>> import numpy
  >>> numpy.test()


The test method may take two or more arguments; the first is a string
label specifying what should be tested and the second is an integer
giving the level of output verbosity. See the docstring for numpy.test
for details.  The default value for the label is 'fast' - which will
run the standard tests.  The string 'full' will run the full battery
of tests, including those identified as being slow to run. If the
verbosity is 1 or less, the tests will just show information messages
about the tests that are run; but if it is greater than 1, then the
tests will also provide warnings on missing tests. So if you want to
run every test and get messages about which modules don't have tests::

  >>> scipy.test(label='full', verbosity=2) # or scipy.test('full', 2)

Finally, if you are only interested in testing a subset of SciPy, for
example, the ``integrate`` module, use the following::

>>> scipy.integrate.test()

The rest of this page will give you a basic idea of how to add unit
tests to modules in SciPy. It is extremely important for us to have
extensive unit testing since this code is going to be used by
scientists and researchers and is being developed by a large number of
people spread across the world. So, if you are writing a package that
you'd like to become part of SciPy, please write the tests as you
develop the package. Also since much of SciPy is legacy code that was
originally written without unit tests, there are still several modules
that don't have tests yet. Please feel free to choose one of these
modules to develop test for either after or even as you read through
this introduction.

Writing your own tests
''''''''''''''''''''''

Every Python module, extension module, or subpackage in the SciPy
package directory should have a corresponding ``test_<name>.py`` file.
Nose examines these files for test methods (named test*) and test
classes (named Test*).

Suppose you have a SciPy module ``scipy/xxx/yyy.py`` containing a
function ``zzz()``.  To test this function you would create a test
module called ``test_yyy.py``.  If you only need to test one aspect of
``zzz``, you can simply add a test function::

  def test_zzz():
      assert zzz() == 'Hello from zzz'

More often, we need to group a number of tests together, so we create
a test class::

  from numpy.testing import *

  # import xxx symbols
  from scipy.xxx.yyy import zzz

  class TestZzz:
      def test_simple(self):
          assert zzz() == 'Hello from zzz'

      def test_invalid_parameter(self):
          assert_raises(...)

Within these test methods, ``assert()`` is used to test whether a
certain assumption is valid. If the assert fails, the test fails.

Sometimes it is convenient to run ``test_yyy.py`` by itself, so we add

::

  if __name__ == "__main__":
      run_module_suite()

at the bottom.

Labeling tests with nose
------------------------

Unlabeled tests like the ones above are run in the default
``scipy.test()`` run.  If you want to label your test as slow - and
therefore reserved for a full ``scipy.test(label='full')`` run, you
can label it with a nose decorator::

  # numpy.testing module includes 'import decorators as dec'
  from numpy.testing import *

  @dec.slow
  def test_big(self):
      print 'Big, slow test'

Similarly for methods::

  class test_zzz:
      @dec.slow
      def test_simple(self):
          assert zzz() == 'Hello from zzz'

Easier setup and teardown functions / methods
---------------------------------------------

Nose looks for module level setup and teardown functions by name;
thus::

  def setup():
      """Module-level setup"""
      print 'doing setup'

  def teardown():
      """Module-level teardown"""
      print 'doing teardown'


You can add setup and teardown functions to functions and methods with
nose decorators::

  import nose
  from numpy.testing import *
  def setup_func():
      """A trivial setup function."""
      global helpful_variable
      helpful_variable = 'pleasant'
      print "In setup_func"

  def teardown_func():
      """A trivial teardown function."""
      global helpful_variable
      del helpful_variable
      print "In teardown_func"

  @nose.with_setup(setup_func, teardown_func)
  def test_with_extras():
      """This test uses the setup/teardown functions."""
      global helpful_variable
      print "  In test_with_extras"
      print "  Helpful is %s" % helpful_variable

Parametric tests
----------------

One very nice feature of nose is allowing easy testing across a range
of parameters - a nasty problem for standard unit tests.  It does this
with test generators::

  def check_even(n, nn):
      """A check function to be used in a test generator."""
      assert n % 2 == 0 or nn % 2 == 0

  def test_evens():
      for i in range(0,4,2):
          yield check_even, i, i*3

Note that 'check_even' is not itself a test (no 'test' in the name),
but 'test_evens' is a generator that returns a series of tests, using
'check_even', across a range of inputs.

.. warning::

   Parametric tests cannot be implemented on classes derived from
   TestCase.

Doctests
--------

Doctests are a convenient way of documenting the behavior a function
and allowing that behavior to be tested at the same time.  The output
of an interactive Python session can be included in the docstring of a
function, and the test framework can run the example and compare the
actual output to the expected output.

The doctests can be run by adding the ``doctests`` argument to the
``test()`` call; for example, to run all tests (including doctests)
for numpy.lib::

>>> import numpy as np
>>> np.lib.test(doctests=True)

The doctests are run as if they are in a fresh Python instance which
has executed ``import numpy as np`` (tests that are part of the SciPy
package also have an implicit ``import scipy as sp``).


``tests/``
----------

Rather than keeping the code and the tests in the same directory, we
put all the tests for a given subpackage in a ``tests/``
subdirectory. For our example, if it doesn't all ready exist you will
need to create a ``tests/`` directory in ``scipy/xxx/``. So the path
for ``test_yyy.py`` is ``scipy/xxx/tests/test_yyy.py``.

Once the ``scipy/xxx/tests/test_yyy.py`` is written, its possible to
run the tests by going to the ``tests/`` directory and typing::

  python test_yyy.py

Or if you add ``scipy/xxx/tests/`` to the Python path, you could run
the tests interactively in the interpreter like this::

  >>> import test_yyy
  >>> test_yyy.test()

``__init__.py`` and ``setup.py``
--------------------------------

Usually, however, adding the ``tests/`` directory to the python path
isn't desirable. Instead it would better to invoke the test straight
from the module ``xxx``. To this end, simply place the following lines
at the end of your package's ``__init__.py`` file::

  ...
  def test(level=1, verbosity=1):
      from numpy.testing import Tester
      return Tester().test(level, verbosity)

You will also need to add the tests directory in the configuration
section of your setup.py::

  ...
  def configuration(parent_package='', top_path=None):
      ...
      config.add_data_dir('tests')
      return config
  ...

Now you can do the following to test your module::

  >>> import scipy
  >>> scipy.xxx.test()

Also, when invoking the entire SciPy test suite, your tests will be
found and run::

  >>> import scipy
  >>> scipy.test()
  # your tests are included and run automatically!

Tips & Tricks
'''''''''''''

Creating many similar tests
---------------------------

If you have a collection of tests that must be run multiple times with
minor variations, it can be helpful to create a base class containing
all the common tests, and then create a subclass for each variation.
Several examples of this technique exist in NumPy; below are excerpts
from one in `numpy/linalg/tests/test_linalg.py
<http://svn.scipy.org/svn/numpy/trunk/numpy/linalg/tests/test_linalg.py>`__::

  class LinalgTestCase:
      def test_single(self):
          a = array([[1.,2.], [3.,4.]], dtype=single)
          b = array([2., 1.], dtype=single)
          self.do(a, b)

      def test_double(self):
          a = array([[1.,2.], [3.,4.]], dtype=double)
          b = array([2., 1.], dtype=double)
          self.do(a, b)

      ...

  class TestSolve(LinalgTestCase):
      def do(self, a, b):
          x = linalg.solve(a, b)
          assert_almost_equal(b, dot(a, x))
          assert imply(isinstance(b, matrix), isinstance(x, matrix))

  class TestInv(LinalgTestCase):
      def do(self, a, b):
          a_inv = linalg.inv(a)
          assert_almost_equal(dot(a, a_inv), identity(asarray(a).shape[0]))
          assert imply(isinstance(a, matrix), isinstance(a_inv, matrix))

In this case, we wanted to test solving a linear algebra problem using
matrices of several data types, using ``linalg.solve`` and
``linalg.inv``.  The common test cases (for single-precision,
double-precision, etc. matrices) are collected in ``LinalgTestCase``.

Known failures & skipping tests
-------------------------------

Sometimes you might want to skip a test or mark it as a known failure,
such as when the test suite is being written before the code it's
meant to test, or if a test only fails on a particular architecture.
The decorators from numpy.testing.dec can be used to do this.

To skip a test, simply use ``skipif``::

  from numpy.testing import *

  @dec.skipif(SkipMyTest, "Skipping this test because...")
  def test_something(foo):
      ...

The test is marked as skipped if ``SkipMyTest`` evaluates to nonzero,
and the message in verbose test output is the second argument given to
``skipif``.  Similarly, a test can be marked as a known failure by
using ``knownfailureif``::

  from numpy.testing import *

  @dec.knownfailureif(MyTestFails, "This test is known to fail because...")
  def test_something_else(foo):
      ...

Of course, a test can be unconditionally skipped or marked as a known
failure by passing ``True`` as the first argument to ``skipif`` or
``knownfailureif``, respectively.

A total of the number of skipped and known failing tests is displayed
at the end of the test run.  Skipped tests are marked as ``'S'`` in
the test results (or ``'SKIPPED'`` for ``verbosity > 1``), and known
failing tests are marked as ``'K'`` (or ``'KNOWN'`` if ``verbosity >
1``).