diff --git a/INSTALL.txt b/INSTALL.txt index 9c12ba602a7b..01726df04f6b 100644 --- a/INSTALL.txt +++ b/INSTALL.txt @@ -28,7 +28,7 @@ Building NumPy requires the following software installed: Python must also be compiled with the zlib module enabled. -2) nose__ (optional) 0.10.3 or later +2) nose__ (optional) 1.0 or later This is required for testing numpy, but not for using it. diff --git a/doc/source/dev/development_environment.rst b/doc/source/dev/development_environment.rst new file mode 100644 index 000000000000..6221353cee09 --- /dev/null +++ b/doc/source/dev/development_environment.rst @@ -0,0 +1,211 @@ +.. _development-environment: + +Setting up and using your development environment +================================================= + + +Recommended development setup +----------------------------- + +Since NumPy contains parts written in C and Cython that need to be +compiled before use, make sure you have the necessary compilers and Python +development headers installed - see :ref:`building-from-source`. + +Having compiled code also means that importing Numpy from the development +sources needs some additional steps, which are explained below. For the rest +of this chapter we assume that you have set up your git repo as described in +:ref:`using-git`. + +To build the development version of NumPy and run tests, spawn +interactive shells with the Python import paths properly set up etc., +do one of:: + + $ python runtests.py -v + $ python runtests.py -v -s random + $ python runtests.py -v -t numpy/core/tests/test_iter.py:test_iter_c_order + $ python runtests.py --ipython + $ python runtests.py --python somescript.py + $ python runtests.py --bench + $ python runtests.py -g -m full + +This builds Numpy first, so the first time it may take a few minutes. If +you specify ``-n``, the tests are run against the version of NumPy (if +any) found on current PYTHONPATH. + +Using ``runtests.py`` is the recommended approach to running tests. +There are also a number of alternatives to it, for example in-place +build or installing to a virtualenv. See the FAQ below for details. + + +Building in-place +----------------- + +For development, you can set up an in-place build so that changes made to +``.py`` files have effect without rebuild. First, run:: + + $ python setup.py build_ext -i + +This allows you to import the in-place built NumPy *from the repo base +directory only*. If you want the in-place build to be visible outside that +base dir, you need to point your ``PYTHONPATH`` environment variable to this +directory. Some IDEs (Spyder for example) have utilities to manage +``PYTHONPATH``. On Linux and OSX, you can run the command:: + + $ export PYTHONPATH=$PWD + +and on Windows:: + + $ set PYTHONPATH=/path/to/numpy + +Now editing a Python source file in NumPy allows you to immediately +test and use your changes (in ``.py`` files), by simply restarting the +interpreter. + +Note that another way to do an inplace build visible outside the repo base dir +is with ``python setup.py develop``. This doesn't work for NumPy, because +NumPy builds don't use ``setuptools`` by default. ``python setupegg.py +develop`` will work though. + + +Other build options +------------------- + +It's possible to do a parallel build with ``numpy.distutils`` with the ``-j`` option; +see :ref:`parallel-builds` for more details. + +In order to install the development version of NumPy in ``site-packages``, use +``python setup.py install --user``. + +A similar approach to in-place builds and use of ``PYTHONPATH`` but outside the +source tree is to use:: + + $ python setup.py install --prefix /some/owned/folder + $ export PYTHONPATH=/some/owned/folder/lib/python3.4/site-packages + +Besides ``numpy.distutils``, NumPy supports building with `Bento`_. +This provides (among other things) faster builds and a build log that's much +more readable than the ``distutils`` one. Note that support is still fairly +experimental, partly due to Bento relying on `Waf`_ which tends to have +non-backwards-compatible API changes. Working versions of Bento and Waf are +run on TravisCI, see ``tools/travis-test.sh``. + + +Using virtualenvs +----------------- + +A frequently asked question is "How do I set up a development version of NumPy +in parallel to a released version that I use to do my job/research?". + +One simple way to achieve this is to install the released version in +site-packages, by using a binary installer or pip for example, and set +up the development version in a virtualenv. First install +`virtualenv`_ (optionally use `virtualenvwrapper`_), then create your +virtualenv (named numpy-dev here) with:: + + $ virtualenv numpy-dev + +Now, whenever you want to switch to the virtual environment, you can use the +command ``source numpy-dev/bin/activate``, and ``deactivate`` to exit from the +virtual environment and back to your previous shell. + + +Running tests +------------- + +Besides using ``runtests.py``, there are various ways to run the tests. Inside +the interpreter, tests can be run like this:: + + >>> np.test() + >>> np.test('full') # Also run tests marked as slow + >>> np.test('full', verbose=2) # Additionally print test name/file + +Or a similar way from the command line:: + + $ python -c "import numpy as np; np.test()" + +Tests can also be run with ``nosetests numpy``, however then the NumPy-specific +``nose`` plugin is not found which causes tests marked as ``KnownFailure`` to +be reported as errors. + +Running individual test files can be useful; it's much faster than running the +whole test suite or that of a whole module (example: ``np.random.test()``). +This can be done with:: + + $ python path_to_testfile/test_file.py + +That also takes extra arguments, like ``--pdb`` which drops you into the Python +debugger when a test fails or an exception is raised. + +Running tests with `tox`_ is also supported. For example, to build NumPy and +run the test suite with Python 3.4, use:: + + $ tox -e py34 + +For more extensive info on running and writing tests, see +https://github.com/numpy/numpy/blob/master/doc/TESTS.rst.txt . + + +Rebuilding & cleaning the workspace +----------------------------------- + +Rebuilding NumPy after making changes to compiled code can be done with the +same build command as you used previously - only the changed files will be +re-built. Doing a full build, which sometimes is necessary, requires cleaning +the workspace first. The standard way of doing this is (*note: deletes any +uncommitted files!*):: + + $ git clean -xdf + +When you want to discard all changes and go back to the last commit in the +repo, use one of:: + + $ git checkout . + $ git reset --hard + + +Debugging +--------- + +Another frequently asked question is "How do I debug C code inside Numpy?". +The easiest way to do this is to first write a Python script that invokes the C +code whose execution you want to debug. For instance ``mytest.py``:: + + from numpy import linspace + x = np.arange(5) + np.empty_like(x) + +Now, you can run:: + + $ gdb --args python runtests.py -g --python mytest.py + +And then in the debugger:: + + (gdb) break array_empty_like + (gdb) run + +The execution will now stop at the corresponding C function and you can step +through it as usual. With the Python extensions for gdb installed (often the +default on Linux), a number of useful Python-specific commands are available. +For example to see where in the Python code you are, use ``py-list``. For more +details, see `DebuggingWithGdb`_. + +Instead of plain ``gdb`` you can of course use your favourite +alternative debugger; run it on the python binary with arguments +``runtests.py -g --python mytest.py``. + +Building NumPy with a Python built with debug support (on Linux distributions +typically packaged as ``python-dbg``) is highly recommended. + + + +.. _Bento: http://cournape.github.io/Bento/ + +.. _DebuggingWithGdb: https://wiki.python.org/moin/DebuggingWithGdb + +.. _tox: http://tox.testrun.org + +.. _virtualenv: http://www.virtualenv.org/ + +.. _virtualenvwrapper: http://www.doughellmann.com/projects/virtualenvwrapper/ + +.. _Waf: https://code.google.com/p/waf/ diff --git a/doc/source/dev/gitwash/index.rst b/doc/source/dev/gitwash/index.rst index 9d733dd1c5bd..ae7ce69ded7f 100644 --- a/doc/source/dev/gitwash/index.rst +++ b/doc/source/dev/gitwash/index.rst @@ -1,7 +1,7 @@ .. _using-git: Working with *NumPy* source code -====================================== +================================ Contents: diff --git a/doc/source/dev/index.rst b/doc/source/dev/index.rst index 2229f3ccb384..b0d0ec483046 100644 --- a/doc/source/dev/index.rst +++ b/doc/source/dev/index.rst @@ -6,5 +6,6 @@ Contributing to Numpy :maxdepth: 3 gitwash/index + development_environment For core developers: see :ref:`development-workflow`. diff --git a/doc/source/user/install.rst b/doc/source/user/install.rst index 29aeff6a3248..dcf20498c132 100644 --- a/doc/source/user/install.rst +++ b/doc/source/user/install.rst @@ -12,15 +12,15 @@ Windows ------- Good solutions for Windows are, `Enthought Canopy -`_ (which provides binary -installers for Windows, OS X and Linux) and `Python (x, y) -`_. Both of these packages include Python, NumPy and -many additional packages. +`_, `Anaconda +`_ (which both provide binary installers +for Windows, OS X and Linux) and `Python (x, y) `_. +Both of these packages include Python, NumPy and many additional packages. A lightweight alternative is to download the Python installer from `www.python.org `_ and the NumPy -installer for your Python version from the Sourceforge `download site `_ +installer for your Python version from the Sourceforge `download site +-superpack-win32.exe /arch nosse -or 'sse2' or 'sse3' instead of 'nosse'. +or ``sse2`` or ``sse3`` instead of ``nosse``. Linux ----- -Most of the major distributions provide packages for NumPy, but these can lag -behind the most recent NumPy release. Pre-built binary packages for Ubuntu are -available on the `scipy ppa -`_. Redhat binaries are -available in the `Enthought Canopy -`_. +All major distributions provide packages for NumPy. These are usually +reasonably up-to-date, but sometimes lag behind the most recent NumPy release. Mac OS X -------- -A universal binary installer for NumPy is available from the `download site -`_. The `Enthought Canopy -`_ provides NumPy binaries. +Universal binary installers for NumPy are available from the `download site +`_ +this will give you a binary install (from the wheel packages) compatible with +at python.org Python, Homebrew and MacPorts:: + + pip install numpy + + +.. _building-from-source: Building from source ==================== @@ -82,7 +84,7 @@ Building NumPy requires the following software installed: Note that NumPy is developed mainly using GNU compilers. Compilers from other vendors such as Intel, Absoft, Sun, NAG, Compaq, Vast, Porland, Lahey, HP, IBM, Microsoft are only supported in the form of community - feedback, and may not work out of the box. GCC 3.x (and later) compilers + feedback, and may not work out of the box. GCC 4.x (and later) compilers are recommended. 3) Linear Algebra libraries @@ -93,6 +95,42 @@ Building NumPy requires the following software installed: can be used, including optimized LAPACK libraries such as ATLAS, MKL or the Accelerate/vecLib framework on OS X. +Basic Installation +------------------ + +To install NumPy run:: + + python setup.py install + +To perform an in-place build that can be run from the source folder run:: + + python setup.py build_ext --inplace + +The NumPy build system uses ``distutils`` and ``numpy.distutils``. +``setuptools`` is only used when building via ``pip`` or with ``python +setupegg.py``. Using ``virtualenv`` should work as expected. + +*Note: for build instructions to do development work on NumPy itself, see +:ref:`development-environment`*. + +.. _parallel-builds: + +Parallel builds +~~~~~~~~~~~~~~~ + +From NumPy 1.10.0 on it's also possible to do a parallel build with:: + + python setup.py build -j 4 install --prefix $HOME/.local + +This will compile numpy on 4 CPUs and install it into the specified prefix. +to perform a parallel in-place build, run:: + + python setup.py build_ext --inplace -j 4 + +The number of build jobs can also be specified via the environment variable +``NPY_NUM_BUILD_JOBS``. + + FORTRAN ABI mismatch -------------------- @@ -147,35 +185,10 @@ Additional compiler flags can be supplied by setting the ``OPT``, Building with ATLAS support --------------------------- -Ubuntu 8.10 (Intrepid) and 9.04 (Jaunty) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Ubuntu +~~~~~~ -You can install the necessary packages for optimized ATLAS with this command:: +You can install the necessary package for optimized ATLAS with this command:: sudo apt-get install libatlas-base-dev -If you have a recent CPU with SIMD suppport (SSE, SSE2, etc...), you should -also install the corresponding package for optimal performances. For example, -for SSE2:: - - sudo apt-get install libatlas3gf-sse2 - -This package is not available on amd64 platforms. - -*NOTE*: Ubuntu changed its default fortran compiler from g77 in Hardy to -gfortran in Intrepid. If you are building ATLAS from source and are upgrading -from Hardy to Intrepid or later versions, you should rebuild everything from -scratch, including lapack. - -Ubuntu 8.04 and lower -~~~~~~~~~~~~~~~~~~~~~ - -You can install the necessary packages for optimized ATLAS with this command:: - - sudo apt-get install atlas3-base-dev - -If you have a recent CPU with SIMD suppport (SSE, SSE2, etc...), you should -also install the corresponding package for optimal performances. For example, -for SSE2:: - - sudo apt-get install atlas3-sse2