Skip to content

Commit

Permalink
DOC: update wheel building procedure for release
Browse files Browse the repository at this point in the history 
Document the single repository wheel building system.  Strip out the old
Wine instructions.
  • Loading branch information
@matthew-brett
matthew-brett committed Jun 29, 2016
1 parent 36508b2 commit 449899b
Showing 1 changed file with 83 additions and 169 deletions.
252 changes: 83 additions & 169 deletions doc/HOWTO_RELEASE.rst.txt
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,5 @@
This file gives an overview of what is necessary to build binary releases for
NumPy. Windows binaries are built here using Wine, they can of course also be
built on Windows itself. Building OS X binaries on another platform is not
possible, but our current OSX binary build procedure uses travis-ci virtual
machines running OSX.
NumPy.

Current build and release info
==============================
Expand Down Expand Up @@ -39,15 +36,15 @@ Release Scripts
Supported platforms and versions
================================

Python 2.6-2.7 and >=3.2 are the currently supported versions when building
from source. We test numpy against all these versions every time we merge
code to trunk. Binary installers may be available for a subset of these
versions (see below).
Python 2.7 and >=3.4 are the currently supported versions when building from
source. We test numpy against all these versions every time we merge code to
trunk. Binary installers may be available for a subset of these versions (see
below).

OS X
----

Python 2.7 and >=3.3 are the versions for which we provide binary installers.
Python 2.7 and >=3.4 are the versions for which we provide binary installers.
OS X versions >= 10.6 are supported. We build binary wheels for OSX that are
compatible with Python.org Python, system Python, homebrew and macports - see
this `OSX wheel building summary
Expand All @@ -56,47 +53,49 @@ this `OSX wheel building summary
Windows
-------

32-bit Python 2.7, 3.3, 3.4 are the versions for which we provide binary
installers. Windows XP, Vista and 7 are supported. Our current windows mingw
toolchain is not able to build 64-bit binaries of numpy. We are hoping to
update to a `mingw-w64 toolchain
<https://github.com/numpy/numpy/wiki/Mingw-w64-faq>`_ soon.
32-bit Python 2.7, 3.4, 3.5 are the versions for which we provide binary
installers. Windows XP, Vista, 7, 8 and 10 are supported. We build numpy
using the MSVC compilers on Appveyor, but we are hoping to update to a
`mingw-w64 toolchain <https://github.com/numpy/numpy/wiki/Mingw-w64-faq>`_
soon.

Linux
-----

Many distributions include NumPy. Building from source is also relatively
straightforward. Only tarballs are created for Linux, no specific binary
installers are provided (yet).
We build and ship `manylinux1 <https://www.python.org/dev/peps/pep-0513>`_
wheels for numpy. Many Linux distributions include their own binary builds
of NumPy.

BSD / Solaris
-------------
No binaries are provided, but succesful builds on Solaris and BSD have been

No binaries are provided, but successful builds on Solaris and BSD have been
reported.

Tool chain
==========

We build all our wheels on cloud infrastructure - so this list of compilers is
for information and debugging builds locally. See the ``.travis.yml`` and
``appveyor.yml`` scripts in the `numpy wheels`_ repo for the definitive source
of the build recipes.

Compilers
---------

The same gcc version is used as the one with which Python itself is built on
each platform. At the moment this means:

* OS X builds on travis currently use `clang`. It appears that binary wheels
for OSX >= 10.6 can be safely built from from OSX 10.9 when building against
the Python from the Python.org installers.
* Windows builds use MinGW 3.4.5. Updating this to a more recent MinGW with
GCC 4.x is desired, but there are still practical difficulties in building
the binary installers.
for OSX >= 10.6 can be safely built from from the travis-ci OSX 10.9 VMs
when building against the Python from the Python.org installers;
* Windows builds use the MSVC version corresponding to the Python being built
against;
* Manylinux1 wheels use the gcc provided on the Manylinux docker images.

You will need Cython for building the binaries. Cython compiles the ``.pyx``
files in the numpy distribution to ``.c`` files.

Fortran: on OS X gfortran from `this site <http://r.research.att.com/tools/>`_
is used. On Windows g77 (included in MinGW) is the current default, in the future
this may shift to gfortran as well.

Python
------
* Python(s) from `python.org <http://python.org>`_
Expand All @@ -110,54 +109,6 @@ Building docs
* Matplotlib
* Texlive (or MikTeX on Windows)

Wine
----
For building Windows binaries on OS X Wine can be used. In Wine the following
needs to be installed:

* Python 2.6-2.7 and 3.3
* MakeNsis
* CpuId plugin for MakeNsis : this can be found in the NumPy source tree under
tools/win32build/cpucaps and has to be built with MinGW (see SConstruct file in
that dir for details)
* MinGW
* ATLAS, 3x ([No SSE, SSE2, SSE3] for superpack installer) : ATLAS does not
compile under wine or on Windows out of the box. Binaries for ATLAS can be
found in the vendor repository on GitHub (http://github.com/numpy/vendor).

To install Wine on OS X Snow Leopard the current options are to compile a
current unstable version ,`<http://wiki.winehq.org/MacOSX/Building>`_, or to use
an install script from `here <http://code.google.com/p/osxwinebuilder/>`_. For
me, the former option did not work (everything compiled, but after installing
Python the command ``import tempfile`` resulted in an exception. The latter
option did work.

After successful installation and an invocation of the wine executable, a
~/.wine folder exists - new programs will be installed there in
~/.wine/drive_c. Installing Windows programs with .exe executables is done by
running

$ wine yourprog.exe

and MSI installers can be installed with

$ msiexec /i yourprog.msi

For the above to work you probably need to put the wine-1.x.x/bin directory in
your PATH.

To install MinGW, the easiest option is to use the automated installer on the
MinGW download page. This will give you (at this moment) GCC 3.4.5; GCC 4.x is
still not supported officially by MinGW.

To be able to use gcc and MakeNsis in Wine, the locations of gcc.exe and
makensis.exe should be added to the Windows environment variable PATH. This can
easily be done by running

$ wine regedit

add adding a PATH variable in HKEY_CURRENT_USER/Environment.

Virtualenv
----------
Virtualenv is a very useful tool to keep several versions of packages around.
Expand All @@ -167,18 +118,17 @@ It is also used in the Paver script to build the docs.
What is released
================

Binaries
--------

Windows binary installers in "superpack" form for Python 2.7/3.3/3.4. A
superpack contains three builds, for SSE2, SSE3 and no SSE.

Wheels
------

OSX wheels built via travis-ci : see - see `building OSX wheels`_.
* Windows wheels for Python 2.7, 3.4, 3.5, for 32- and 64-bit, built using
Appveyor;
* Dual architecture OSX wheels built via travis-ci;
* 32- and 64-bit Manylinux1 wheels built via travis-ci.

See the `numpy wheels`_ building repository for more detail.

.. _build OSX wheels: https://github.com/MacPython/numpy-wheels
.. _numpy wheels : https://github.com/MacPython/numpy-wheels

Other
-----
Expand Down Expand Up @@ -366,78 +316,36 @@ Increment the release number in setup.py. Release candidates should have "rc1"
Also create a new version hash in cversions.txt and a corresponding version
define NPY_x_y_API_VERSION in numpyconfig.h

Trigger the OSX builds on travis
--------------------------------
Trigger the wheel builds on travis-ci and Appveyor
--------------------------------------------------

See the `numpy wheels` repository.

See `build OSX wheels`_.
In that repository edit the files:

You may need to check the ``.travis.yml`` file of the
https://github.com/MacPython/numpy-wheels repository.
* ``.travis.yml``;
* ``appveyor.yml``.

Make sure that the releast tag has been pushed, and that the ``.travis.yml``
is set thusly::
In both cases, set the ``BUILD_COMMIT`` variable to the current release tag -
e.g. ``v1.11.1``.

- NP_COMMIT=latest-tag # comment out to build version in submodule
Make sure that the release tag has been pushed.

Trigger a build by doing an empty (or otherwise) commit to the repository::
Trigger a build by doing a commit of your edits to ``.travis.yml`` and
``appveyor.yml`` to the repository::

cd /path/to/numpy-wheels
git commit --allow-empty
# Edit .travis.yml, appveyor.yml
git commit
git push

The wheels, once built, appear in http://wheels.scipy.org

Trigger Windows builds on Appveyor
----------------------------------

See: `build Windows wheels`_

* Clone / update the https://github.com/numpy/windows-wheel-builder repository;
* Check the ``appveyor.yml`` file in that repository;
* Edit the line starting ``NP_VERSION:`` to give the numpy tag that you want
to build;
* Push up to github to trigger a build.

The wheels appear in a Rackspace CDN container at:

* http://58688808cd85529d4031-38dee5dca2544308e91131f21428d924.r12.cf2.rackcdn.com
* https://84c1a9a06db6836f5a98-38dee5dca2544308e91131f21428d924.ssl.cf2.rackcdn.com

The contents via the HTTPS URL seems to get updated more slowly than via the
HTTP URL, so if you need the binaries quickly, prefer the HTTP URL.

.. _build Windows wheels: https://github.com/numpy/windows-wheel-builder
The wheels, once built, appear at a Rackspace container pointed at by:

Trigger Manylinux builds on travis-ci
-------------------------------------
* http://wheels.scipy.org
* https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com

.. note::

Until we move the manylinux build scripts, you'll need to ask
``@matthew-brett`` to make you a collaborator on the manylinux repos.

* Clone / update the repository at
https://github.com/matthew-brett/manylinux-builds
* Edit the line in ``.travis.yml`` starting ``NUMPY_VERSIONS=`` to set the
numpy tag to build;
* Push your edits to ``.travis.yml`` up to github to trigger a mass manylinux
build;
* Clone / update the repository at
https://github.com/matthew-brett/manylinux-testing;
* Push an empty commit to the ``manylinux-testing`` repo to trigger a test run
of the newly-built numpy wheels with a range of dependent libraries, as well
as numpy's own unit tests. The tests will take several hours.

The built wheels will be available from a Rackspace CDN container at:

* http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
* https://d9a97980b71d47cde94d-aae005c4999d7244ac63632f8b80e089.ssl.cf2.rackcdn.com

As for the other Rackspace containers, the HTTP address may update first, and
you should wait 15 minutes after the build finishes before fetching the
binaries. For the manylinux wheels, the time to run ``manylinux-testing`` is
much greater than 15 minutes, so waiting for the tests to pass will be enough
time for the binaries to refresh on Rackspace.
The HTTP address may update first, and you should wait 15 minutes after the
build finishes before fetching the binaries.

Make the release
----------------
Expand Down Expand Up @@ -471,30 +379,39 @@ downloading all the Windows, Manylinux, OSX wheels and uploading to PyPI.
::

cd ~/wheelhouse # local directory to cache wheel downloads
MANYLINUX_URL=http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
WINDOWS_URL=http://58688808cd85529d4031-38dee5dca2544308e91131f21428d924.r12.cf2.rackcdn.com
OSX_URL=http://wheels.scipy.org
wheel-uploader -u $MANYLINUX_URL -v -s -t manylinux1 numpy 1.11.1rc1
wheel-uploader -u $WINDOWS_URL -v -s -t win numpy 1.11.1rc1
wheel-uploader -u $OSX_URL -v -s -t macosx numpy 1.11.1rc1
CDN_URL=https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com
wheel-uploader -u $CDN_URL -w warehouse -v -s -t win numpy 1.11.1rc1
wheel-uploader -u $CDN_URL -w warehouse -v -s -t macosx numpy 1.11.1rc1
wheel-uploader -u $CDN_URL -w warehouse -v -s -t manylinux1 numpy 1.11.1rc1

The ``-v`` flag gives verbose feedback, ``-s`` causes the script to sign the
wheels with your GPG key before upload.

You may well find that these uploads break at some point, with error messages
from the PyPI server. In this case you'll have to continue the uploads by
hand using `twine <https://pypi.python.org/pypi/twine>`_, using something
like::

twine upload -s numpy-1.11.1rc1-cp34-*.whl

Do this for the wheel files that ``wheel-uploader`` downloaded, but for which
the upload failed.

The ``warehouse`` PyPI server seems to be more reliable in receiving automated
wheel uploads. You can set the repository to upload to with the ``-r`` flag
to ``wheel-uploader`` and ``twine``. The warehouse repository URL for your
``~/.pypirc`` file is https://upload.pypi.io/legacy/
wheels with your GPG key before upload. ``-r warehouse`` causes the upload to
use the Warehouse PyPI server. This is a good idea because the Warehouse
server seems to be a lot more reliable in receiving automated wheel uploads.
For this flag to work, you will need a ``warehouse`` section in your
``~/.pypirc`` file, of form:

[distutils]
index-servers =
pypi
warehouse

[pypi]
username:your_user_name
password:your_password

[warehouse]
repository: https://upload.pypi.io/legacy/
username: your_user_name
password: your_password

[server-login]
username:your_user_name
password:your_password

Don't forget to upload the wheels before the source tarball, so there is no
period for which people switch from an expected binary install to a source
install from PyPI.

There are two ways to update the source release on PyPI, the first one is::

Expand All @@ -509,9 +426,6 @@ The second way is to upload the PKG_INFO file inside the sdist dir in the
web interface of PyPI. The source tarball can also be uploaded through this
interface.

To push the travis-ci OSX wheels up to PyPI see :
https://github.com/MacPython/numpy-wheels#uploading-the-built-wheels-to-pypi

.. _push-tag-and-commit:

Push the release tag and commit
Expand Down

0 comments on commit 449899b

Please sign in to comment.