Skip to content

Commit

Permalink
Merge pull request numpy#9 from hameerabbasi/uarray-me
Browse files Browse the repository at this point in the history
DOC: Adjust NEP-31 to new template.
  • Loading branch information
hameerabbasi authored Oct 28, 2019
2 parents 142f291 + fb3fbc4 commit 35708e8
Showing 1 changed file with 65 additions and 42 deletions.
107 changes: 65 additions & 42 deletions doc/neps/nep-0031-uarray.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,12 @@ This NEP proposes to make all of NumPy's public API overridable via an
extensible backend mechanism.

Acceptance of this NEP means NumPy would provide global and context-local
overrides, as well as a dispatch mechanism similar to NEP-18 [2]_. First
experiences with ``__array_function__`` show that it is necessary to be able
to override NumPy functions that *do not take an array-like argument*, and
hence aren't overridable via ``__array_function__``. The most pressing need is
array creation and coercion functions, such as ``numpy.zeros`` or
``numpy.asarray``; see e.g. NEP-30 [9]_.
overrides in a separate namespace, as well as a dispatch mechanism similar
to NEP-18 [2]_. First experiences with ``__array_function__`` show that it
is necessary to be able to override NumPy functions that *do not take an
array-like argument*, and hence aren't overridable via
``__array_function__``. The most pressing need is array creation and coercion
functions, such as ``numpy.zeros`` or ``numpy.asarray``; see e.g. NEP-30 [9]_.

This NEP proposes to allow, in an opt-in fashion, overriding any part of the
NumPy API. It is intended as a comprehensive resolution to NEP-22 [3]_, and
Expand All @@ -32,19 +32,6 @@ type of function or object that needs to become overridable.
Motivation and Scope
--------------------

The motivation behind ``uarray`` is manyfold: First, there have been several
attempts to allow dispatch of parts of the NumPy API, including (most
prominently), the ``__array_ufunc__`` protocol in NEP-13 [4]_, and the
``__array_function__`` protocol in NEP-18 [2]_, but this has shown the need
for further protocols to be developed, including a protocol for coercion (see
[5]_, [9]_). The reasons these overrides are needed have been extensively
discussed in the references, and this NEP will not attempt to go into the
details of why these are needed; but in short: It is necessary for library
authors to be able to coerce arbitrary objects into arrays of their own types,
such as CuPy needing to coerce to a CuPy array, for example, instead of
a NumPy array. In simpler words, one needs things like ``np.asarray(...)`` or
an alternative to "just work" and return duck-arrays.

The primary end-goal of this NEP is to make the following possible:

.. code:: python
Expand Down Expand Up @@ -94,14 +81,60 @@ vendored into a new namespace within NumPy to give users and downstream
dependencies access to these overrides. This vendoring mechanism is similar
to what SciPy decided to do for making ``scipy.fft`` overridable (see [10]_).

The motivation behind ``uarray`` is manyfold: First, there have been several
attempts to allow dispatch of parts of the NumPy API, including (most
prominently), the ``__array_ufunc__`` protocol in NEP-13 [4]_, and the
``__array_function__`` protocol in NEP-18 [2]_, but this has shown the need
for further protocols to be developed, including a protocol for coercion (see
[5]_, [9]_). The reasons these overrides are needed have been extensively
discussed in the references, and this NEP will not attempt to go into the
details of why these are needed; but in short: It is necessary for library
authors to be able to coerce arbitrary objects into arrays of their own types,
such as CuPy needing to coerce to a CuPy array, for example, instead of
a NumPy array. In simpler words, one needs things like ``np.asarray(...)`` or
an alternative to "just work" and return duck-arrays.

Detailed description
--------------------
Usage and Impact
----------------

Using overrides
~~~~~~~~~~~~~~~
This NEP allows for global and context-local overrides, as well as
automatic overrides a-la ``__array_function__``.

Here are a few examples of how an end-user would use overrides.
Here are some use-cases this NEP would enable, besides the
first one stated in the motivation section:

The first is allowing alternate dtypes to return their
respective arrays.

.. code:: python
# Returns an XND array
x = unp.ones((5, 5), dtype=xnd_dtype) # Or torch dtype
The second is allowing overrides for parts of the API.
This is to allow alternate and/or optimised implementations
for ``np.linalg``, BLAS, and ``np.random``.

.. code:: python
import numpy as np
import pyfftw # Or mkl_fft
# Makes pyfftw the default for FFT
np.set_global_backend(pyfftw)
# Uses pyfftw without monkeypatching
np.fft.fft(numpy_array)
with np.set_backend(pyfftw) # Or mkl_fft, or numpy
# Uses the backend you specified
np.fft.fft(numpy_array)
This will allow an official way for overrides to work with NumPy without
monkeypatching or distributing a modified version of NumPy.

Here are a few other use-cases, implied but not already
stated:

.. code:: python
Expand All @@ -110,11 +143,8 @@ Here are a few examples of how an end-user would use overrides.
result = library_function(data)
result.to_zarr('output.zarr')
This would keep on working, assuming the Dask backend was either set or
registered. Registration can also be done at import-time.

Now consider another function, and what would need to happen in order to
make this work:
This second one would work if ``magic_library`` was built
on top of ``unumpy``.

.. code:: python
Expand All @@ -126,15 +156,13 @@ make this work:
result = pytorch_predict(data)
result.to_zarr('output.zarr')
This would work in two scenarios: The first is that ``pytorch_predict`` was a
multimethod, and implemented by the Dask backend. Dask could provide utility
functions to allow external libraries to register implementations.
Backward compatibility
----------------------

The second, and perhaps more useful way, is that ``pytorch_predict`` was defined
in an idiomatic style true to NumPy in terms of other multimethods, and that Dask
implemented the required multimethods itself, e.g. ``np.convolve``. If this
happened, then the above example would work without either ``magic_library``
or Dask having to do anything specific to the other.
There are no backward incompatible changes proposed in this NEP.

Detailed description
--------------------

Composing backends
~~~~~~~~~~~~~~~~~~
Expand Down Expand Up @@ -562,11 +590,6 @@ manner (as an example)::
def full(shape, fill_value, dtype=None, order='C'):
# Code here

Backward compatibility
----------------------

There are no backward incompatible changes proposed in this NEP.

Alternatives
------------

Expand Down

0 comments on commit 35708e8

Please sign in to comment.