Merge pull request numpy#25149 from seberg/maxdims

API: bump MAXDIMS/MAXARGS to 64 introduce NPY_AXIS_RAVEL

doc/release/upcoming_changes/25149.c_api.rst

@@ -0,0 +1,10 @@
+Lager ``NPY_MAXDIMS`` and ``NPY_MAXARGS``, ``NPY_RAVEL_AXIS`` introduced
+------------------------------------------------------------------------
+
+``NPY_MAXDIMS`` is now 64, you may want to review its use.  This is usually
+used in a stack allocation, where the increase should be safe.
+However, we do encourage generally to remove any use of ``NPY_MAXDIMS`` and
+``NPY_MAXARGS`` to eventually allow removing the constraint completely.
+For the conversion helper and C-API functions mirrowing Python ones such as
+``tale``, ``NPY_MAXDIMS`` was used to mean ``axis=None`` these must be
+replaced with ``NPY_RAVEL_AXIS``.  See also :ref:`migration_maxdims`.

doc/release/upcoming_changes/25149.c_api_removal.rst

@@ -0,0 +1,2 @@
+* ``NPY_MAX_ELSIZE`` macro has been removed as it only ever reflected
+  builtin numeric types and served no internal purpose.

doc/release/upcoming_changes/25149.change.rst

@@ -0,0 +1,8 @@
+Out-of-bound axis not the same as ``axis=None``
+-----------------------------------------------
+In some cases ``axis=32`` or for concatenate any large value
+was the same as ``axis=None``.
+Except for ``concatenate`` this was deprecate.
+Any out of bound axis value will now error, make sure to use
+``axis=None``.
+

doc/source/numpy_2_0_migration_guide.rst

@@ -41,8 +41,32 @@ Some are defined in ``numpy/_core/include/numpy/npy_2_compat.h``
 (for example ``NPY_DEFAULT_INT``) which can be vendored in full or part
 to have the definitions available when compiling against NumPy 1.x.
 
+If necessary, ``PyArray_RUNTIME_VERSION >= NPY_2_0_API_VERSION`` can be
+used to explicitly implement different behavior on NumPy 1.x and 2.0.
+(The compat header defines it in a way compatible with such use.)
+
 Please let us know if you require additional workarounds here.
 
+.. _migration_maxdims:
+
+Increased maximum number of dimensions
+--------------------------------------
+The maximum number of dimensions (and arguments) was increased to 64, this
+affects the ``NPY_MAXDIMS`` and ``NPY_MAXARGS`` macros.
+It may be good to review their use, and we generally encourage you to
+not use these macros (especially ``NPY_MAXARGS``), so that a future version of
+NumPy can remove this limitation on the number of dimensions.
+
+``NPY_MAXDIMS`` was also used to signal ``axis=None`` in the C-API, including
+the ``PyArray_AxisConverter``.
+The latter will return ``-2147483648`` as an axis (the smallest integer value).
+Other functions may error with
+``AxisError: axis 64 is out of bounds for array of dimension`` in which
+case you need to pass ``NPY_RAVEL_AXIS`` instead of ``NPY_MAXDIMS``.
+``NPY_RAVEL_AXIS`` is defined in the ``npy_2_compat.h`` header and runtime
+dependent (mapping to 32 on NumPy 1.x and ``-2147483648`` on NumPy 2.x).
+
+
 Namespace changes
 =================
 

doc/source/reference/c-api/array.rst

@@ -1949,7 +1949,7 @@ Calculation
 
 .. tip::
 
-    Pass in :c:data:`NPY_MAXDIMS` for axis in order to achieve the same
+    Pass in :c:data:`NPY_RAVEL_AXIS` for axis in order to achieve the same
     effect that is obtained by passing in ``axis=None`` in Python
     (treating the array as a 1-d array).
 
@@ -2970,7 +2970,7 @@ to.
     Convert a Python object, *obj*, representing an axis argument to
     the proper value for passing to the functions that take an integer
     axis. Specifically, if *obj* is None, *axis* is set to
-    :c:data:`NPY_MAXDIMS` which is interpreted correctly by the C-API
+    :c:data:`NPY_RAVEL_AXIS` which is interpreted correctly by the C-API
     functions that take axis arguments.
 
 .. c:function:: int PyArray_BoolConverter(PyObject* obj, npy_bool* value)
@@ -3413,11 +3413,26 @@ Other constants
 
 .. c:macro:: NPY_MAXDIMS
 
-    The maximum number of dimensions allowed in arrays.
+    The maximum number of dimensions that may be used by NumPy.
+    This is set to 64 and was 32 before NumPy 2.
+
+    .. note::
+        We encourage you to avoid ``NPY_MAXDIMS``.  A future version of NumPy
+        may wish to remove any dimension limitation (and thus the constant).
+        The limitation was created so that NumPy can use stack allocations
+        internally for scratch space.
+
+        If your algorithm has a reasonable maximum number of dimension you
+        could check and use that locally.
 
 .. c:macro:: NPY_MAXARGS
 
-    The maximum number of array arguments that can be used in functions.
+    The maximum number of array arguments that can be used in some
+    functions.  This used to be 32 before NumPy 2 and is now 64.
+
+    .. note::
+        You should never use this.  We may remove it in future versions of
+        NumPy.
 
 .. c:macro:: NPY_FALSE
 
@@ -3437,6 +3452,16 @@ Other constants
     The return value of successful converter functions which are called
     using the "O&" syntax in :c:func:`PyArg_ParseTuple`-like functions.
 
+.. c:macro:: NPY_RAVEL_AXIS
+
+    Some NumPy functions (mainly the C-entrypoints for Python functions)
+    have an ``axis`` argument.  This macro may be passed for ``axis=None``.
+
+    .. note::
+        This macro is NumPy version dependent at runtime. The value is now
+        the minimum integer. However, on NumPy 1.x ``NPY_MAXDIMS`` was used
+        (at the time set to 32).
+
 
 Miscellaneous Macros
 ~~~~~~~~~~~~~~~~~~~~

doc/source/reference/c-api/types-and-structures.rst

@@ -119,8 +119,12 @@ PyArray_Type and PyArrayObject
        array. When nd is 0, the array is sometimes called a rank-0
        array. Such arrays have undefined dimensions and strides and
        cannot be accessed. Macro :c:data:`PyArray_NDIM` defined in
-       ``ndarraytypes.h`` points to this data member. :c:data:`NPY_MAXDIMS`
-       is the largest number of dimensions for any array.
+       ``ndarraytypes.h`` points to this data member.
+       ``NPY_MAXDIMS`` is defined as a compile time constant limiting the
+       number of dimensions.  This number is 64 since NumPy 2 and was 32
+       before. However, we may wish to remove this limitations in the future
+       so that it is best to explicitly check dimensionality for code
+       that relies on such an upper bound.
 
    .. c:member:: npy_intp *dimensions
 
@@ -986,11 +990,11 @@ PyArrayIter_Type and PyArrayIterObject
           int   nd_m1;
           npy_intp  index;
           npy_intp  size;
-          npy_intp  coordinates[NPY_MAXDIMS];
-          npy_intp  dims_m1[NPY_MAXDIMS];
-          npy_intp  strides[NPY_MAXDIMS];
-          npy_intp  backstrides[NPY_MAXDIMS];
-          npy_intp  factors[NPY_MAXDIMS];
+          npy_intp  coordinates[NPY_MAXDIMS_LEGACY_ITERS];
+          npy_intp  dims_m1[NPY_MAXDIMS_LEGACY_ITERS];
+          npy_intp  strides[NPY_MAXDIMS_LEGACY_ITERS];
+          npy_intp  backstrides[NPY_MAXDIMS_LEGACY_ITERS];
+          npy_intp  factors[NPY_MAXDIMS_LEGACY_ITERS];
           PyArrayObject *ao;
           char  *dataptr;
           npy_bool  contiguous;
@@ -1086,8 +1090,8 @@ PyArrayMultiIter_Type and PyArrayMultiIterObject
           npy_intp size;
           npy_intp index;
           int nd;
-          npy_intp dimensions[NPY_MAXDIMS];
-          PyArrayIterObject *iters[NPY_MAXDIMS];
+          npy_intp dimensions[NPY_MAXDIMS_LEGACY_ITERS];
+          PyArrayIterObject *iters[NPY_MAXDIMS_LEGACY_ITERS];
       } PyArrayMultiIterObject;
 
    .. c:macro: PyObject_HEAD
@@ -1141,20 +1145,20 @@ PyArrayNeighborhoodIter_Type and PyArrayNeighborhoodIterObject
           PyObject_HEAD
           int nd_m1;
           npy_intp index, size;
-          npy_intp coordinates[NPY_MAXDIMS]
-          npy_intp dims_m1[NPY_MAXDIMS];
-          npy_intp strides[NPY_MAXDIMS];
-          npy_intp backstrides[NPY_MAXDIMS];
-          npy_intp factors[NPY_MAXDIMS];
+          npy_intp coordinates[NPY_MAXDIMS_LEGACY_ITERS]
+          npy_intp dims_m1[NPY_MAXDIMS_LEGACY_ITERS];
+          npy_intp strides[NPY_MAXDIMS_LEGACY_ITERS];
+          npy_intp backstrides[NPY_MAXDIMS_LEGACY_ITERS];
+          npy_intp factors[NPY_MAXDIMS_LEGACY_ITERS];
           PyArrayObject *ao;
           char *dataptr;
           npy_bool contiguous;
-          npy_intp bounds[NPY_MAXDIMS][2];
-          npy_intp limits[NPY_MAXDIMS][2];
-          npy_intp limits_sizes[NPY_MAXDIMS];
+          npy_intp bounds[NPY_MAXDIMS_LEGACY_ITERS][2];
+          npy_intp limits[NPY_MAXDIMS_LEGACY_ITERS][2];
+          npy_intp limits_sizes[NPY_MAXDIMS_LEGACY_ITERS];
           npy_iter_get_dataptr_t translate;
           npy_intp nd;
-          npy_intp dimensions[NPY_MAXDIMS];
+          npy_intp dimensions[NPY_MAXDIMS_LEGACY_ITERS];
           PyArrayIterObject* _internal_iter;
           char* constant;
           int mode;

numpy/__init__.cython-30.pxd

@@ -187,9 +187,8 @@ cdef extern from "numpy/arrayobject.h":
         NPY_ARRAY_UPDATE_ALL
 
     cdef enum:
-        NPY_MAXDIMS
-
-    npy_intp NPY_MAX_ELSIZE
+        NPY_MAXDIMS  # 64 on NumPy 2.x and 32 on NumPy 1.x
+        NPY_RAVEL_AXIS  # Used for functions like PyArray_Mean
 
     ctypedef void (*PyArray_VectorUnaryFunc)(void *, void *, npy_intp, void *,  void *)
 

numpy/__init__.pxd

@@ -182,9 +182,8 @@ cdef extern from "numpy/arrayobject.h":
         NPY_ARRAY_UPDATE_ALL
 
     cdef enum:
-        NPY_MAXDIMS
-
-    npy_intp NPY_MAX_ELSIZE
+        NPY_MAXDIMS  # 64 on NumPy 2.x and 32 on NumPy 1.x
+        NPY_RAVEL_AXIS  # Used for functions like PyArray_Mean
 
     ctypedef void (*PyArray_VectorUnaryFunc)(void *, void *, npy_intp, void *,  void *)
 

numpy/_core/include/numpy/ndarrayobject.h

@@ -103,7 +103,6 @@ extern "C" {
 
 #define PyArray_FILLWBYTE(obj, val) memset(PyArray_DATA(obj), val, \
                                            PyArray_NBYTES(obj))
-#define NPY_MAX_ELSIZE (2 * NPY_SIZEOF_LONGDOUBLE)
 
 #define PyArray_ContiguousFromAny(op, type, min_depth, max_depth) \
         PyArray_FromAny(op, PyArray_DescrFromType(type), min_depth, \

numpy/_core/include/numpy/ndarraytypes.h

@@ -35,10 +35,17 @@
  * The array creation itself could have arbitrary dimensions but all
  * the places where static allocation is used would need to be changed
  * to dynamic (including inside of several structures)
+ *
+ * As of NumPy 2.0, we strongly discourage the downstream use of NPY_MAXDIMS,
+ * but since auditing everything seems a big ask, define it as 64.
+ * A future version could:
+ * - Increase or remove the limit and require recompilation (like 2.0 did)
+ * - Deprecate or remove the macro but keep the limit (at basically any time)
  */
-
-#define NPY_MAXDIMS 32
-#define NPY_MAXARGS 32
+#define NPY_MAXDIMS 64
+/* We cannot change this as it would break ABI: */
+#define NPY_MAXDIMS_LEGACY_ITERS 32
+#define NPY_MAXARGS 64
 
 /* Used for Converter Functions "O&" code in ParseTuple */
 #define NPY_FAIL 0
@@ -1096,18 +1103,18 @@ struct PyArrayIterObject_tag {
         PyObject_HEAD
         int               nd_m1;            /* number of dimensions - 1 */
         npy_intp          index, size;
-        npy_intp          coordinates[NPY_MAXDIMS];/* N-dimensional loop */
-        npy_intp          dims_m1[NPY_MAXDIMS];    /* ao->dimensions - 1 */
-        npy_intp          strides[NPY_MAXDIMS];    /* ao->strides or fake */
-        npy_intp          backstrides[NPY_MAXDIMS];/* how far to jump back */
-        npy_intp          factors[NPY_MAXDIMS];     /* shape factors */
+        npy_intp          coordinates[NPY_MAXDIMS_LEGACY_ITERS];/* N-dimensional loop */
+        npy_intp          dims_m1[NPY_MAXDIMS_LEGACY_ITERS];    /* ao->dimensions - 1 */
+        npy_intp          strides[NPY_MAXDIMS_LEGACY_ITERS];    /* ao->strides or fake */
+        npy_intp          backstrides[NPY_MAXDIMS_LEGACY_ITERS];/* how far to jump back */
+        npy_intp          factors[NPY_MAXDIMS_LEGACY_ITERS];     /* shape factors */
         PyArrayObject     *ao;
         char              *dataptr;        /* pointer to current item*/
         npy_bool          contiguous;
 
-        npy_intp          bounds[NPY_MAXDIMS][2];
-        npy_intp          limits[NPY_MAXDIMS][2];
-        npy_intp          limits_sizes[NPY_MAXDIMS];
+        npy_intp          bounds[NPY_MAXDIMS_LEGACY_ITERS][2];
+        npy_intp          limits[NPY_MAXDIMS_LEGACY_ITERS][2];
+        npy_intp          limits_sizes[NPY_MAXDIMS_LEGACY_ITERS];
         npy_iter_get_dataptr_t translate;
 } ;
 
@@ -1231,7 +1238,7 @@ typedef struct {
         npy_intp             size;                    /* broadcasted size */
         npy_intp             index;                   /* current index */
         int                  nd;                      /* number of dims */
-        npy_intp             dimensions[NPY_MAXDIMS]; /* dimensions */
+        npy_intp             dimensions[NPY_MAXDIMS_LEGACY_ITERS]; /* dimensions */
         PyArrayIterObject    *iters[NPY_MAXARGS];     /* iterators */
 } PyArrayMultiIterObject;
 
@@ -1336,18 +1343,18 @@ typedef struct {
      */
     int               nd_m1;            /* number of dimensions - 1 */
     npy_intp          index, size;
-    npy_intp          coordinates[NPY_MAXDIMS];/* N-dimensional loop */
-    npy_intp          dims_m1[NPY_MAXDIMS];    /* ao->dimensions - 1 */
-    npy_intp          strides[NPY_MAXDIMS];    /* ao->strides or fake */
-    npy_intp          backstrides[NPY_MAXDIMS];/* how far to jump back */
-    npy_intp          factors[NPY_MAXDIMS];     /* shape factors */
+    npy_intp          coordinates[NPY_MAXDIMS_LEGACY_ITERS];/* N-dimensional loop */
+    npy_intp          dims_m1[NPY_MAXDIMS_LEGACY_ITERS];    /* ao->dimensions - 1 */
+    npy_intp          strides[NPY_MAXDIMS_LEGACY_ITERS];    /* ao->strides or fake */
+    npy_intp          backstrides[NPY_MAXDIMS_LEGACY_ITERS];/* how far to jump back */
+    npy_intp          factors[NPY_MAXDIMS_LEGACY_ITERS];     /* shape factors */
     PyArrayObject     *ao;
     char              *dataptr;        /* pointer to current item*/
     npy_bool          contiguous;
 
-    npy_intp          bounds[NPY_MAXDIMS][2];
-    npy_intp          limits[NPY_MAXDIMS][2];
-    npy_intp          limits_sizes[NPY_MAXDIMS];
+    npy_intp          bounds[NPY_MAXDIMS_LEGACY_ITERS][2];
+    npy_intp          limits[NPY_MAXDIMS_LEGACY_ITERS][2];
+    npy_intp          limits_sizes[NPY_MAXDIMS_LEGACY_ITERS];
     npy_iter_get_dataptr_t translate;
 
     /*
@@ -1356,7 +1363,7 @@ typedef struct {
     npy_intp nd;
 
     /* Dimensions is the dimension of the array */
-    npy_intp dimensions[NPY_MAXDIMS];
+    npy_intp dimensions[NPY_MAXDIMS_LEGACY_ITERS];
 
     /*
      * Neighborhood points coordinates are computed relatively to the

numpy/_core/include/numpy/npy_2_compat.h

@@ -30,6 +30,20 @@
 #ifndef NUMPY_CORE_INCLUDE_NUMPY_NPY_2_COMPAT_H_
 #define NUMPY_CORE_INCLUDE_NUMPY_NPY_2_COMPAT_H_
 
+/*
+ * Allow users to use `PyArray_RUNTIME_VERSION` when vendoring the file for
+ * compilation with NumPy 1.x.
+ * Simply do not define when compiling with 2.x.  It must be defined later
+ * as it is set during `import_array()`.
+ */
+#if !defined(PyArray_RUNTIME_VERSION) && NPY_ABI_VERSION < 0x02000000
+  /*
+   * If we are compiling with NumPy 1.x, PyArray_RUNTIME_VERSION so we
+   * pretend the `PyArray_RUNTIME_VERSION` is `NPY_FEATURE_VERSION`.
+   */
+  #define PyArray_RUNTIME_VERSION NPY_FEATURE_VERSION
+#endif
+
 /*
  * New macros for accessing real and complex part of a complex number can be
  * found in "npy_2_complexcompat.h".
@@ -52,4 +66,14 @@
 #endif
 
 
+#if NPY_FEATURE_VERSION >= NPY_2_0_API_VERSION
+    #define NPY_RAVEL_AXIS NPY_MIN_INT
+#elif NPY_ABI_VERSION < 0x02000000
+    #define NPY_RAVEL_AXIS 32
+#else
+    #define NPY_RAVEL_AXIS  \
+        (PyArray_RUNTIME_VERSION >= NPY_2_0_API_VERSION ? -1 : 32)
+#endif
+
+
 #endif  /* NUMPY_CORE_INCLUDE_NUMPY_NPY_2_COMPAT_H_ */