DOC: write all "customizing build" and "background info" build docs

doc/source/building/blas_lapack.rst

@@ -3,39 +3,85 @@
 BLAS and LAPACK
 ===============
 
+.. _blas-lapack-selection:
+
 Selecting BLAS and LAPACK libraries
 -----------------------------------
 
+BLAS and LAPACK library selection, other than the OpenBLAS default, is
+implemented via Meson `build options
+<https://mesonbuild.com/Build-options.html#build-options>`__. For example, to
+select plain ``libblas`` and ``liblapack`` (this is typically Netlib
+BLAS/LAPACK on Linux distros, and can be dynamically switched between
+implementations on conda-forge), use::
+
+    $ # for a development build
+    $ python dev.py build -C-Dblas=blas -C-Dlapack=lapack
+
+    $ # to build and install a wheel
+    $ python -m build -Csetup-args=-Dblas=blas -Csetup-args=-Dlapack=lapack
+    $ pip install dist/scipy*.whl
+
+    $ # Or, with pip>=23.1, this works too:
+    $ python -m pip -Csetup-args=-Dblas=blas -Csetup-args=-Dlapack=lapack
+
+Other options that should work (as long as they're installed with
+``pkg-config`` or CMake support) include ``mkl``, ``atlas`` and ``blis``.
+
+
 Using pkg-config to detect libraries in a nonstandard location
 --------------------------------------------------------------
 
+The way BLAS and LAPACK detection works under the hood is that Meson tries
+to discover the specified libraries first with ``pkg-config``, and then
+with CMake. If all you have is a standalone shared library file (e.g.,
+``armpl_lp64.so`` in ``/a/random/path/lib/`` and a corresponding header
+file in ``/a/random/path/include/``), then what you have to do is craft
+your own pkg-config file. It should have a matching name (so in this
+example, ``armpl_lp64.pc``) and may be located anywhere. The
+``PKG_CONFIG_PATH`` environment variable should be set to point to the
+location of the ``.pc`` file. The contents of that file should be::
 
-How do I deal with Fortran ABI mismatch?
-----------------------------------------
+    libdir=/path/to/library-dir      # e.g., /a/random/path/lib
+    includedir=/path/to/include-dir  # e.g., /a/random/path/include
+    version=1.2.3                    # set to actual version
+    extralib=-lm -lpthread -lgfortran   # if needed, the flags to link in dependencies
+    Name: armpl_lp64
+    Description: ArmPL - Arm Performance Libraries
+    Version: ${version}
+    Libs: -L${libdir} -larmpl_lp64      # linker flags
+    Libs.private: ${extralib}
+    Cflags: -I${includedir}
 
-Some linear algebra libraries are built with g77 ABI and others with
-GFortran ABI, and these two ABIs are incompatible. Therefore, if you
-build SciPy with ``gfortran`` and link to a linear algebra library, like
-MKL, which is built with g77 ABI, then there'll be an exception or a
-segfault. SciPy fixes this by using the CBLAS API for the few
-functions in the BLAS API that suffers from this issue.
+To check that this works as expected, you should be able to run::
+
+    $ pkg-config --libs armpl_lp64
+    -L/path/to/library-dir -larmpl_lp64
+    $ pkg-config --cflags armpl_lp64
+    -I/path/to/include-dir
 
-Note that SciPy needs to know at build time, what needs to be done and
-the build system will automatically check whether linear algebra
-library is MKL and if so, use the CBLAS API instead of the BLAS API.
-If autodetection fails or if the user wants to override this
-autodetection mechanism, use the following:
 
-Use the ``-Duse-g77-abi=true`` build option. E.g.,::
+Specifying the Fortran ABI to use
+---------------------------------
 
-    $ meson setup build -Duse-g77-abi=true
+Some linear algebra libraries are built with the ``g77`` ABI (also known as
+"the ``f2c`` calling convention") and others with GFortran ABI, and these two
+ABIs are incompatible. Therefore, if you build SciPy with ``gfortran`` and link
+to a linear algebra library like MKL, which is built with a ``g77`` ABI,
+there'll be an exception or a segfault. SciPy fixes this by using ABI wrappers
+which rely on the CBLAS API for the few functions in the BLAS API that suffer
+from this issue.
+
+Note that SciPy needs to know at build time, what needs to be done and
+the build system will automatically check whether linear algebra
+library is MKL or Accelerate (which both always use the ``g77`` ABI) and if so,
+use the CBLAS API instead of the BLAS API. If autodetection fails or if the
+user wants to override this autodetection mechanism for building against plain
+``libblas``/``liblapack`` (this is what conda-forge does for example), use the
+``-Duse-g77-abi=true`` build option. E.g.,::
 
-A more complete example, also configuring the BLAS/LAPACK libraries and picking
-a better Python install behavior (this is what conda-forge could be using for
-example)::
+    $ python -m build -C-Duse-g77-abi=true -Csetup-args=-Dblas=blas -Csetup-args=-Dlapack=lapack 
 
-    $ meson setup builddir -Duse-g77-abi=true -Dblas=blas -Dlapack=lapack -Dpython.install_env=auto
-    $ meson install -C builddir
 
 Work-in-progress
 ----------------

doc/source/building/compilers_and_options.rst

@@ -1,4 +1,101 @@
 Compiler selection and custom build flags
 =========================================
 
-TODO
+Selecting a specific compiler
+-----------------------------
+
+Meson supports the standard environment variables ``CC``, ``CXX`` and ``FC`` to
+select specific C, C++ and/or Fortran compilers. These environment variables are
+documented in `the reference tables in the Meson docs
+<https://mesonbuild.com/Reference-tables.html#compiler-and-linker-flag-environment-variables>`__.
+
+Note that environment variables only get applied from a clean build, because
+they affect the configure stage (i.e., ``meson setup``). An incremental rebuild
+does not react to changes in environment variables - you have to run ``git
+clean -xdf`` and do a full rebuild, or run ``meson setup --reconfigure``.
+
+
+Adding a custom compiler or linker flag
+---------------------------------------
+
+Meson by design prefers builds being configured through command-line options
+passed to ``meson setup``. It provides many built-in options:
+
+- For enabling a debug build and the optimization level, see the next section
+  on "build types",
+- Enabling ``-Werror`` in a portable manner is done via ``-Dwerror=true``,
+- Enabling warning levels is done via ``-Dwarning_level=<int>``,
+- There are many other builtin options, from activating Visual Studio (``-D--vsenv``)
+  and building with link time optimization (``-Db_lto``) to changing the default
+  C++ language level (``-Dcpp_std='c++17'``) or linker flags
+  (``-Dcpp_link_args='-Wl,-z,defs'``).
+
+For a comprehensive overview of options, see `Meson's builtin options docs page
+<https://mesonbuild.com/Builtin-options.html>`__.
+
+Meson also supports the standard environment variables ``CFLAGS``,
+``CXXFLAGS``, ``FFLAGS`` and ``LDFLAGS`` to inject extra flags - with the same
+caveat as in the previous section about those environment variables being
+picked up only for a clean build and not an incremental build.
+
+
+Using different build types with Meson
+--------------------------------------
+
+Meson provides different build types while configuring the project. You can see
+the available options for build types in
+`the "core options" section of the Meson documentation <https://mesonbuild.com/Builtin-options.html#core-options>`__.
+
+Assuming that you are building from scratch (do ``git clean -xdf`` if needed),
+you can configure the build as following to use the ``debug`` build type::
+
+    meson setup build --buildtype debug  --prefix=$PWD/build-install
+
+Now, you can use the ``dev.py`` interface for further building, installing and
+testing SciPy::
+
+    python dev.py -s linalg
+
+This will work because after initial configuration, Meson will remember the
+config options.
+
+
+Use GCC and Clang builds in parallel
+------------------------------------
+
+It may be useful to have several builds of SciPy in the same repo, for example
+to compare the differences between two compilers for diagnosing an issue. As
+discussed, Meson is fully out-of-place, so different builds will not interfere
+with each other. We assume in the rest of this section that GCC is the default.
+For example, let us build using GCC and Clang.
+
+1. Build with GCC::
+
+    python dev.py build
+
+Using the above command, meson will build with the (default) GCC compilers in
+the ``build`` directory, and install to the ``build-install`` directory.
+
+2. Build with Clang::
+
+    CC=clang CXX=clang++ FC=gfortran python dev.py --build-dir=build-clang build
+
+Using the above commands, Meson will build with the Clang, Clang++ and Gfortran
+compilers in the ``build-clang`` directory, and then install SciPy into
+``build-clang-install``.
+
+Meson will remember the compiler selection for the ``build-clang`` directory and
+it cannot be changed, so each future invocation of
+``python dev.py --build-dir=build-clang <command>`` it will automatically use Clang.
+
+Tip: use an alias to make this easier to use, e.g.,
+``alias dev-clang="python dev.py --build-dir=build-clang"`` and then
+``dev-clang build``.
+
+A common reason to have two builds is to compare between them. For example,
+to run the ``scipy.linalg`` tests for builds with both compilers, do::
+
+    python dev.py -s linalg                          # run tests for the GCC build
+    python dev.py --build-dir build-clang -s linalg  # run tests for the Clang build
+
+

doc/source/building/cross_compilation.rst

@@ -1,4 +1,34 @@
 Cross compilation
 =================
 
-TODO
+Cross compilation is a complex topic, we only add some hopefully helpful hints
+here (for now). As of May 2023, cross-compilation based on ``crossenv`` is
+known to work, as used (for example) in conda-forge. Cross-compilation without
+``crossenv`` requires some manual overrides, as done in (for example) Void Linux.
+
+Please see `Meson's documentation on cross compilation
+<https://mesonbuild.com/Cross-compilation.html>`__
+for details on Meson's support for cross-compilation.
+
+One common hiccup is that ``numpy`` and ``pythran`` require
+running Python code in order to obtain their include directories. This tends to
+not work well, either accidentally picking up the packages from the build
+(native) Python rather than the host (cross) Python or requiring ``crossenv``
+or QEMU to run the host Python. To avoid this problem, specify the paths to the
+relevant directories in your *cross file*:
+
+.. code:: ini
+
+    [constants]
+    sitepkg = '/abspath/to/host-pythons/site-packages/'
+
+    [properties]
+    numpy-include-dir = sitepkg + 'numpy/core/include'
+    pythran-include-dir = sitepkg + 'pythran'
+
+For more details and the current status around cross compilation, see:
+
+- Tracking issue for SciPy cross-compilation needs and issues:
+  `scipy#14812 <https://github.com/scipy/scipy/issues/14812>`__
+- The state of cross compilation in Python:
+  `pypackaging-native key issue page <https://pypackaging-native.github.io/key-issues/cross_compilation/>`__

doc/source/building/distutils_equivalents.rst

@@ -0,0 +1,27 @@
+Meson and ``distutils`` ways of doing things
+--------------------------------------------
+
+*Old workflows (numpy.distutils based):*
+
+1. ``python runtests.py``
+2. ``python setup.py build_ext -i`` + ``export
+   PYTHONPATH=/home/username/path/to/scipy/reporoot`` (and then edit pure
+   Python code in SciPy and run it with ``python some_script.py``).
+3. ``python setup.py develop`` - this is similar to (2), except in-place build
+   is made permanently visible in env.
+4. ``python setup.py bdist_wheel`` + ``pip install dist/scipy*.whl`` - build
+   wheel in current env (i.e. uses installed numpy, etc.) and install it.
+5. ``pip install .`` - build wheel in an isolated build env against deps in
+   ``pyproject.toml`` and install it. *Note: be careful, this is usually not
+   the correct command for development installs - typically you want to use (4)
+   or* ``pip install . -v --no-build-isolation``.
+
+*New workflows (Meson and meson-python based):*
+
+1. ``python dev.py``
+2. ``pip install -e . --no-build-isolation`` (see the ``meson-python`` docs)
+3. the same as (2)
+4. ``python -m build --no-isolation`` + ``pip install dist/scipy*.whl`` - see
+   `pypa/build <https://pypa-build.readthedocs.io/en/latest/>`_.
+5. ``pip install .``
+

doc/source/building/index.rst

@@ -346,6 +346,7 @@ Background information
    :maxdepth: 1
 
    understanding_meson
+   introspecting_a_build
    distutils_equivalents