Skip to content

Latest commit

 

History

History
346 lines (284 loc) · 15.4 KB

packaging.md

File metadata and controls

346 lines (284 loc) · 15.4 KB

Packaging Python Code

  • Wheel is a newer format than egg.
  • Distutils is part of standard library, but functionality is very basic.
  • Setuptools recommended over Distutils.

Terminology

The following are from the glossary of the Pythong Packaging User Guide at packaging.python.org and the terms from the pkg-resources page of the setuptools documentation.

  • Project: A library, app, etc. intended to be packaged into a distribution package. Typcially available as a Git repo, other VCS checkout, or (for a single version) a source archive. Contains a project specification file. It will also generally use one of two standard layouts, "flat" or "src," for the files and directories.
  • [Source Archive]: Contains raw source code for a release, from which you create source distributions and built distributions.
  • Specification File: Build information for a project. Usually one of:
    • pyproject.toml: PEP 518, used by many tools including setuptools.
    • setup.py and/or setup.cfg: distutils; legacy support in setuptools.

Distribution formats:

  • Release: A snapshot of a project with a version identifier. May be published as multiple distribution packages (e.g., source, Windows installer).
  • Distribution Package: (Often just "package" or "distribution", but both terms are used in other ways as well.) A versioned archive file built from a project and downloaded by end-users. Note that this is different from an import package, as used by the Python module system.
  • Source Distribution or sdist: [sdist-spec] An unbuilt distribution in a file named {pkgname}-{version}.tar.gz, containing a {pkgname}-{version}/ directory which in turn has a pyproject.toml, a PKG-INFO with metadata, and the source files. See also PEP 643, PEP 625, PEP 721. Pip can build/install these.
  • Built Distribution: Files and metadata that need only be copied to the install location. E.g, wheel, egg.
  • [Binary Distribution]: A built distribution that contains compiled extensions.
  • Importable Distribution: A file or directory that can be placed directly on sys.path.
  • Wheel: Built distribution format that replaces Egg. Has a standard spec later approved as PEP 427. Supported by Pip.
  • [Egg]: Older built distribution format introduced by setuptools. Quick guide. Internal structure.

Version Specifiers

Per the Python Packaging User Guide Version specifiers:

[N!]N(.N)*[{a|b|rc}N][.postN][.devN]

All numbers N are ordered numerically with no regard for leading zeros, i.e., 10 succeeds 9.

All the following are optional except for the release segment:

  • N! Epoch segment. default 0!, used when changing versioning scheme. Rarely used.
  • N(.N)* Release segment. Required. E.g., 2.1, 2.12.3.
  • {a|b|rc}N Pre-release segment. Appended to the next release version number, e.g., 2.3a7 → 2.3b1 → 2.3rc4 → 2.3 for alpha, beta, release candidate and final release. c may be accepted as meaning rc.
  • .postN Post-release segment. Used for small fixes that do not affect the distributed software (e.g., correcting release notes).
  • .devN Development release segment. Appended to the next release version number.

A final release version specifier may consist of only an optional epoch segment followed by a release segment.

Example sorts (left is from [ppug-ver], right is locally made):

1.dev0                              1.0.0.dev4
1.0.dev456                          1.0.0a3
1.0a1                               1.0.0b1.dev1
1.0a2.dev456                        1.0.0b1
1.0a12.dev456                       1.0.0b2
1.0a12                              1.0.0rc1        # or `c1`
1.0b1.dev456                        1.0.0
1.0b2                               1.0.0.post1.dev1
1.0b2.post345.dev456                1.0.0.post1
1.0b2.post345                       1.0.1.dev1
1.0rc1.dev456
1.0rc1
1.0
1.0+abc.5
1.0+abc.7
1.0+5
1.0.post456.dev34
1.0.post456
1.0.15
1.1.dev1

Dependency Specifiers

PEP 508-compliant dependency specifiers have additional syntax to specify version ranges and other depencency options.

Note that ~= 0.x does not follow semantic versioning; it treats 0.2 as compatible with 0.1, exactly as if the major version number were non-zero. Instead use == 0.x.*.

One of these is extras, which adds sets of optional dependencies. These are specified as a comma-separated list of package-specific names in square brackets after the package name, e.g., requests[security,tests].

Package Metadata Fields

Libraries and Packaging Tools

Build frontends:

  • [Pip]: Can build and install from a project or source archive, as well as a release.
  • build: Standard simple build frontend.
  • Hatch: (GitHub) CLI tool to manage dependencies and environment isolation. Includes build backend hatchling.

Build backends (may include a frontend, too):

  • setuptools: Enhanced distutils. Includes easy_install.
  • hatchling: see Hatch above.

Libraries:

  • distlib: Library to aid third-party packaging tools, succeeds packaging.
  • packaging; Packaging library. Used by Pip and setuptools.
  • distutils: Original packaging system. Deprecated 3.10, removed 3.12. Use setuptools instead.
  • importlib.metadata: Python standard library module for getting information about installed packages.

More at packaging.python.org [Key Projects] page.

PyInstaller

PyInstaller (pip install pyinstaller; manual) can package a script and all its dependencies (including the Python interpreter binary!) in a single folder or single executable file that can be run on systems without a Python interpreter installed. However, the build must be done on the target platform; there are no cross-build facilities.

Notes:

  • The app creates a new console window by default; this can be overridden.
  • Only .pyc files are included. Options are available for further obfuscation, or use cython.
  • Arbitrary data can be appended to the end of an ELF or .EXE file; the system loader ignores this. PyInstaller appends a CArchive format archive; pyi-archive_viewer can view the archive.
  • See Run-time Information for use of __file__, sys.executable, sys.argv[0], and notes on finding data files.
  • See Advanced Topics for a description of the the application startup process ("bootloader"), the CArchive format, executable inspection program pyi-bindepend, and using PYTHONHASHSEED to create bit-for-bit reproducable builds.

pyinstaller myscript.py (manpage) will analyze all import statements, but may miss more clever ways of importing code. You can give additional dependencies (files and import paths) on the command line or edit the myscript.spec created by the first PyInstaller run. (These may include data files as well.) A "hook" system is also available to specify "hidden" imports; hooks are included for many popular libraries.

One-file mode changes distribution only; when run it builds a temporary folder and extracts files to it before running as it would in one-folder mode. Implications:

  • Make sure the bundled app works in one-folder mode before building it in one-file mode.
  • The temporary folder will be left behind on program crash.
  • No-exec /tmp will break things; --runtime-tmpdir may help with this.
  • Do not give admin privs to a one-file app; the extraction has race conditions. seteuid() may also be problematic.

Programs that use PyInstaller include docker-compose.

Environment and Dependency Managers

  • Pip w/manual requirements.txt: Dependency versions must be managed manually. Locks or ranges, not both.
  • pip-tools
  • Pipenv Wrapper around pip and virtual environments. Uses Pipfile and Pipfile.lock for dependency specs. Recommended for applications but not libraries due to strict pinning in Pipfile.lock.
  • Poetry has better and more reliable dependency determination than Pipenv. Designed for both apps and libraries. Quite slow.
  • Hatch simplifies/wraps process of creating/managing/testing libs and apps (more features than Poetry in this area). No dependency graph calculation?
  • uv is very new and very fast (it's written in Rust).
    • uv run|lock|sync do cross-platform lock files (> Poetry/PDM/Rye)
    • uv run handles PEP 723 standalone scripts with inline dependency data
    • uv tool is an alternative to pipx
    • uv python is an alternative to pyenv, pythonx, etc.

Pip

The Pip requirements.txt format follows PEP 508/PEP 440. [Version specifiers][ver] are documented above; watch out for the non-semver ~= operator when used with 0.x versions.

pkgname                     # ordinary package names
pkgname == 1.0              # specific version: does not match higher
pkgname == 2.*              # highest 2.x.y version
pkgname ~= 3.4.5            # no less than given, less than 4.0.0
pkgname ~= 0.5.6            # no less than given, less than 0.6.0
pkgname <= 7                # version ranges
pkgname >= 5.1.2

./downloads/foo-1.2.3.whl
pkgname @ git+https://github.com/…/pkgname.git
pkgname @ git+ssh://[email protected]./…/pkgname.git
#   Append `@REF` to use a particular ref, e.g., `@refs/pull/123/head`
requests [security] @ https://github.com/psf/requests/archive/refs/heads/main.zip ; python_version >= "3.11"
#   Note above is a .zip file downloaded from GitHub releases, not the repo.

-r other-requirements.txt
-c constraints.txt

Using the --editable/-e option of Pip will do an editable VCS install. The default clone location is VENV/src/PKGNAME/ (when using a virtualenv) or CWD/src/PKGNAME/ (when not). This can be modified with the --src option.

The optional dependencies brought in with e.g. pip install .[foo,bar] can be specified (among other ways) as optional-dependencies.NAME entries in the [project] section of pyproject.toml files, e.g.:

[project]
optional-dependencies.foo = [ 'abc', 'def', ]

[project.optional-dependencies]
bar = [ 'ghi', 'jkl', ]

pipx

  • pipx -h lists global options and commands; pipx CMD -h gives more detailed help for a particular command.
  • pipx ensurepath [--global]: Updates .bashrc, etc. and $PATH to add directories where pipx stores apps.
  • pipx environment shows where various things are stored.
  • pipx run PKG: Installs PKG to a temporary virtual environment (cached for 14 days) and run the command.

Installation notes:

  • pae -C pipx pipx will not create a working version on Debian if you've not installed the python3-venv package. Installing virtualenv into that pae environment doesn't help. Get around this by using a "full" Python build such as one made by pythonz.

Package Repositories

The PyPI Package Index is the most widely used repository of Python packages, and is the default source for tools like Pip and Poetry.

  • You can file a PyPI issue to handle things like account recovery, name squatting, etc. (PEP 541 "Package Index Name Retention" covers taking over of existing package names on PyPI.)
  • There are "Organizations" accounts available for those handling packages maintained by organisations.

To-read