Skip to content

Commit

Permalink
Backed out 5 changesets (bug 1766497) for causing build bustages with…
Browse files Browse the repository at this point in the history
… "No module named 'encodings'" CLOSED TREE

Backed out changeset 09f7f3d1c6dd (bug 1766497)
Backed out changeset 6bb5f754957a (bug 1766497)
Backed out changeset aedf2d7af408 (bug 1766497)
Backed out changeset 394555265c86 (bug 1766497)
Backed out changeset 41be40696e00 (bug 1766497)
  • Loading branch information
Cristian Tuns committed Oct 13, 2022
1 parent 00d5a50 commit c8b4aae
Show file tree
Hide file tree
Showing 328 changed files with 46,495 additions and 256 deletions.
63 changes: 38 additions & 25 deletions build/docs/python.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,26 @@ related to the build system, Python is typically the first choice.
Python Requirements
===================

The tree requires Python 3.6 or greater to build.
The tree requires Python 2.7.3 or greater but not Python 3 to build.
All Python packages not in the Python distribution are included in the
source tree. So all you should need is a vanilla Python install and you
should be good to go.

Only CPython (the Python distribution available from www.python.org) is
supported.

We require Python 2.7.3 (and not say 2.7.2) to build because Python
2.7.3 contains numerous bug fixes, especially around the area of Unicode
handling. These bug fixes are extremely annoying and have to be worked
around. The build maintainers were tired of doing this, so the minimum
version requirement was upped (bug 870420).

We intend to eventually support Python 3. This will come by way of dual
2.7/3.x compatibility because a single flag day conversion to 3.x will
be too cumbersome given the amount of Python that would need converted.
We will not know which 3.x minor release we are targeting until this
effort is underway. This is tracked in bug 636155.

Compiled Python Packages
========================

Expand All @@ -35,7 +47,8 @@ Issues with OS X System Python
==============================

The Python that ships with OS X has historically been littered with
subtle bugs and suboptimalities.
subtle bugs and suboptimalities. Furthermore, OS X up through 10.8 don't
ship with Python 2.7.3 (10.8 ships with 2.7.2).

OS X 10.8 and below users will be required to install a new Python
distribution. This may not be necessary for OS X 10.9+. However, we
Expand All @@ -45,48 +58,48 @@ OS X's system Python issues.
We recommend installing Python through Homebrew or MacPorts. If you run
``mach bootstrap``, this should be done for you.

Virtual Environments
====================
Virtualenvs
===========

The build system relies heavily on
`venv <https://docs.python.org/3/library/venv.html>`_. Venv provides
standalone and isolated Python "virtual environments". The problem a venv
`virtualenvs <http://www.virtualenv.org/en/latest/>`_. Virtualenvs are
standalone and isolated Python environments. The problem a virtualenv
solves is that of dependencies across multiple Python components. If two
components on a system relied on different versions of a package, there
could be a conflict. Instead of managing multiple versions of a package
simultaneously, Python and venv take the route that it is easier
simultaneously, Python and virtualenvs take the route that it is easier
to just keep them separate so there is no potential for conflicts.

Very early in the build process, a venv is created inside the
:term:`object directory`. The venv is configured such that it can
Very early in the build process, a virtualenv is created inside the
:term:`object directory`. The virtualenv is configured such that it can
find all the Python packages in the source tree. The code for this lives
in ``mach.site``.
in :py:mod:`mach.site`.

Deficiencies
------------

There are numerous deficiencies with the way virtual environments are
handled in the build system.
There are numerous deficiencies with the way virtualenvs are handled in
the build system.

* mach reinvents the venv.
* mach reinvents the virtualenv.

There is code in ``build/mach_initialize.py`` that configures ``sys.path``
much the same way the venv does. There are various bugs tracking
much the same way the virtualenv does. There are various bugs tracking
this. However, no clear solution has yet been devised. It's not a huge
problem and thus not a huge priority.

* They aren't preserved across copies and packaging.

If you attempt to copy an entire tree from one machine to another or
from one directory to another, chances are the venv will fall
from one directory to another, chances are the virtualenv will fall
apart. It would be nice if we could preserve it somehow. Instead of
actually solving portable venv, all we really need to solve is
encapsulating the logic for populating the venv along with all
actually solving portable virtualenvs, all we really need to solve is
encapsulating the logic for populating the virtualenv along with all
dependent files in the appropriate place.

* .pyc files written to source directory.

We rely heavily on ``.pth`` files in our venv. A ``.pth`` file
We rely heavily on ``.pth`` files in our virtualenv. A ``.pth`` file
is a special file that contains a list of paths. Python will take the
set of listed paths encountered in ``.pth`` files and add them to
``sys.path``.
Expand Down Expand Up @@ -124,11 +137,11 @@ should be viewed as the least desirable option.
Common Issues with Python
=========================

Upgrading your Python distribution breaks the venv
Upgrading your Python distribution breaks the virtualenv
--------------------------------------------------------

If you upgrade the Python distribution (e.g. install Python 3.6.15
from 3.6.9), chances are parts of the venv will break.
If you upgrade the Python distribution (e.g. install Python 2.7.5
from 2.7.3, chances are parts of the virtualenv will break.
This commonly manifests as a cryptic ``Cannot import XXX`` exception.
More often than not, the module being imported contains binary/compiled
components.
Expand All @@ -146,15 +159,15 @@ It is common for people to install Python packages using ``sudo`` (e.g.
A problem with this is that packages installed at the system level may
conflict with the package provided by the source tree. As of bug 907902
and changeset f18eae7c3b27 (September 16, 2013), this should no longer
be an issue since the venv created as part of the build doesn't
be an issue since the virtualenv created as part of the build doesn't
add the system's ``site-packages`` directory to ``sys.path``. However,
poorly installed packages may still find a way to creep into the mix and
interfere with our venv.
interfere with our virtualenv.

As a general principle, we recommend against using your system's package
manager or using ``sudo`` to install Python packages. Instead, create
virtual environments and isolated Python environments for all of your
Python projects.
virtualenvs and isolated Python environments for all of your Python
projects.

Python on $PATH is not appropriate
----------------------------------
Expand Down
15 changes: 15 additions & 0 deletions mach
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,21 @@ def main(args):
""").strip())
sys.exit(1)

if sys.version_info >= (3, 10) and sys.platform.startswith("darwin"):
version = f"{sys.version_info[0]}.{sys.version_info[1]}"
if f"python@{version}" in sys.executable:
print(dedent(f"""
You are using python {version} from homebrew.
There is a bug in virtualenv that prevents us from starting up mach in that situation.
Please uninstall it with the following and try again:
brew unlink python@{version}
If you need python {version} later, you can re-link it with:
brew link python@{version}
""").strip())
sys.exit(1)

# XCode python sets __PYVENV_LAUNCHER__, which overrides the executable
# used when a python subprocess is created. This is an issue when we want
# to run using our virtualenv python executables.
Expand Down
Loading

0 comments on commit c8b4aae

Please sign in to comment.