Python 3.11, pip and (breaking) system packages

As Debian Bookworm is released, I thought I’d share one change in Python 3.11 that will surely affect many people.

Python 3.11 implements the new PEP 668, Marking Python base environments as “externally managed”. If you use pip regularly on Debian, it’s likely you’ll eventually hit the externally-managed-environment error:

error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.

    See /usr/share/doc/python3.11/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

With this PEP, Python tools can now distinguish between packages that have been installed by the user with a tool like pip and ones installed using a distribution’s package manager, like apt.

This is generally great news: it was previously too easy to break a system by mixing the two types of packages. This PEP will simplify our role as a distribution, as well as improve the overall Python user experience in Debian.

Sadly, it’s also likely this change will break some of your scripts, especially CI that (legitimately) install packages via pip alongside system packages. For example, the following gitlab-ci snippet to make sure PRs don’t break a build process:

build:flit:
  stage: build
  script:
  - apt-get update && apt-get install -y flit python3-pip
  - FLIT_ROOT_INSTALL=1 flit install
  - metalfinder --help

With Python 3.11, this snippet will error out, as pip will refuse to install packages alongside the system’s. The fix is to tell pip it’s OK to “break” your system packages, either using the --break-system-packages parameter, or the PIP_BREAK_SYSTEM_PACKAGES=1 environment variable.

This, of course, is not something you should be using in production to restore the old behavior! The “proper” way to fix this issue, as the externally-managed-environment error message aptly (har har) informs you, is to use virtual environments.

Source:
https://veronneau.org/python-311-pip-and-breaking-system-packages.html