Providers

Providers are the parties (organization, person, community, etc.) that supply a service or software tool which interacts with Python packaging and consequently Python package installers. Two different types of providers are considered:

Developers

Developers are teams, people, or communities that code and use Python package installers and Python packages. Three different types of developers are considered:

Upgraders & Package Infrastructure Providers

Package upgraders and package infrastructure in CI/CD such as Dependabot, PyUP, etc. currently support a few installers. They work by parsing and editing the installer-specific dependency files directly (such as requirements.txt or poetry.lock) with relevant package information such as upgrades, downgrades, or new hashes. Similar to Platform and IDE providers, most of these providers do not want to support N different Python package installers as that would require supporting N different file types.

Currently, these services/bots have to implement support for each package installer individually. Inevitably, the most popular installers are supported first, and less popular tools are often never supported. By implementing this specification, these services/bots can support any (compliant) installer, allowing users to select the tool of their choice. This will allow for more innovation in the space, as platforms and IDEs are no longer forced to prematurely select a “winner”.

Open Source Community

Specifying installer requirements and adopting this PEP will reduce the friction between Python package installers and people’s workflows. Consequently, it will reduce the friction between Python package installers and 3rd party infrastructure/technologies such as PaaS or IDEs. Overall, it will allow for easier development, deployment and maintenance of Python projects as Python package installation becomes simpler and more interoperable.

Specifying requirements and creating an interface for installers can also increase the pace of innovation around installers. This would allow for installers to experiment and add unique functionality without requiring the rest of the ecosystem to do the same. Support becomes easier and more likely for a new installer regardless of the functionality it adds and the format in which it writes dependencies, while reducing the developer time and resources needed to do so.

[install-system]

The install-system table is used to store install-system relevant data and information. There are multiple required keys for this table: requires and install-backend. The requires key holds the minimum requirements for the installer backend to execute and which will be installed by the universal installer. The install-backend key holds the name of the install backend’s entry point. This will allow the universal installer to install the requirements for the installer backend itself to execute (not the requirements that the installer backend itself will install) as well as invoke the installer backend.

If either of the required keys are missing or empty then the universal installer SHOULD raise an error.

All package names interacting with this interface are assumed to follow PEP 508’s “Dependency specification for Python Software Packages” format.

An example install-system table:

#pyproject.toml
[install-system]
#Eg : pipenv
requires = ["pipenv"]
install-backend = "pipenv.api:main"

[tool.poetry.dev-dependencies]
dependencies = "dev"

[tool.poetry.deploy]
dependencies = "deploy"

Installer interface:

The installer interface contains mandatory and optional hooks. Compliant installer backends MUST implement the mandatory hooks and MAY implement the optional hooks. A universal installer MAY implement any of the installer backend hooks itself, to act as both a universal installer and installer backend, but this is not required.

All hooks take **kwargs arbitrary parameters that a installer backend may require that are not already specified, allowing for backwards compatibility. If unexpected parameters are passed to the installer backend, it should ignore them.

The following information is akin to the corresponding section in PEP 517. The hooks may be called with keyword arguments, so installer backends implementing them should be careful to make sure that their signatures match both the order and the names of the arguments above.

All hooks MAY print arbitrary informational text to stdout and stderr. They MUST NOT read from stdin, and the universal installer MAY close stdin before invoking the hooks.

The universal installer may capture stdout and/or stderr from the backend. If the backend detects that an output stream is not a terminal/console (e.g. not sys.stdout.isatty()), it SHOULD ensure that any output it writes to that stream is UTF-8 encoded. The universal installer MUST NOT fail if captured output is not valid UTF-8, but it MAY not preserve all the information in that case (e.g. it may decode using the replace error handler in Python). If the output stream is a terminal, the installer backend is responsible for presenting its output accurately, as for any program running in a terminal.

If a hook raises an exception, or causes the process to terminate, then this indicates an error.

Mandatory hooks:

def invoke_install(
    path: Union[str, bytes, PathLike[str]],
    *,
    dependency_group: str = None,
    **kwargs
) -> int:
    ...

Optional hooks:

def invoke_uninstall(
    path: Union[str, bytes, PathLike[str]],
    *,
    dependency_group: str = None,
    **kwargs
) -> int:
    ...

def get_dependencies_to_install(
    path: Union[str, bytes, PathLike[str]],
    *,
    dependency_group: str = None,
    **kwargs
) -> Sequence[str]:
    ...

def get_dependency_groups(
    path: Union[str, bytes, PathLike[str]],
    **kwargs
) -> AbstractSet[str]:
    ...

def update_dependencies(
    path: Union[str, bytes, PathLike[str]],
    dependency_specifiers: Iterable[str],
    *,
    dependency_group=None,
    **kwargs
) -> int:
    ...

A standardized lock file

A standardized lock file would solve a lot of the same problems that specifying installer requirements would. For example, it would allow for PaaS/IaaS to just support one installer that could read the standardized lock file regardless of the installer that created it. The problem with a standardized lock file is the difference in needs between Python package installers as well as a fundamental issue with creating reproducible environments via the lockfile (one of the main benefits).

Needs and information stored in dependency files between installers differ significantly and are dependent on installer functionality. For example, a Python package installer such as Poetry requires information for all Python versions and platforms and calculates appropriate hashes while pip doesn’t. Additionally, pip would not be able to guarantee recreating the same environment (install the exact same dependencies) as it is outside the scope of its functionality. This makes a standardized lock file harder to implement and makes it seem more appropriate to make lock files tool specific.

Have installer backends support creating virtual environments

Because installer backends will very likely have a concept of virtual environments and how to install into them, it was briefly considered to have them also support creating virtual environments. In the end, though, it was considered an orthogonal idea.

Should the dependency_group argument take an iterable?

This would allow for specifying non-overlapping dependency groups in a single call, e.g. “docs” and “test” groups which have independent dependencies but which a developer may want to install simultaneously while doing development.

Is the installer backend executed in-process?

If the installer backend is executed in-process then it greatly simplifies knowing what environment to install for/into, as the live Python environment can be queried for appropriate information.

Executing out-of-process allows for minimizing potential issues of clashes between the environment being installed into and the installer backend (and potentially universal installer).

Enforce that results from the proposed interface feed into other parts?

E.g. the results from get_dependencies_to_install() and get_dependency_groups() can be passed into invoke_install(). This would prevent drift between the results of various parts of the proposed interface, but it makes more of the interface required instead of optional.

Raising exceptions instead of exit codes for failure conditions

It has been suggested that instead of returning an exit code the API should raise exceptions. If you view this PEP as helping to translate current installers into installer backends, then relying on exit codes makes sense. There’s is also the point that the APIs have no specific return value, so passing along an exit code does not interfere with what the functions return.

Compare that to raising exceptions in case of an error. That could potentially provide a more structured approach to error raising, although to be able to capture errors it would require specifying exception types as part of the interface.