Baseline feature releases and feature release series

Baseline feature releases are essentially just the existing feature releases, given a new name to help distinguish them from the new incremental feature releases, and also to help indicate that unlike their predecessors, they are no longer considered feature complete at release.

Baseline feature releases would continue to define a new feature release series, locking in the following language, build, and installation compatibility constraints for the remainder of that series:

• Python language grammar
• ast module AST format
• CPython interpreter opcode format
• pyc file magic number and filename compatibility tags
• extension module filename compatibility tags
• wheel archive compatibility tags
• default package and module import directories
• default installation filename and directories

Baseline feature releases would also continue to be the only releases where:

• new deprecations, pending deprecations, and other warnings can be introduced
• existing pending deprecations can be converted to full deprecations
• existing warnings can be converted to errors
• other changes requiring “Porting to Python X.Y” entries in the What’s New document can be introduced

Key characteristics of a feature release series:

• an installation within one feature release series does not conflict with installations of other feature release series (i.e. they can be installed in parallel)
• an installation within a feature release series can be updated to a later micro release within the same series without requiring reinstallation or any other changes to previously installed components

Key characteristics of a baseline feature release:

• in a baseline feature release, sys.version_info.feature_complete == False
• in a baseline feature release, sys.version_info.micro == 0
• baseline feature releases may contain higher risk changes to the language and interpreter, such as grammar modifications, major refactoring of interpreter and standard library internals, or potentially invasive feature additions that carry a risk of unintended side effects on other existing functionality
• features introduced in a baseline feature release are the only features permitted to rely on sys.version_info as their sole runtime indicator of the feature’s availability

Key expectations around feature release series and baseline feature releases:

• most public projects will only actively test against the most recent micro release within a release series
• many (most?) public projects will only add a new release series to their test matrix after the initial baseline feature release has already been published, which can make it difficult to resolve issues that require providing new flags or APIs to explicitly opt-in to old behaviour after a default behaviour changed
• private projects with known target environments will test against whichever micro release version they’re actually using
• most private projects will also only consider migrating to a new release series after the initial baseline feature release has already been published, again posing a problem if the resolution of their problems requires an API addition

The key motivation of the proposal in this PEP is that the public and private project behaviours described above aren’t new expectations: they’re descriptions of the way CPython release series are already handled by the wider community today. As such, the PEP represents an attempt to adjust our release policies and processes to better match the way the wider community already handles them, rather than changing our processes in a way that then means the wider community needs to adjust to us rather than the other way around.

Incremental feature releases

Incremental feature releases are the key new process addition being proposed by this PEP. They are subject to the same strict runtime compatibility requirements as the existing maintenance releases, but would have the following more relaxed policies around API additions and enhancements:

• new public APIs can be added to any standard library module (including builtins)
• subject to the feature detection requirement below, new optional arguments can be added to existing APIs (including builtins)
• new public APIs can be added to the stable C ABI (with appropriate version guards)
• new public APIs can be added to the CPython C API
• with the approval of the release manager, backwards compatible reliability improvements can be made to existing APIs and syntactic constructs
• with the approval of the release manager, performance improvements can be incorporated for existing APIs and syntactic constructs

The intent of this change in policy is to allow usability improvements for new (and existing!) language features to be delivered in a more timely fashion, rather than requiring users to incur the inherent delay and costs of waiting for and then upgrading to the next feature release series.

It is also designed such that the approval to add a feature to the next baseline feature release can be considered separately from the question of whether or not to make it available in the next incremental feature release for the current release series, potentially allowing the first task to be completed by volunteer contributors, while the latter activity could be handled by paid contributors (e.g. customers of commercial Python redistributors could potentially request that their vendor backport a feature, or core developers could offer to undertake specific backports on a contract basis). (There would be potential ethical concerns with gating bug fixes this way, but those concerns don’t apply for backports of new features)

Key characteristics of an incremental feature release:

• in an incremental feature release, sys.version_info.feature_complete == False
• in an incremental feature release, sys.version_info.micro != 0
• all API additions made in an incremental feature release must support efficient runtime feature detection that doesn’t rely on either sys.version_info or runtime code object introspection. In most cases, a simple hasattr check on the affected module will serve this purpose, but when it doesn’t, an alternative approach will need to be implemented as part of the feature addition. Prior art in this area includes the pickle.HIGHEST_PROTOCOL attribute, the hashlib.algorithms_available set, and the various os.supports_* sets that the os module already offers for platform dependent capability detection
• to maintain pickle compatibility in mixed version environments, and to enable easier compatibility testing across multiple API versions within the same release series, all API additions made in an incremental feature release must support the new sys.feature_limit setting as described in the next section

Key expectations around incremental feature releases:

• “don’t break existing installations on upgrade” remains a key requirement for all micro releases, even with the more permissive change inclusion policy
• more intrusive changes should still be deferred to the next baseline feature release
• public Python projects that start relying on features added in an incremental feature release should set their Python-Requires metadata appropriately (projects already do this when necessary - e.g. aiohttp specifically requires 3.5.3 or later due to an issue with asyncio.get_event_loop() in earlier versions)

Some standard library modules may also impose their own restrictions on acceptable changes in incremental feature releases (for example, only a baseline feature release should ever add new hash algorithms to hashlib.algorithms_guaranteed - incremental feature releases would only be permitted to add algorithms to hashlib.algorithms_available)

Maintaining interoperability across incremental feature releases

It is a common practice to use Python’s pickle module to exchange information between Python processes running on different versions of Python. Between release series, this compatibility is expected to only run one way (i.e. excluding deprecated APIs, Python “X.Y+1” processes should be able to read pickle archives produced by Python “X.Y” processes, but the reverse does not hold, as the newer archives may reference attributes and parameters that don’t exist in the older version).

Within a release series, however, it is expected to hold in both directions, as the “No new features” policy means that almost all pickle archives created on Python “X.Y.Z+1” will be readable by Python “X.Y.Z” processes.

Similarly, Python libraries and applications are often only tested against the latest version in a release series, and this is usually sufficient to keep code working on earlier releases in that same series.

Allowing feature additions in later “X.Y.Z” releases with no way to turn them off would pose a problem for these common practices, as a library or application that works fine when tested on CPython version “X.Y.Z” would fail on earlier versions if it used a feature newly introduced in “X.Y.Z”, and any pickle archives it creates that rely on those new interfaces may also not be readable on the older versions.

To help address these problems, a new sys.feature_limit attribute would be added, as a structured sequence corresponding to the first 3 fields in sys.version_info (major, minor, micro).

A new CLI option (--feature-limit X.Y.Z) and environment variable (PYTHONFEATURELIMIT=X.Y.Z) would be used to set this attribute. The PyCoreConfig struct would also gain a new field:

wchar_t *feature_limit;

If the limit is not set explicitly, it would default to the first 3 fields in sys.version_info. If the limit is set to a value outside the lower bound of sys.version_info[:2] and the upper bound of sys.version_info[:3], it will be clamped to those bounds, padding with zeroes if necessary.

For example, given a current version of “3.9.3”, nominal limits would be converted to runtime sys.feature_limit values as follows:

3 => (3, 9, 0)
3.8.1 => (3, 9, 0)
3.9 => (3, 9, 0)
3.9.2 => (3, 9, 2)
<unset> => (3, 9, 3)
3.9.3 => (3, 9, 3)
3.9.4 => (3, 9, 3)
4 => (3, 9, 3)

New APIs backported to an incremental feature release would be expected to include a guard that deletes the API from the module if the feature limit is too low:

def feature_api():
    ...

_version_feature_api_added = (3, 9, 1)
if _version_feature_api_added > sys.feature_limit:
    del feature_api

Similarly, new parameters would be expected to include a guard that adjusts the function signature to match the old one:

def feature_api(old_param1, old_param2, new_param=default):
    """Updated API docstring"""
    ...

_version_feature_api_changed = (3, 9, 1)
if _version_feature_api_changed > sys.feature_limit:
    _new_feature_api = feature_api
    def feature_api(old_param1, old_param2):
        """Legacy API docstring"""
        return _new_feature_api(old_param1, old_param2)

Structuring the guards this way would keep the code structure as similar as possible between the main development branch and the backport branches, so future bug fixes can still be backported automatically.

It is expected that convenience functions and/or additional automated tests would eventually be added to help ensure these backported APIs are guarded appropriately, but it seems reasonable to wait until specific concrete examples are available to drive the design of those APIs and automated tests, rather than designing them solely on the basis of hypothetical examples.

Feature complete release and subsequent maintenance releases

The feature complete release for a given feature release series would be developed under the normal policy for an incremental feature release, but would have one distinguishing feature:

• in a feature complete release, sys.version_info.feature_complete == True

Any subsequent maintenance and security fix only releases would also have that flag set, and may informally be referred to as “feature complete releases”. For release series definition purposes though, the feature complete release is the first one that sets that flag to “True”.

Proposed policy adjustment for provisional APIs

To help improve consistency in management of provisional APIs, this PEP proposes that provisional APIs be subject to regular backwards compatibility requirements following the feature complete release for a given release series.

Other aspects of managing provisional APIs would remain as they are today, so as long as an API remains in the provisional state, regular backwards compatibility requirements would not apply to that API in baseline and incremental feature releases.

This policy is expected to provide increased clarity to end users (as even provisional APIs will become stable for that release series in the feature complete release), with minimal practical downsides for standard library maintainers, based on the following analysis of documented API additions and changes in micro releases of CPython since 3.0.0:

• 21 3.x.1 version added/changed notes
• 30 3.x.2 version added/changed notes
• 18 3.x.3 version added/changed notes
• 11 3.x.4 version added/changed notes
• 1 3.x.5 version added/changed notes
• 0 3.x.6+ version added/changed notes

When post-baseline-release changes need to be made, the majority of them occur within the first two maintenance releases, which have always occurred within 12 months of the baseline release.

(Note: these counts are not solely for provisional APIs - they cover all APIs where semantic changes were made after the baseline release that were considered necessary to cover in the documentation. To avoid double counting changes, the numbers exclude any change markers from the What’s New section)

Why this proposal over simply doing more frequent baseline feature releases?

The filesystem layout changes and other inherently incompatible changes involved in a baseline feature release create additional work for large sections of the wider Python community.

Decoupling those layout changes from the Python version numbering scheme is also something that would in and of itself involve making backwards incompatible changes, as well as adjusting community expectations around which versions will install over the top of each other, and which can be installed in parallel on a single system.

We also don’t have a straightforward means to communicate to the community variations in support periods like “Only support Python version X.Y until X.Y+1 is out, but support X.Z until X.Z+2 is out”.

So this PEP takes as its starting assumption that the vast majority of Python users simply shouldn’t need to care that we’re changing our release policy, and the only folks that should be affected are those that are eagerly waiting for standard library improvements (and other backwards compatible interpreter enhancements), and those that need to manage mission critical applications in complex deployment environments.

Implications for Python library development

Many Python libraries (both open source and proprietary) currently adopt the practice of testing solely against the latest micro release within each feature release series that the project still supports.

The design assumption in this PEP is that this practice will continue to be followed during the feature release phase of a release series, with the expectation being that anyone choosing to adopt a new release series before it is feature complete will closely track the incremental feature releases.

Libraries that support a previous feature release series are unlikely to adopt features added in an incremental feature release, and if they do adopt such a feature, then any associated fallback compatibility strategies should be implemented in such a way that they’re also effective on the earlier releases in that release series.

Implications for the proposed Scientific Python ecosystem support period

Based on discussions at SciPy 2019, a NEP is currently being drafted [2] to define a common convention across the Scientific Python ecosystem for dropping support for older Python versions.

While the exact formulation of that policy is still being discussed, the initial proposal was very simple: support any Python feature release published within the last 42 months.

For an 18-month feature release cadence, that works out to always supporting at least the two most recent feature releases, and then dropping support for all X.Y.z releases around 6 months after X.(Y+2).0 is released. This means there is a 6-month period roughly every other year where the three most recent feature releases are supported.

For a 12-month release cadence, it would work out to always supporting at least the three most recent feature releases, and then dropping support for all X.Y.z releases around 6 months after X.(Y+3).0 is released. This means that for half of each year, the four most recent feature releases would be supported.

For a 24-month release cadence, a 42-month support cycle works out to always supporting at least the most recent feature release, and then dropping support for all X.Y.z feature releases around 18 months after X.(Y+1).0 is released. This means there is a 6-month period every other year where only one feature release is supported (and that period overlaps with the pre-release testing period for the X.(Y+2).0 baseline feature release).

Importantly for the proposal in this PEP, that support period would abide by the recommendation that library developers maintain support for the previous release series until the latest release series has attained feature complete status: dropping support 18 months after the baseline feature release will be roughly equivalent to dropping support 6 months after the feature complete release, without needing to track exactly which release marked the series as feature complete.

Implications for simple deployment environments

For the purposes of this PEP, a “simple” deployment environment is any use case where it is straightforward to ensure that all target environments are updated to a new Python micro version at the same time (or at least in advance of the rollout of new higher level application versions), and there isn’t any requirement for older Python versions to be able to reliably read pickle streams generated with the newer Python version, such that any pre-release testing that occurs need only target a single Python micro version.

The simplest such case would be scripting for personal use, where the testing and target environments are the exact same environment.

Similarly simple environments would be containerised web services, where the same Python container is used in the CI pipeline as is used on deployment, and any application that bundles its own Python runtime, rather than relying on a pre-existing Python deployment on the target system.

For these use cases, this PEP shouldn’t have any significant implications - only a single micro version needs to be tested, independently of whether that version is feature complete or not.

Implications for complex deployment environments

For the purposes of this PEP, “complex” deployment environments are use cases which don’t meet the “simple deployment” criterion above: new application versions are combined with two or more distinct micro versions within the same release series as part of the deployment process, rather than always targeting exactly one micro version at a time.

If the proposal in this PEP has the desired effect of reducing feature delivery latency, then it can be expected that developers using a release series that is not yet feature complete will actually make use of the new features as they’re made available. This then means that testing against a newer incremental feature release becomes an even less valid test of compatibility with the baseline feature release and older incremental feature releases than testing against a newer maintenance release is for older maintenance releases.

One option for handling such cases is to simply prohibit the use of new Python versions until the series has reached “feature complete” status. Such a policy is effectively already adopted by many organisations when it comes to new feature release series, with acceptance into operational environments occurring months or years after the original release. If this policy is adopted, then such organisations could potentially still adopt a new Python version every other year - it would just be based on the availability of the feature complete releases, rather than the baseline feature releases.

A less strict alternative to outright prohibition would be to make use of the proposed PYTHONFEATURELIMIT setting to enable phased migrations to new incremental feature releases:

• initially roll out Python X.Y.0 with PYTHONFEATURELIMIT=X.Y.0 set in CI and on deployment
• roll out Python X.Y.1 to CI, keeping the PYTHONFEATURELIMIT=X.Y.0 setting
• deploy Python X.Y.1 to production based on successful CI results
• update deployment environments to set PYTHONFEATURELIMIT=X.Y.1
• set PYTHONFEATURELIMIT=X.Y.1 in CI only after all deployment environments have been updated
• repeat this process for each new release up to and including the feature complete release for the release series
• once the series is feature complete, either continue with this same process for consistency’s sake, or else stop updating PYTHONFEATURELIMIT and leave it at the feature complete version number

Duration of the feature additions period

This PEP proposes that feature additions be limited to 12 months after the initial baseline feature release.

The primary motivation for that is specifically to sync up with the Ubuntu LTS timing, such that the feature complete release for the Python 3.9.x series gets published in October 2021, ready for inclusion in the Ubuntu 22.04 release. (other LTS Linux distributions like RHEL, SLES, and Debian don’t have a fixed publishing cadence, so they can more easily tweak their LTS timing a bit to align with stable versions of their inputs. Canonical deliberately haven’t given themselves that flexibility with their own release cycle).

The 12 month feature addition period then arises from splitting the time from the 2019-10 release of Python 3.8.0 and a final Python 3.9.x incremental feature release in 2021-10 evenly between pre-release development and subsequent incremental feature releases.

This is an area where this PEP could adopt part of the proposal in PEP 596, by instead making that split ~9 months of pre-release development, and ~15 months of incremental feature releases:

• 2019-11: 3.9.0a1
• … additional alpha releases as determined by the release manager
• 2020-03: 3.9.0b1
• 2020-04: 3.9.0b2
• 2020-05: 3.9.0b3 (final beta release that locks ABI compatibility)
• 2020-06: 3.9.0rc1
• … additional release candidates as determined by the release manager
• 2020-07: 3.9.0 (BFR)
• 2020-10: 3.9.1 (IFR)
• 2021-01: 3.9.2 (IFR)
• 2021-04: 3.9.3 (IFR)
• 2021-07: 3.9.4 (IFR)
• 2021-10: 3.9.5
• 2022-01: 3.9.6
• 2022-04: 3.9.7
• 2022-07: 3.9.8
• 2022-10: 3.9.9 (final regular maintenance release)
• … additional security fix only releases as needed
• 2025-10: 3.9.x branch closed

This approach would mean there were still always two or three active branches, it’s just that proportionally more time would be spent with a branch in the “feature releases” phase, as compared to the “pre-alpha”, “pre-beta”, and “pre-release” phases:

• 2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x maintenance
• 2019-10 -> 2020-03: 3.9.0 pre-beta, 3.8.x maintenance
• 2020-03 -> 2020-07: 3.10.0 pre-alpha, 3.9.0 pre-release, 3.8.x maintenance
• 2020-07 -> 2021-10: 3.10.0 pre-alpha, 3.9.x feature releases, 3.8.x maintenance
• 2021-10 -> 2022-03: 3.10.0 pre-beta, 3.9.x maintenance
• 2022-03 -> 2022-07: 3.11.0 pre-alpha, 3.10.0 pre-release, 3.9.x maintenance
• 2022-07 -> 2023-10: 3.11.0 pre-alpha, 3.10.x feature releases, 3.9.x maintenance
• 2023-10 -> 2024-03: 3.11.0 pre-beta, 3.10.x maintenance
• 2024-03 -> 2024-07: 3.12.0 pre-alpha, 3.11.0 pre-release, 3.10.x maintenance
• … etc

Duration of the unreleased pre-alpha period

In the baseline proposal in this PEP, the proposed timelines still include periods where we go for 18 months without making a release from the main git branch (e.g. 3.9.0b1 would branch off in 2020-05, and 3.10.0a1 wouldn’t be published until 2021-11). They just allow for a wider variety of changes to be backported to the most recent maintenance branch for 12 of those months.

The variant of the proposal that moves the beta branch point earlier in the release series lifecycle would increase that period of no direct releases to 21 months - the only period where releases were made directly from the main branch would be during the relatively short window between the last incremental feature release of the previous release series, and the beta branch point a few months later.

While alternating the annual cadence between “big foundational enhancements” and “targeted low risk API usability improvements” is a deliberate feature of this proposal, it still seems strange to wait that long for feedback in the event that changes are made shortly after the previous release series is branched.

An alternative way of handling this would be to start publishing alpha releases for the next baseline feature release during the feature addition period (similar to the way that PEP 596 proposes to starting publishing Python 3.9.0 alpha releases during the Python 3.8.0 release candidate period).

However, rather than setting specific timelines for that at a policy level, it may make sense to leave that decision to individual release managers, based on the specific changes that are being proposed for the release they’re managing.

Why not switch directly to full semantic versioning?

If this were a versioning design document for a new language, it would use semantic versioning: the policies described above for baseline feature releases would be applied to X.0.0 releases, the policies for incremental feature releases would be applied to X.Y.0 releases, and the policies for maintenance releases would be applied to X.Y.Z releases.

The problem for Python specifically is that all the policies and properties for parallel installation support and ABI compatibility definitions are currently associated with the first two fields of the version number, and it has been that way for the better part of thirty years.

As a result, it makes sense to split out the policy question of introducing incremental feature releases in the first place from the technical question of making the version numbering scheme better match the semantics of the different release types.

If the proposal in this PEP were to be accepted by the Steering Council for Python 3.9, then a better time to tackle that technical question would be for the subsequent October 2022 baseline feature release, as there are already inherent compatibility risks associated with the choice of either “Python 4.0” (erroneous checks for the major version being exactly 3 rather than 3 or greater), or “Python 3.10” (code incorrectly assuming that the minor version will always contain exactly one decimal digit) [1].

While the text of this PEP assumes that the release published in 2022 will be 3.10 (as the PEP author personally considers that the more reasonable and most likely choice), there are complex pros and cons on both sides of that decision, and this PEP does arguably add a potential pro in favour of choosing the “Python 4.0” option (with the caveat that we would also need to amend the affected installation layout and compatibility markers to only consider the major version number, rather than both the major and minor version).

If such a version numbering change were to be proposed and accepted, then the example 3.10.x timeline given above would instead become the following 4.x series timeline:

• 2021-11: 4.0.0a1
• … additional alpha releases as determined by the release manager
• 2022-05: 4.0.0b1
• … additional beta releases as determined by the release manager
• 2022-08: 4.0.0bX (final beta release that locks ABI compatibility)
• 2022-09: 4.0.0rc1
• … additional release candidates as determined by the release manager
• 2022-10: 4.0.0 (BFR)
• 2023-01: 4.1.0 (IFR)
• 2023-04: 4.2.0 (IFR)
• 2023-07: 4.3.0 (IFR)
• 2023-10: 4.4.0 (IFR)
• 2024-01: 4.4.1
• 2024-04: 4.4.2
• 2024-07: 4.4.3
• 2024-10: 4.4.4 (final regular maintenance release)
• … additional security fix only releases as needed
• 2027-10: 4.x branch closed

And the 5 year schedule forecast would look like:

• 2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x maintenance
• 2019-10 -> 2020-05: 3.9.0 pre-beta, 3.8.x maintenance
• 2020-05 -> 2020-10: 4.0.0 pre-alpha, 3.9.0 pre-release, 3.8.x maintenance
• 2020-10 -> 2021-10: 4.0.0 pre-alpha, 3.9.x feature releases, 3.8.x maintenance
• 2021-10 -> 2022-05: 4.0.0 pre-beta, 3.9.x maintenance
• 2022-05 -> 2022-10: 5.0.0 pre-alpha, 4.0.0 pre-release, 3.9.x maintenance
• 2022-10 -> 2023-10: 5.0.0 pre-alpha, 4.x.0 feature releases, 3.9.x maintenance
• 2023-10 -> 2024-05: 5.0.0 pre-beta, 4.x.y maintenance
• 2024-05 -> 2024-10: 6.0.0 pre-alpha, 5.0.0 pre-release, 4.x.y maintenance
• … etc