Scope

Object immortality is meant to be an internal-only feature, so this proposal does not include any changes to public API or behavior (with one exception). As usual, we may still add some private (yet publicly accessible) API to do things like immortalize an object or tell if one is immortal. Any effort to expose this feature to users would need to be proposed separately.

There is one exception to “no change in behavior”: refcounting semantics for immortal objects will differ in some cases from user expectations. This exception, and the solution, are discussed below.

Most of this PEP focuses on an internal implementation that satisfies the above mandate. However, those implementation details are not meant to be strictly proscriptive. Instead, at the least they are included to help illustrate the technical considerations required by the mandate. The actual implementation may deviate somewhat as long as it satisfies the constraints outlined below. Furthermore, the acceptability of any specific implementation detail described below does not depend on the status of this PEP, unless explicitly specified.

For example, the particular details of:

• how to mark something as immortal
• how to recognize something as immortal
• which subset of functionally immortal objects are marked as immortal
• which memory-management activities are skipped or modified for immortal objects

are not only CPython-specific but are also private implementation details that are expected to change in subsequent versions.

Implementation Summary

Here’s a high-level look at the implementation:

If an object’s refcount matches a very specific value (defined below) then that object is treated as immortal. The CPython C-API and runtime will not modify the refcount (or other runtime state) of an immortal object. The runtime will now be explicitly responsible for deallocating all immortal objects during finalization, unless statically allocated. (See Object Cleanup below.)

Aside from the change to refcounting semantics, there is one other possible negative impact to consider. The threshold for an “acceptable” performance penalty for immortal objects is 2% (the consensus at the 2022 Language Summit). A naive implementation of the approach described below makes CPython roughly 4% slower. However, the implementation is ~performance-neutral~ once known mitigations are applied.

TODO: Update the performance impact for the latest branch (both for GCC and for clang).

Reducing CPU Cache Invalidation

Every modification of a refcount causes the corresponding CPU cache line to be invalidated. This has a number of effects.

For one, the write must be propagated to other cache levels and to main memory. This has small effect on all Python programs. Immortal objects would provide a slight relief in that regard.

On top of that, multi-core applications pay a price. If two threads (running simultaneously on distinct cores) are interacting with the same object (e.g. None) then they will end up invalidating each other’s caches with each incref and decref. This is true even for otherwise immutable objects like True, 0, and str instances. CPython’s GIL helps reduce this effect, since only one thread runs at a time, but it doesn’t completely eliminate the penalty.

Avoiding Data Races

Speaking of multi-core, we are considering making the GIL a per-interpreter lock, which would enable true multi-core parallelism. Among other things, the GIL currently protects against races between multiple concurrent threads that may incref or decref the same object. Without a shared GIL, two running interpreters could not safely share any objects, even otherwise immutable ones like None.

This means that, to have a per-interpreter GIL, each interpreter must have its own copy of every object. That includes the singletons and static types. We have a viable strategy for that but it will require a meaningful amount of extra effort and extra complexity.

The alternative is to ensure that all shared objects are truly immutable. There would be no races because there would be no modification. This is something that the immortality proposed here would enable for otherwise immutable objects. With immortal objects, support for a per-interpreter GIL becomes much simpler.

Avoiding Copy-on-Write

For some applications it makes sense to get the application into a desired initial state and then fork the process for each worker. This can result in a large performance improvement, especially memory usage. Several enterprise Python users (e.g. Instagram, YouTube) have taken advantage of this. However, the above refcount semantics drastically reduce the benefits and have led to some sub-optimal workarounds.

Also note that “fork” isn’t the only operating system mechanism that uses copy-on-write semantics. Another example is mmap. Any such utility will potentially benefit from fewer copy-on-writes when immortal objects are involved, when compared to using only “mortal” objects.

Benefits

Most notably, the cases described in the above examples stand to benefit greatly from immortal objects. Projects using pre-fork can drop their workarounds. For the per-interpreter GIL project, immortal objects greatly simplifies the solution for existing static types, as well as objects exposed by the public C-API.

In general, a strong immutability guarantee for objects enables Python applications to scale better, particularly in multi-process deployments. This is because they can then leverage multi-core parallelism without such a significant tradeoff in memory usage as they now have. The cases we just described, as well as those described above in Motivation, reflect this improvement.

Performance

A naive implementation shows a 4% slowdown. We have demonstrated a return to ~performance-neutral~ with a handful of basic mitigations applied. See the mitigations section below.

On the positive side, immortal objects save a significant amount of memory when used with a pre-fork model. Also, immortal objects provide opportunities for specialization in the eval loop that would improve performance.

TODO: Update the performance impact for the latest branch (both for GCC and for clang).

Backward Compatibility

Ideally this internal-only feature would be completely compatible. However, it does involve a change to refcount semantics in some cases. Only immortal objects are affected, but this includes high-use objects like None, True, and False.

Specifically, when an immortal object is involved:

• code that inspects the refcount will see a really, really large value
• the new noop behavior may break code that:
  • depends specifically on the refcount to always increment or decrement (or have a specific value from Py_SET_REFCNT())
  • relies on any specific refcount value, other than 0 or 1
  • directly manipulates the refcount to store extra information there
• in 32-bit pre-3.12 Stable ABI extensions, objects may leak due to Accidental Immortality
• such extensions may crash due to Accidental De-Immortalizing

Again, those changes in behavior only apply to immortal objects, not the vast majority of objects a user will use. Furthermore, users cannot mark an object as immortal so no user-created objects will ever have that changed behavior. Users that rely on any of the changing behavior for global (builtin) objects are already in trouble. So the overall impact should be small.

Also note that code which checks for refleaks should keep working fine, unless it checks for hard-coded small values relative to some immortal object. The problems noticed by Pyston shouldn’t apply here since we do not modify the refcount.

See Public Refcount Details below for further discussion.

Alternate Python Implementations

This proposal is CPython-specific. However, it does relate to the behavior of the C-API, which may affect other Python implementations. Consequently, the effect of changed behavior described in Backward Compatibility above also applies here (e.g. if another implementation is tightly coupled to specific refcount values, other than 0, or on exactly how refcounts change, then they may impacted).

Security Implications

This feature has no known impact on security.

Maintainability

This is not a complex feature so it should not cause much mental overhead for maintainers. The basic implementation doesn’t touch much code so it should have much impact on maintainability. There may be some extra complexity due to performance penalty mitigation. However, that should be limited to where we immortalize all objects post-init and later explicitly deallocate them during runtime finalization. The code for this should be relatively concentrated.

Public Refcount Details

In Backward Compatibility we introduced possible ways that user code might be broken by the change in this proposal. Any contributing misunderstanding by users is likely due in large part to the names of the refcount-related API and to how the documentation explains those API (and refcounting in general).

Between the names and the docs, we can clearly see answers to the following questions:

• what behavior do users expect?
• what guarantees do we make?
• do we indicate how to interpret the refcount value they receive?
• what are the use cases under which a user would set an object’s refcount to a specific value?
• are users setting the refcount of objects they did not create?

As part of this proposal, we must make sure that users can clearly understand on which parts of the refcount behavior they can rely and which are considered implementation details. Specifically, they should use the existing public refcount-related API and the only refcount values with any meaning are 0 and 1. (Some code relies on 1 as an indicator that the object can be safely modified.) All other values are considered “not 0 or 1”.

This information will be clarified in the documentation.

Arguably, the existing refcount-related API should be modified to reflect what we want users to expect. Something like the following:

• Py_INCREF() -> Py_ACQUIRE_REF() (or only support Py_NewRef())
• Py_DECREF() -> Py_RELEASE_REF()
• Py_REFCNT() -> Py_HAS_REFS()
• Py_SET_REFCNT() -> Py_RESET_REFS() and Py_SET_NO_REFS()

However, such a change is not a part of this proposal. It is included here to demonstrate the tighter focus for user expectations that would benefit this change.

Constraints

• ensure that otherwise immutable objects can be truly immutable
• minimize performance penalty for normal Python use cases
• be careful when immortalizing objects that we don’t actually expect to persist until runtime finalization.
• be careful when immortalizing objects that are not otherwise immutable
• __del__ and weakrefs must continue working properly

Regarding “truly” immutable objects, this PEP doesn’t impact the effective immutability of any objects, other than the per-object runtime state (e.g. refcount). So whether or not some immortal object is truly (or even effectively) immutable can only be settled separately from this proposal. For example, str objects are generally considered immutable, but PyUnicodeObject holds some lazily cached data. This PEP has no influence on how that state affects str immutability.

Immortal Mutable Objects

Any object can be marked as immortal. We do not propose any restrictions or checks. However, in practice the value of making an object immortal relates to its mutability and depends on the likelihood it would be used for a sufficient portion of the application’s lifetime. Marking a mutable object as immortal can make sense in some situations.

Many of the use cases for immortal objects center on immutability, so that threads can safely and efficiently share such objects without locking. For this reason a mutable object, like a dict or list, would never be shared (and thus no immortality). However, immortality may be appropriate if there is sufficient guarantee that the normally mutable object won’t actually be modified.

On the other hand, some mutable objects will never be shared between threads (at least not without a lock like the GIL). In some cases it may be practical to make some of those immortal too. For example, sys.modules is a per-interpreter dict that we do not expect to ever get freed until the corresponding interpreter is finalized (assuming it isn’t replaced). By making it immortal, we would no longer incur the extra overhead during incref/decref.

We explore this idea further in the mitigations section below.

Implicitly Immortal Objects

If an immortal object holds a reference to a normal (mortal) object then that held object is effectively immortal. This is because that object’s refcount can never reach 0 until the immortal object releases it.

Examples:

• containers like dict and list
• objects that hold references internally like PyTypeObject with its tp_subclasses and tp_weaklist
• an object’s type (held in ob_type)

Such held objects are thus implicitly immortal for as long as they are held. In practice, this should have no real consequences since it really isn’t a change in behavior. The only difference is that the immortal object (holding the reference) doesn’t ever get cleaned up.

We do not propose that such implicitly immortal objects be changed in any way. They should not be explicitly marked as immortal just because they are held by an immortal object. That would provide no advantage over doing nothing.

Un-Immortalizing Objects

This proposal does not include any mechanism for taking an immortal object and returning it to a “normal” condition. Currently there is no need for such an ability.

On top of that, the obvious approach is to simply set the refcount to a small value. However, at that point there is no way in knowing which value would be safe. Ideally we’d set it to the value that it would have been if it hadn’t been made immortal. However, that value will have long been lost. Hence the complexities involved make it less likely that an object could safely be un-immortalized, even if we had a good reason to do so.

_Py_IMMORTAL_REFCNT

We will add two internal constants:

_Py_IMMORTAL_BIT - has the top-most available bit set (e.g. 2^62)
_Py_IMMORTAL_REFCNT - has the two top-most available bits set

The actual top-most bit depends on existing uses for refcount bits, e.g. the sign bit or some GC uses. We will use the highest bit possible after consideration of existing uses.

The refcount for immortal objects will be set to _Py_IMMORTAL_REFCNT (meaning the value will be halfway between _Py_IMMORTAL_BIT and the value at the next highest bit). However, to check if an object is immortal we will compare (bitwise-and) its refcount against just _Py_IMMORTAL_BIT.

The difference means that an immortal object will still be considered immortal, even if somehow its refcount were modified (e.g. by an older stable ABI extension).

Note that top two bits of the refcount are already reserved for other uses. That’s why we are using the third top-most bit.

The implementation is also open to using other values for the immortal bit, such as the sign bit or 2^31 (for saturated refcounts on 64-bit).

Affected API

API that will now ignore immortal objects:

• (public) Py_INCREF()
• (public) Py_DECREF()
• (public) Py_SET_REFCNT()
• (private) _Py_NewReference()

API that exposes refcounts (unchanged but may now return large values):

• (public) Py_REFCNT()
• (public) sys.getrefcount()

(Note that _Py_RefTotal, and consequently sys.gettotalrefcount(), will not be affected.)

TODO: clarify the status of _Py_RefTotal.

Also, immortal objects will not participate in GC.

Immortal Global Objects

All runtime-global (builtin) objects will be made immortal. That includes the following:

• singletons (None, True, False, Ellipsis, NotImplemented)
• all static types (e.g. PyLong_Type, PyExc_Exception)
• all static objects in _PyRuntimeState.global_objects (e.g. identifiers, small ints)

The question of making the full objects actually immutable (e.g. for per-interpreter GIL) is not in the scope of this PEP.

Object Cleanup

In order to clean up all immortal objects during runtime finalization, we must keep track of them.

For GC objects (“containers”) we’ll leverage the GC’s permanent generation by pushing all immortalized containers there. During runtime shutdown, the strategy will be to first let the runtime try to do its best effort of deallocating these instances normally. Most of the module deallocation will now be handled by pylifecycle.c:finalize_modules() where we clean up the remaining modules as best as we can. It will change which modules are available during __del__, but that’s already explicitly undefined behavior in the docs. Optionally, we could do some topological ordering to guarantee that user modules will be deallocated first before the stdlib modules. Finally, anything left over (if any) can be found through the permanent generation GC list which we can clear after finalize_modules() is done.

For non-container objects, the tracking approach will vary on a case-by-case basis. In nearly every case, each such object is directly accessible on the runtime state, e.g. in a _PyRuntimeState or PyInterpreterState field. We may need to add a tracking mechanism to the runtime state for a small number of objects.

None of the cleanup will have a significant effect on performance.

Performance Regression Mitigations

In the interest of clarity, here are some of the ways we are going to try to recover some of the 4% performance we lose with the naive implementation of immortal objects.

Note that none of this section is actually part of the proposal.

Solutions for Accidental De-Immortalization

In the Accidental De-Immortalizing section we outlined a possible negative consequence of immortal objects. Here we look at some of the options to deal with that.

Note that we enumerate solutions here to illustrate that satisfactory options are available, rather than to dictate how the problem will be solved.

Also note the following:

• this only matters in the 32-bit stable-ABI case
• it only affects immortal objects
• there are no user-defined immortal objects, only built-in types
• most immortal objects will be statically allocated (and thus already must fail if tp_dealloc() is called)
• only a handful of immortal objects will be used often enough to possibly face this problem in practice (e.g. None)
• the main problem to solve is crashes coming from tp_dealloc()

One fundamental observation for a solution is that we can reset an immortal object’s refcount to _Py_IMMORTAL_REFCNT when some condition is met.

With all that in mind, a simple, yet effective, solution would be to reset an immortal object’s refcount in tp_dealloc(). NoneType and bool already have a tp_dealloc() that calls Py_FatalError() if triggered. The same goes for other types based on certain conditions, like PyUnicodeObject (depending on unicode_is_singleton()), PyTupleObject, and PyTypeObject. In fact, the same check is important for all statically declared object. For those types, we would instead reset the refcount. For the remaining cases we would introduce the check. In all cases, the overhead of the check in tp_dealloc() should be too small to matter.

Other (less practical) solutions:

• periodically reset the refcount for immortal objects
• only do that for high-use objects
• only do it if a stable-ABI extension has been imported
• provide a runtime flag for disabling immortality

(The discussion thread has further detail.)

Regardless of the solution we end up with, we can do something else later if necessary.

TODO: Add a note indicating that the implemented solution does not affect the overall ~performance-neutral~ outcome.

Documentation

The immortal objects behavior and API are internal, implementation details and will not be added to the documentation.

However, we will update the documentation to make public guarantees about refcount behavior more clear. That includes, specifically:

• Py_INCREF() - change “Increment the reference count for object o.” to “Indicate taking a new reference to object o.”
• Py_DECREF() - change “Decrement the reference count for object o.” to “Indicate no longer using a previously taken reference to object o.”
• similar for Py_XINCREF(), Py_XDECREF(), Py_NewRef(), Py_XNewRef(), Py_Clear()
• Py_REFCNT() - add “The refcounts 0 and 1 have specific meanings and all others only mean code somewhere is using the object, regardless of the value. 0 means the object is not used and will be cleaned up. 1 means code holds exactly a single reference.”
• Py_SET_REFCNT() - refer to Py_REFCNT() about how values over 1 may be substituted with some over value

We may also add a note about immortal objects to the following, to help reduce any surprise users may have with the change:

• Py_SET_REFCNT() (a no-op for immortal objects)
• Py_REFCNT() (value may be surprisingly large)
• sys.getrefcount() (value may be surprisingly large)

Other API that might benefit from such notes are currently undocumented. We wouldn’t add such a note anywhere else (including for Py_INCREF() and Py_DECREF()) since the feature is otherwise transparent to users.

Prior Art

• Pyston

Discussions

This was discussed in December 2021 on python-dev:

• https://mail.python.org/archives/list/python-dev@python.org/thread/7O3FUA52QGTVDC6MDAV5WXKNFEDRK5D6/#TBTHSOI2XRWRO6WQOLUW3X7S5DUXFAOV
• https://mail.python.org/archives/list/python-dev@python.org/thread/PNLBJBNIQDMG2YYGPBCTGOKOAVXRBJWY

Runtime Object State

Here is the internal state that the CPython runtime keeps for each Python object:

• PyObject.ob_refcnt: the object’s refcount
• _PyGC_Head: (optional) the object’s node in a list of “GC” objects
• _PyObject_HEAD_EXTRA: (optional) the object’s node in the list of heap objects

ob_refcnt is part of the memory allocated for every object. However, _PyObject_HEAD_EXTRA is allocated only if CPython was built with Py_TRACE_REFS defined. PyGC_Head is allocated only if the object’s type has Py_TPFLAGS_HAVE_GC set. Typically this is only container types (e.g. list). Also note that PyObject.ob_refcnt and _PyObject_HEAD_EXTRA are part of PyObject_HEAD.

Reference Counting, with Cyclic Garbage Collection

Garbage collection is a memory management feature of some programming languages. It means objects are cleaned up (e.g. memory freed) once they are no longer used.

Refcounting is one approach to garbage collection. The language runtime tracks how many references are held to an object. When code takes ownership of a reference to an object or releases it, the runtime is notified and it increments or decrements the refcount accordingly. When the refcount reaches 0, the runtime cleans up the object.

With CPython, code must explicitly take or release references using the C-API’s Py_INCREF() and Py_DECREF(). These macros happen to directly modify the object’s refcount (unfortunately, since that causes ABI compatibility issues if we want to change our garbage collection scheme). Also, when an object is cleaned up in CPython, it also releases any references (and resources) it owns (before it’s memory is freed).

Sometimes objects may be involved in reference cycles, e.g. where object A holds a reference to object B and object B holds a reference to object A. Consequently, neither object would ever be cleaned up even if no other references were held (i.e. a memory leak). The most common objects involved in cycles are containers.

CPython has dedicated machinery to deal with reference cycles, which we call the “cyclic garbage collector”, or often just “garbage collector” or “GC”. Don’t let the name confuse you. It only deals with breaking reference cycles.

See the docs for a more detailed explanation of refcounting and cyclic garbage collection:

• https://docs.python.org/3.11/c-api/intro.html#reference-counts
• https://docs.python.org/3.11/c-api/refcounting.html
• https://docs.python.org/3.11/c-api/typeobj.html#c.PyObject.ob_refcnt
• https://docs.python.org/3.11/c-api/gcsupport.html