Following system colour scheme Selected dark colour scheme Selected light colour scheme

Python Enhancement Proposals

PEP 702 – Marking deprecations using the type system

Author:
Jelle Zijlstra <jelle.zijlstra at gmail.com>
Discussions-To:
Discourse thread
Status:
Draft
Type:
Standards Track
Topic:
Typing
Created:
30-Dec-2022
Python-Version:
3.12
Post-History:
01-Jan-2023, 22-Jan-2023

Table of Contents

Abstract

This PEP adds an @typing.deprecated() decorator that marks a class or function as deprecated, enabling static checkers to warn when it is used.

Motivation

As software evolves, new functionality is added and old functionality becomes obsolete. Library developers want to work towards removing obsolete code while giving their users time to migrate to new APIs. Python provides a mechanism for achieving these goals: the DeprecationWarning warning class, which is used to show warnings when deprecated functionality is used. This mechanism is widely used: as of the writing of this PEP, the CPython main branch contains about 150 distinct code paths that raise DeprecationWarning. Many third-party libraries also use DeprecationWarning to mark deprecations. In the top 5000 PyPI packages, there are:

  • 1911 matches for the regex warnings\.warn.*\bDeprecationWarning\b, indicating use of DeprecationWarning (not including cases where the warning is split over multiple lines);
  • 1661 matches for the regex ^\s*@deprecated, indicating use of some sort of deprecation decorator.

However, the current mechanism is often insufficient to ensure that users of deprecated functionality update their code in time. For example, the removal of various long-deprecated unittest features had to be reverted from Python 3.11 to give users more time to update their code. Users may run their test suite with warnings disabled for practical reasons, or deprecations may be triggered in code paths that are not covered by tests.

Providing more ways for users to find out about deprecated functionality can speed up the migration process. This PEP proposes to leverage static type checkers to communicate deprecations to users. Such checkers have a thorough semantic understanding of user code, enabling them to detect and report deprecations that a single grep invocation could not find. In addition, many type checkers integrate with IDEs, enabling users to see deprecation warnings right in their editors.

Rationale

At first glance, deprecations may not seem like a topic that type checkers should touch. After all, type checkers are concerned with checking whether code will work as is, not with potential future changes. However, the analysis that type checkers perform on code to find type errors is very similar to the analysis that would be needed to detect usage of many deprecations. Therefore, type checkers are well placed to find and report deprecations.

Other languages already have similar functionality:

  • GCC supports a deprecated attribute on function declarations. This powers CPython’s Py_DEPRECATED macro.
  • GraphQL supports marking fields as @deprecated.
  • Kotlin supports a Deprecated annotation.
  • Scala supports an @deprecated annotation.
  • Swift supports using the @available attribute to mark APIs as deprecated.
  • TypeScript uses the @deprecated JSDoc tag to issue a hint marking use of deprecated functionality.

Several users have requested support for such a feature:

There are similar existing third-party tools:

  • Deprecated provides a decorator to mark classes, functions, or methods as deprecated. Access to decorated objects raises a runtime warning, but is not detected by type checkers.
  • flake8-deprecated is a linter plugin that warns about use of deprecated features. However, it is limited to a short, hard-coded list of deprecations.

Specification

A new decorator @deprecated() is added to the typing module. This decorator can be used on a class, function or method to mark it as deprecated. This includes typing.TypedDict and typing.NamedTuple definitions. With overloaded functions, the decorator may be applied to individual overloads, indicating that the particular overload is deprecated. The decorator may also be applied to the overload implementation function, indicating that the entire function is deprecated.

The decorator takes a single argument of type str, which is a message that should be shown by the type checker when it encounters a usage of the decorated object. The message must be a string literal. The content of deprecation messages is up to the user, but it may include the version in which the deprecated object is to be removed, and information about suggested replacement APIs.

Type checkers should produce a diagnostic whenever they encounter a usage of an object marked as deprecated. For deprecated overloads, this includes all calls that resolve to the deprecated overload. For deprecated classes and functions, this includes:

  • References through module, class, or instance attributes (module.deprecated_object, module.SomeClass.deprecated_method, module.SomeClass().deprecated_method)
  • Any usage of deprecated objects in their defining module (x = deprecated_object() in module.py)
  • If import * is used, usage of deprecated objects from the module (from module import *; x = deprecated_object())
  • from imports (from module import deprecated_object)

There are some additional scenarios where deprecations could come into play:

  • An object implements a typing.Protocol, but one of the methods required for protocol compliance is deprecated.
  • A class uses the @override decorator from PEP 698 to assert that its method overrides a base class method, but the base class method is deprecated.

As these scenarios appear complex and relatively unlikely to come up in practice, this PEP does not mandate that type checkers detect them.

Example

As an example, consider this library stub named library.pyi:

from typing import deprecated

@deprecated("Use Spam instead")
class Ham: ...

@deprecated("It is pining for the fiords")
def norwegian_blue(x: int) -> int: ...

@overload
@deprecated("Only str will be allowed")
def foo(x: int) -> str: ...
@overload
def foo(x: str) -> str: ...

Here is how type checkers should handle usage of this library:

from library import Ham  # error: Use of deprecated class Ham. Use Spam instead.

import library

library.norwegian_blue(1)  # error: Use of deprecated function norwegian_blue. It is pining for the fiords.
map(library.norwegian_blue, [1, 2, 3])  # error: Use of deprecated function norwegian_blue. It is pining for the fiords.

library.foo(1)  # error: Use of deprecated overload for foo. Only str will be allowed.
library.foo("x")  # no error

Runtime behavior

At runtime, the decorator sets an attribute __deprecated__ on the decorated object. The value of the attribute is the message passed to the decorator. The decorator returns the original object. Notably, it does not issue a runtime DeprecationWarning.

For compatibility with typing.get_overloads(), the @deprecated decorator should be placed after the @overload decorator.

Type checker behavior

This PEP does not specify exactly how type checkers should present deprecation diagnostics to their users. However, some users (e.g., application developers targeting only a specific version of Python) may not care about deprecations, while others (e.g., library developers who want their library to remain compatible with future versions of Python) would want to catch any use of deprecated functionality in their CI pipeline. Therefore, it is recommended that type checkers provide configuration options that cover both use cases. As with any other type checker error, it is also possible to ignore deprecations using # type: ignore comments.

Deprecation policy

We propose that CPython’s deprecation policy (PEP 387) is updated to require that new deprecations use the functionality in this PEP to alert users about the deprecation, if possible. Concretely, this means that new deprecations should be accompanied by a change to the typeshed repo to add the @deprecated decorator in the appropriate place. This requirement does not apply to deprecations that cannot be expressed using this PEP’s functionality.

Backwards compatibility

Creating a new decorator poses no backwards compatibility concerns. As with all new typing functionality, the @deprecated decorator will be added to the typing_extensions module, enabling its use in older versions of Python.

How to teach this

For users who encounter deprecation warnings in their IDE or type checker output, the messages they receive should be clear and self-explanatory. Usage of the @deprecated decorator will be an advanced feature mostly relevant to library authors. The decorator should be mentioned in relevant documentation (e.g., PEP 387 and the DeprecationWarning documentation) as an additional way to mark deprecated functionality.

Reference implementation

A runtime implementation of the @deprecated decorator is available. The pyanalyze type checker has prototype support for emitting deprecation errors.

Rejected ideas

Deprecation of modules and attributes

This PEP covers deprecations of classes, functions and overloads. This allows type checkers to detect many but not all possible deprecations. To evaluate whether additional functionality would be worthwhile, I examined all current deprecations in the CPython standard library.

I found:

  • 74 deprecations of functions, methods and classes (supported by this PEP)
  • 28 deprecations of whole modules (largely due to PEP 594)
  • 9 deprecations of function parameters (supported by this PEP through decorating overloads)
  • 1 deprecation of a constant
  • 38 deprecations that are not easily detectable in the type system (for example, for calling asyncio.get_event_loop() without an active event loop)

Modules could be marked as deprecated by adding a __deprecated__ module-level constant. However, the need for this is limited, and it is relatively easy to detect usage of deprecated modules simply by grepping. Therefore, this PEP omits support for whole-module deprecations. As a workaround, users could mark all module-level classes and functions with @deprecated.

For deprecating module-level constants, object attributes, and function parameters, a Deprecated[type, message] type modifier, similar to Annotated could be added. However, this would create a new place in the type system where strings are just strings, not forward references, complicating the implementation of type checkers. In addition, my data show that this feature is not commonly needed.

Features for deprecating more kinds of objects could be added in a future PEP.

Open issues

Runtime warnings

Users might expect usage of the @deprecated decorator to issue a DeprecationWarning at runtime. However, this would raise a number of thorny issues:

  • When the decorator is applied to a class or an overload, the warning would not be raised as expected. For classes, the warning could be raised on instantiation, but this would not cover usage in type annotations or isinstance() checks.
  • Users may want to control the warn() call in more detail (e.g., changing the warning class).
  • typing.py generally aims to avoid affecting runtime behavior.
  • To raise a warning, the @deprecated decorator would have to wrap functions and patch __new__ on classes. This would complicate runtime introspection.
  • Users may not expect usage of an object from the typing module to affect runtime behavior.

Users who want to use @deprecated while also issuing a runtime warning can use the if TYPE_CHECKING: idiom, for example:

from typing import TYPE_CHECKING
import functools
import warnings

if TYPE_CHECKING:
    from typing import deprecated
else:
    def deprecated(msg):
        def decorator(func):
            @functools.wraps(func)
            def wrapper(*args, **kwargs):
                warnings.warn(msg, DeprecationWarning, stacklevel=2)
                return func(*args, **kwargs)
            wrapper.__deprecated__ = msg
            return wrapper
        return decorator

While this code block looks complex, it could be encapsulated in a library.

Still, the behavior could be made opt-in, and perhaps the benefits of incorporating a runtime warning outweigh the costs.

Acknowledgments

A call with the typing-sig meetup group led to useful feedback on this proposal.


Source: https://github.com/python/peps/blob/main/pep-0702.rst

Last modified: 2023-01-24 15:23:21 GMT