PEP 576 – Rationalize Built-in function classes
- Author:
- Mark Shannon <mark at hotpy.org>
- BDFL-Delegate:
- Petr Viktorin
- Status:
- Withdrawn
- Type:
- Standards Track
- Created:
- 10-May-2018
- Python-Version:
- 3.8
- Post-History:
- 17-May-2018, 23-Jun-2018, 08-Jul-2018, 29-Mar-2019
Abstract
Expose the “FastcallKeywords” convention used internally by CPython to third-party code, and make the inspect
module use duck-typing.
In combination this will allow third-party C extensions and tools like Cython to create objects that use the same calling conventions as built-in and Python functions, thus gaining performance parity with built-in functions like len
or print
.
A small improvement in the performance of existing code is expected.
Motivation
Currently third-party module authors face a dilemma when implementing functions in C. Either they can use one of the pre-existing built-in function or method classes or implement their own custom class in C. The first choice causes them to lose the ability to access the internals of the callable object. The second choice is an additional maintenance burden and, more importantly, has a significant negative impact on performance.
This PEP aims to allow authors of third-party C modules, and tools like to Cython, to utilize the faster calling convention used internally by CPython for built-in functions and methods, and to do so without a loss of capabilities relative to a function implemented in Python.
Introspection
The inspect module will fully support duck-typing when introspecting callables.
The inspect.Signature.from_callable()
function computes the signature of a callable. If an object has a __signature__
property, then inspect.Signature.from_callable()
simply returns that. To further support duck-typing, if a callable has a __text_signature__
then the __signature__
will be created from that.
This means that 3rd party builtin-functions can implement __text_signature__
if sufficient,
and the more expensive __signature__
if necessary.
Efficient calls to third-party callables
Currently the majority of calls are dispatched to function
s and method_descriptor
s in custom code, using the “FastcallKeywords” internal calling convention. This PEP proposes that this calling convention is implemented via a C function pointer. Third-party callables which implement this binary interface will have the potential to be called as fast as a built-in function.
Continued prohibition of callable classes as base classes
Currently any attempt to use function
, method
or method_descriptor
as a base class for a new class will fail with a TypeError
. This behaviour is desirable as it prevents errors when a subclass overrides the __call__
method. If callables could be sub-classed then any call to a function
or a method_descriptor
would need an additional check that the __call__
method had not been overridden. By exposing an additional call mechanism, the potential for errors becomes greater. As a consequence, any third-party class implementing the addition call interface will not be usable as a base class.
New classes and changes to existing classes
Python visible changes
- A new built-in class,
builtin_function
, will be added. types.BuiltinFunctionType
will refer tobuiltin_function
notbuiltin_function_or_method
.- Instances of the
builtin_function
class will retain the__module__
property ofbuiltin_function_or_method
and gain thefunc_module
andfunc_globals
properties. Thefunc_module
allows access to the module to which the function belongs. Note that this is different from the__module__
property which merely returns the name of the module. Thefunc_globals
property is equivalent tofunc_module.__dict__
and is provided to mimic the Python function property of the same name. - When binding a
method_descriptor
instance to an instance of its owning class, abound_method
will be created instead of abuiltin_function_or_method
. This means that themethod_descriptors
now mimic the behaviour of Python functions more closely. In other words,[].append
becomes abound_method
instead of abuiltin_function_or_method
.
C API changes
- A new function
PyBuiltinFunction_New(PyMethodDef *ml, PyObject *module)
is added to create built-in functions. PyCFunction_NewEx()
andPyCFunction_New()
are deprecated and will return aPyBuiltinFunction
if able, otherwise abuiltin_function_or_method
.
Retaining backwards compatibility in the C API and ABI
The proposed changes are fully backwards and forwards compatible at both the API and ABI level.
Internal C changes
Two new flags will be allowed for the typeobject.tp_flags
field.
These are Py_TPFLAGS_EXTENDED_CALL
and Py_TPFLAGS_FUNCTION_DESCRIPTOR
Py_TPFLAGS_EXTENDED_CALL
For any built-in class that sets Py_TPFLAGS_EXTENDED_CALL
The C struct corresponding to this built-in class must begin with the struct PyExtendedCallable
which is defined as follows:
typedef PyObject *(*extended_call_ptr)(PyObject *callable, PyObject** args,
int positional_argcount, PyTupleObject* kwnames);
typedef struct {
PyObject_HEAD
extended_call_ptr ext_call;
} PyExtendedCallable;
Any class that sets the Py_TPFLAGS_EXTENDED_CALL
cannot be used as a base class and a TypeError will be raised if any Python code tries to use it a base class.
Py_TPFLAGS_FUNCTION_DESCRIPTOR
If this flag is set for a built-in class F
, then instances of that class are expected to behave the same as a Python function when used as a class attribute.
Specifically, this mean that the value of c.m
where C.m
is an instanceof the built-in class F
(and c
is an instance of C
) must be a bound-method binding C.m
and c
.
Without this flag, it would be impossible for custom callables to behave like Python functions and be efficient as Python or built-in functions.
Changes to existing C structs
The function
, method_descriptor
and method
classes will have their corresponding structs changed to
start with the PyExtendedCallable
struct.
Third-party built-in classes using the new extended call interface
To enable call performance on a par with Python functions and built-in functions, third-party callables should set the Py_TPFLAGS_EXTENDED_CALL
bit of tp_flags
and ensure that the corresponding C struct starts with the PyExtendedCallable
.
Any built-in class that has the Py_TPFLAGS_EXTENDED_CALL
bit set must also implement the tp_call
function and make sure its behaviour is consistent with the ext_call
function.
Performance implications of these changes
Adding a function pointer to each callable, rather than each class of callable, enables the choice of dispatching function (the code to shuffle arguments about and do error checking) to be made when the callable object is created rather than when it is called. This should reduce the number of instructions executed between the call-site in the interpreter and the execution of the callee.
Alternative Suggestions
PEP 580 is an alternative approach to solving the same problem as this PEP.
Reference implementation
A draft implementation can be found at https://github.com/markshannon/cpython/tree/pep-576-minimal
Copyright
This document has been placed in the public domain.
Source: https://github.com/python/peps/blob/main/pep-0576.rst
Last modified: 2022-03-09 16:04:44 GMT