Expanding PyCodeObject

One field is to be added to the PyCodeObject struct [3]:

typedef struct {
   ...
   void *co_extra;  /* "Scratch space" for the code object. */
} PyCodeObject;

The co_extra will be NULL by default and only filled in as needed. Values stored in the field are expected to not be required in order for the code object to function, allowing the loss of the data of the field to be acceptable.

A private API has been introduced to work with the field:

PyAPI_FUNC(Py_ssize_t) _PyEval_RequestCodeExtraIndex(freefunc);
PyAPI_FUNC(int) _PyCode_GetExtra(PyObject *code, Py_ssize_t index,
                                 void **extra);
PyAPI_FUNC(int) _PyCode_SetExtra(PyObject *code, Py_ssize_t index,
                                 void *extra);

Users of the field are expected to call _PyEval_RequestCodeExtraIndex() to receive (what should be considered) an opaque index value to adding data into co-extra. With that index, users can set data using _PyCode_SetExtra() and later retrieve the data with _PyCode_GetExtra(). The API is purposefully listed as private to communicate the fact that there are no semantic guarantees of the API between Python releases.

Using a list and tuple were considered but was found to be less performant, and with a key use-case being JIT usage the performance consideration won out for using a custom struct instead of a Python object.

A dict was also considered, but once again performance was more important. While a dict will have constant overhead in looking up data, the overhead for the common case of a single object being stored in the data structure leads to a tuple having better performance characteristics (i.e. iterating a tuple of length 1 is faster than the overhead of hashing and looking up an object in a dict).

Expanding PyInterpreterState

The entrypoint for the frame evaluation function is per-interpreter:

// Same type signature as PyEval_EvalFrameEx().
typedef PyObject* (*_PyFrameEvalFunction)(PyFrameObject*, int);

typedef struct {
    ...
    _PyFrameEvalFunction eval_frame;
} PyInterpreterState;

By default, the eval_frame field will be initialized to a function pointer that represents what PyEval_EvalFrameEx() currently is (called _PyEval_EvalFrameDefault(), discussed later in this PEP). Third-party code may then set their own frame evaluation function instead to control the execution of Python code. A pointer comparison can be used to detect if the field is set to _PyEval_EvalFrameDefault() and thus has not been mutated yet.

Changes to Python/ceval.c

PyEval_EvalFrameEx() [5] as it currently stands will be renamed to _PyEval_EvalFrameDefault(). The new PyEval_EvalFrameEx() will then become:

PyObject *
PyEval_EvalFrameEx(PyFrameObject *frame, int throwflag)
{
    PyThreadState *tstate = PyThreadState_GET();
    return tstate->interp->eval_frame(frame, throwflag);
}

This allows third-party code to place themselves directly in the path of Python code execution while being backwards-compatible with code already using the pre-existing C API.

Updating python-gdb.py

The generated python-gdb.py file used for Python support in GDB makes some hard-coded assumptions about PyEval_EvalFrameEx(), e.g. the names of local variables. It will need to be updated to work with the proposed changes.

A JIT for CPython

def eval_frame(frame, throw_flag):
    pyjion_code = frame.code.co_extra
    if not pyjion_code:
        frame.code.co_extra = PyjionJittedCode()
    elif not pyjion_code.jit_failed:
        if not pyjion_code.jit_code:
            return pyjion_code.eval(pyjion_code.jit_code, frame)
        elif pyjion_code.exec_count > 20_000:
            if jit_compile(frame):
                return pyjion_code.eval(pyjion_code.jit_code, frame)
            else:
                pyjion_code.jit_failed = True
    pyjion_code.exec_count += 1
    return _PyEval_EvalFrameDefault(frame, throw_flag)

Debugging

In conversations with the Python Tools for Visual Studio team (PTVS) [12], they thought they would find these API changes useful for implementing more performant debugging. As mentioned in the Rationale section, this API would allow for switching on debugging functionality only in frames where it is needed. This could allow for either skipping information that sys.settrace() normally provides and even go as far as to dynamically rewrite bytecode prior to execution to inject e.g. breakpoints in the bytecode.

It also turns out that Google provides a very similar API internally. It has been used for performant debugging purposes.

Allow eval_frame to be NULL

Currently the frame evaluation function is expected to always be set. It could very easily simply default to NULL instead which would signal to use _PyEval_EvalFrameDefault(). The current proposal of not special-casing the field seemed the most straightforward, but it does require that the field not accidentally be cleared, else a crash may occur.

A JIT-specific C API

Originally this PEP was going to propose a much larger API change which was more JIT-specific. After soliciting feedback from the Numba team [6], though, it became clear that the API was unnecessarily large. The realization was made that all that was truly needed was the opportunity to provide a trampoline function to handle execution of Python code that had been JIT-compiled and a way to attach that compiled machine code along with other critical data to the corresponding Python code object. Once it was shown that there was no loss in functionality or in performance while minimizing the API changes required, the proposal was changed to its current form.

Is co_extra needed?

While discussing this PEP at PyCon US 2016, some core developers expressed their worry of the co_extra field making code objects mutable. The thinking seemed to be that having a field that was mutated after the creation of the code object made the object seem mutable, even though no other aspect of code objects changed.

The view of this PEP is that the co_extra field doesn’t change the fact that code objects are immutable. The field is specified in this PEP to not contain information required to make the code object usable, making it more of a caching field. It could be viewed as similar to the UTF-8 cache that string objects have internally; strings are still considered immutable even though they have a field that is conditionally set.

Performance measurements were also made where the field was not available for JIT workloads. The loss of the field was deemed too costly to performance when using an unordered map from C++ or Python’s dict to associated a code object with JIT-specific data objects.