Abstract

The Python C-API currently has no mechanism for specifying and auto-generating function signatures, annotations or custom argument converters.

There are several possible approaches to the problem. Cython uses cdef definitions in .pyx files to generate the required information. However, CPython’s C-API functions often require additional initialization and cleanup snippets that would be hard to specify in a cdef.

PEP 436 proposes a domain specific language (DSL) enclosed in C comments that largely resembles a per-parameter configuration file. A preprocessor reads the comment and emits an argument parsing function, docstrings and a header for the function that utilizes the results of the parsing step.

The latter function is subsequently referred to as the implementation function.

Rejection Notice

This PEP was rejected by Guido van Rossum at PyCon US 2013. However, several of the specific issues raised by this PEP were taken into account when designing the second iteration of the PEP 436 DSL.

Rationale

Opinions differ regarding the suitability of the PEP 436 DSL in the context of a C file. This PEP proposes an alternative DSL. The specific issues with PEP 436 that spurred the counter proposal will be explained in the final section of this PEP.

Scope

The PEP focuses exclusively on the DSL. Topics like the output locations of docstrings or the generated code are outside the scope of this PEP.

It is however vital that the DSL is suitable for generating custom argument parsers, a feature that is already implemented in Cython. Therefore, one of the goals of this PEP is to keep the DSL close to existing solutions, thus facilitating a possible inclusion of the relevant parts of Cython into the CPython source tree.

DSL overview

/*[converter]
##### Default converters #####
"s":  str                                -> const char *res;
"s*": [str, bytes, bytearray, rw_buffer] -> Py_buffer &res;
[...]
"es#": str -> (const char *res_encoding, char **res, Py_ssize_t *res_length);
[...]
##### Custom converters #####
path_converter:           [str, bytes, int]  -> path_t &res;
OS_STAT_DIR_FD_CONVERTER: [int, None]        -> int res;
[converter_end]*/

/*[define posix_stat]
def os.stat(path: path_converter, *, dir_fd: OS_STAT_DIR_FD_CONVERTER = None,
            follow_symlinks: "p" = True) -> os.stat_result: pass
%%
path_t path = PATH_T_INITIALIZE("stat", 0, 1);
int dir_fd = DEFAULT_DIR_FD;
int follow_symlinks = 1;
%%
path_cleanup(&path);
[define_end]*/

<literal C output>

/*[define_output_end]*/

/*[define stat_float_times]
def os.stat_float_times(/, newval: "i") -> os.stat_result: pass
%%
int newval = -1;
[define_end]*/

/*[define]
def curses.window.addch(y: "i", x: "i", ch: "O", attr: "l") -> None: pass
where groups = [[ch], [ch, attr], [y, x, ch], [y, x, ch, attr]]
[define_end]*/

Flexibility in formatting

If the above os.stat example is considered too compact, it can easily be formatted this way:

/*[define posix_stat]
def os.stat(path: path_converter,
            *,
            dir_fd: OS_STAT_DIR_FD_CONVERTER = None,
            follow_symlinks: "p" = True)
-> os.stat_result: pass
%%
path_t path = PATH_T_INITIALIZE("stat", 0, 1);
int dir_fd = DEFAULT_DIR_FD;
int follow_symlinks = 1;
%%
path_cleanup(&path);
[define_end]*/

<literal C output>

/*[define_output_end]*/

Benefits of a compact notation

The advantages of a concise notation are especially obvious when a large number of parameters is involved. The argument parsing part of _posixsubprocess.fork_exec is fully specified by this definition:

/*[define subprocess_fork_exec]
def _posixsubprocess.fork_exec(
    process_args: "O", executable_list: "O",
    close_fds: "p", py_fds_to_keep: "O",
    cwd_obj: "O", env_list: "O",
    p2cread: "i", p2cwrite: "i", c2pread: "i", c2pwrite: "i",
    errread: "i", errwrite: "i", errpipe_read: "i", errpipe_write: "i",
    restore_signals: "i", call_setsid: "i", preexec_fn: "i", /) -> int: pass
[define_end]*/

Note that the preprocess tool currently emits a redundant C-declaration section for this example, so the output is longer than necessary.

Easy validation of the definition

How can an inexperienced user validate a definition like os.stat? Simply by changing os.stat to os_stat, defining missing converters and pasting the definition into the Python interactive interpreter!

In fact, a converters.py module could be auto-generated from converters.h.

Reference implementation

A reference implementation is available at issue 16612. Since this PEP was written under time constraints and the author is unfamiliar with the PLY toolchain, the software is written in Standard ML and utilizes the ml-yacc/ml-lex toolchain.

The grammar is conflict-free and available in ml-yacc readable BNF form.

Two tools are available:

• printsemant reads a converter header and a .c file and dumps the semantically checked parse tree to stdout.
• preprocess reads a converter header and a .c file and dumps the preprocessed .c file to stdout.

Known deficiencies:

• The Python ‘test’ expression is not semantically checked. The syntax however is checked since it is part of the grammar.
• The lexer does not handle triple quoted strings.
• C declarations are parsed in a primitive way. The final implementation should utilize ‘declarator’ and ‘init-declarator’ from the C grammar.
• The preprocess tool does not emit code for the left-and-right optional arguments case. The printsemant tool can deal with this case.
• Since the preprocess tool generates the output from the parse tree, the original indentation of the define block is lost.

Grammar

TBD: The grammar exists in ml-yacc readable form, but should probably be included here in EBNF notation.

Comparison with PEP 436

The author of this PEP has the following concerns about the DSL proposed in PEP 436:

• The whitespace sensitive configuration file like syntax looks out of place in a C file.
• The structure of the function definition gets lost in the per-parameter specifications. Keywords like positional-only, required and keyword-only are scattered across too many different places.

  By contrast, in the alternative DSL the structure of the function definition can be understood at a single glance.

• The PEP 436 DSL has 14 documented flags and at least one undocumented (allow_fd) flag. Figuring out which of the 2**15 possible combinations are valid places an unnecessary burden on the user.

  Experience with the PEP 3118 buffer flags has shown that sorting out (and exhaustively testing!) valid combinations is an extremely tedious task. The PEP 3118 flags are still not well understood by many people.

  By contrast, the alternative DSL has a central file Include/converters.h that can be quickly searched for the desired converter. Many of the converters are already known, perhaps even memorized by people (due to frequent use).

• The PEP 436 DSL allows too much freedom. Types can apparently be omitted, the preprocessor accepts (and ignores) unknown keywords, sometimes adding white space after a docstring results in an assertion error.

  The alternative DSL on the other hand allows no such freedoms. Omitting converter or return value annotations is plainly a syntax error. The LALR(1) grammar is unambiguous and specified for the complete translation unit.

Copyright

This document is licensed under the Open Publication License.

References and Footnotes