[C API] PEP 670: Convert macros to functions in the Python C API #89653

vstinner · 2021-10-15T17:43:26Z

BPO	45490
Nosy	@malemburg, @vstinner, @erlend-aasland
PRs	#29728 #31217 #31221
Files	macros-that-reuse-args.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-10-15.17:43:25.813>
labels = ['expert-C-API', '3.11']
title = '[C API]  PEP 670: Convert macros to functions in the Python C API'
updated_at = <Date 2022-02-11.16:01:26.957>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-02-11.16:01:26.957>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['C API']
creation = <Date 2021-10-15.17:43:25.813>
creator = 'vstinner'
dependencies = []
files = ['50616']
hgrepos = []
issue_num = 45490
keywords = ['patch']
message_count = 7.0
messages = ['404038', '404045', '404183', '404185', '412861', '412995', '413079']
nosy_count = 3.0
nosy_names = ['lemburg', 'vstinner', 'erlendaasland']
pr_nums = ['29728', '31217', '31221']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue45490'
versions = ['Python 3.11']

vstinner · 2021-10-15T17:43:25Z

C macros are really cool and useful, but there are a bunch of pitfalls which are better to avoid:
https://gcc.gnu.org/onlinedocs/cpp/Macro-Pitfalls.html

Some macros of the Python C API have been converted to static inline functions over the last years. It went smoothly, I am not aware of any major issue with these conversions.

This meta issue tracks other issues related to macros and static inline functions.

=== Return void ===

One issue is that some macros are treated as an expression and can be reused, whereas it was not intended. For example PyList_SET_ITEM() was implemented as (simplified code):

  #define PyList_SET_ITEM(op, i, v) (op->ob_item[i] = v)

This expression has a value! Two projects used this value by mistake, like:

"if (obj == NULL || PyList_SET_ITEM (l, i, obj) < 0)"

PyList_SET_ITEM() was fixed by casting the expression to void:

  #define PyList_SET_ITEM(op, i, v) ((void)(op->ob_item[i] = v))

=> bpo-30459

=== Abuse macros as an l-value ===

The Py_TYPE() macro could be used to assign a value: "Py_TYPE(obj) = new_type".

The macro was defined as:

  #define Py_TYPE(ob) (ob->ob_type)

It was converted to a static inline function to disallow using it as an l-value and a new Py_SET_TYPE(op, new_type) function was added. These changes give more freedom to other Python implementations to implement "PyObject" and Py_SET_TYPE().

=> bpo-45476 "[C API] Disallow using PyFloat_AS_DOUBLE() as l-value"
=> bpo-39573 PyObject Py_TYPE/Py_SET_TYPE

=== C API: Macros and embedded Python ===

Sadly, only symbols exported by libpython are accessible to other programming languages embedding Python. Macros of the Python C API are simply not available to them. Projects embedding Python have to hardcode constants and copy macros to their own language, with the risk of being outdated when Python macros are updated.

Even some projects written in C cannot use macros, because they only use libpython symbols. The vim text editor embeds Python this way.

Also, macros are simply excluded from the Python stable ABI (PEP-384).

=== Performance of static inline functions ===

In bpo-45116, it seems like _PyEval_EvalFrameDefault() reached Visual Studio thresholds and some static inline functions are no longer inlined (Py_INCREF/Py_DECREF).

I also noticed that when Python is built in debug mode in Visual Studio, static inline functions are not inlined. Well, the compiler is free to not inline in debug mode. I guess that GCC and clang also skip inlining using -Og and/or -O0 optimization levels. Using __forceinline and __attribute__((always_inline)) on static inline functions (Py_INCREF, Py_TYPE) for debug builds was discussed in bpo-45094, but the idea was rejected.

On the other side, sometimes it's better to *disable* inlining on purpose to reduce the stack memory consumption, using the Py_NO_INLINE macro. See recent measurements of the stack memory usage:
https://bugs.python.org/issue45439#msg403768

In #73079, I noticed that converting a static inline function (PyObject_CallOneArg) to a regular function made it faster. I am not really sure, more benchmarks should be run to really what's going on.

=== Advantages of static inline functions ===

It's possible to put a breakpoint on a static inline functions.
Debuggers and profilers are able to get the static inline function names from the machine line, even with inline functions.
Parameters and the return value have well defined types.
Variables have a local scope.
There is no risk of evaluating an expression multiple times.
Regular C code. No need to use "\" character to multi-line statement. No need for "do { ... } while (0)" and other quicks to workaround preprocessor pitfalls. No abuse of (((parenthesis))).

malemburg · 2021-10-15T19:18:45Z

Meta comment :-) ... wouldn't it be better to enable the Github wiki feature for
such collections ?

erlend-aasland · 2021-10-18T13:17:24Z

+1!

See also bpo-43502

erlend-aasland · 2021-10-18T13:18:51Z

Previous discussion on Discourse:

https://discuss.python.org/t/what-to-do-with-unsafe-macros/7771

vstinner · 2022-02-08T19:15:41Z

I will use this issue to track changes related to PEP-670.

erlend-aasland · 2022-02-10T12:50:37Z

I made a list of macros that reuse their argument some time around February/March 2021. (Each macro is squashed into a single line for some reason I can't remember.) See attachment, or check out the gist version:

https://gist.github.com/erlend-aasland/a7ca3cff95b136e272ff5b03447aff21

vstinner · 2022-02-11T16:01:27Z

New changeset e0bcfd0 by Victor Stinner in branch 'main':
bpo-45490: Rename static inline functions (GH-31217)
e0bcfd0

vstinner · 2022-04-19T06:05:35Z

Update: the SC approved PEP 670: https://mail.python.org/archives/list/python-dev@python.org/thread/QQFCJ7LR36RUZSC3WI6WZZMQVQ3ZI4MS/

…ions * Convert unicodeobject.h macros to static inline functions. * Reorder functions to declare functions before their first usage. * Add "kind" variable to PyUnicode_READ_CHAR() and PyUnicode_MAX_CHAR_VALUE() functions to only call PyUnicode_KIND() once. * PyUnicode_KIND() return type is "unsigned int" (not "enum PyUnicode_Kind"). * Simplify PyUnicode_GET_SIZE(). * PyUnicode_READ_CHAR() now uses PyUnicode_1BYTE_DATA(), PyUnicode_2BYTE_DATA() and PyUnicode_4BYTE_DATA(). Assertions: * Remove redundant PyUnicode_Check() assertions. * Add assertions to PyUnicode_WRITE() on the max value. Part of PEP 670 implementation.

…ions * Convert unicodeobject.h macros to static inline functions. * Reorder functions to declare functions before their first usage. * PyUnicode_READ_CHAR() and PyUnicode_MAX_CHAR_VALUE() now only call PyUnicode_KIND() once. * PyUnicode_KIND() return type is "unsigned int" (not "enum PyUnicode_Kind"). * Simplify PyUnicode_GET_SIZE(). * PyUnicode_READ_CHAR() now uses PyUnicode_1BYTE_DATA(), PyUnicode_2BYTE_DATA() and PyUnicode_4BYTE_DATA(). Assertions: * Remove redundant PyUnicode_Check() assertions. * Add assertions to PyUnicode_WRITE() on the max value. Part of PEP 670 implementation.

…ions * Convert unicodeobject.h macros to static inline functions. * Reorder functions to declare functions before their first usage. * PyUnicode_READ_CHAR() and PyUnicode_MAX_CHAR_VALUE() now only call PyUnicode_KIND() once. * Simplify PyUnicode_GET_SIZE(). * PyUnicode_READ_CHAR() now uses PyUnicode_1BYTE_DATA(), PyUnicode_2BYTE_DATA() and PyUnicode_4BYTE_DATA(). * Remove redundant PyUnicode_Check() assertions. PyUnicode_KIND() return type is "unsigned int" rather than "enum PyUnicode_Kind" to prevent introducing new compiler warnings. Part of PEP 670 implementation.

* Convert unicodeobject.h macros to static inline functions. * Reorder functions to declare functions before their first usage. * PyUnicode_READ_CHAR() and PyUnicode_MAX_CHAR_VALUE() now only call PyUnicode_KIND() once. * Simplify PyUnicode_GET_SIZE(). * PyUnicode_READ_CHAR() now uses PyUnicode_1BYTE_DATA(), PyUnicode_2BYTE_DATA() and PyUnicode_4BYTE_DATA(). * Remove redundant PyUnicode_Check() assertions. PyUnicode_KIND() return type is "unsigned int" rather than "enum PyUnicode_Kind" to prevent introducing new compiler warnings.

Convert unicodeobject.h macros to static inline functions: * Reorder functions to declare functions before their first usage. * PyUnicode_READ_CHAR() and PyUnicode_MAX_CHAR_VALUE() now only call PyUnicode_KIND() once. * Simplify PyUnicode_GET_SIZE(). * PyUnicode_READ_CHAR() now uses PyUnicode_1BYTE_DATA(), PyUnicode_2BYTE_DATA() and PyUnicode_4BYTE_DATA(). * Remove redundant PyUnicode_Check() assertions. PyUnicode_KIND() return type is "unsigned int" rather than "enum PyUnicode_Kind" to prevent introducing new compiler warnings.

Convert unicodeobject.h macros to static inline functions: * Reorder functions to declare functions before their first usage. * PyUnicode_READ_CHAR() and PyUnicode_MAX_CHAR_VALUE() now only call PyUnicode_KIND() once. * Simplify PyUnicode_GET_SIZE(). * PyUnicode_READ_CHAR() now uses PyUnicode_1BYTE_DATA(), PyUnicode_2BYTE_DATA() and PyUnicode_4BYTE_DATA(). * Remove redundant PyUnicode_Check() assertions. Static inline functions are wrapped into macros which casts pointer types (PyObject*, void*) to prevent introducing new compiler warnings when passing const pointers (ex: PyUnicode_WRITE). PyUnicode_KIND() return type is "unsigned int" rather than "enum PyUnicode_Kind" to prevent introducing new compiler warnings.

In the limited C API version 3.11 and newer, the following functions no longer cast their object pointer argument with _PyObject_CAST() or _PyObject_CAST_CONST(): * Py_REFCNT(), Py_TYPE(), Py_SIZE() * Py_SET_REFCNT(), Py_SET_TYPE(), Py_SET_SIZE() * Py_IS_TYPE() * Py_INCREF(), Py_DECREF() * Py_XINCREF(), Py_XDECREF() * Py_NewRef(), Py_XNewRef() * PyObject_TypeCheck() * PyType_Check() * PyType_CheckExact() Update the xxlimited.c extension, which uses the limited C API version 3.11, to pass PyObject* to these functions.

In the limited C API version 3.11 and newer, the following functions no longer cast their object pointer argument with _PyObject_CAST() or _PyObject_CAST_CONST(): * Py_REFCNT(), Py_TYPE(), Py_SIZE() * Py_SET_REFCNT(), Py_SET_TYPE(), Py_SET_SIZE() * Py_IS_TYPE() * Py_INCREF(), Py_DECREF() * Py_XINCREF(), Py_XDECREF() * Py_NewRef(), Py_XNewRef() * PyObject_TypeCheck() * PyType_Check() * PyType_CheckExact() Split Py_DECREF() implementation in 3 versions to make the code more readable. Update the xxlimited.c extension, which uses the limited C API version 3.11, to pass PyObject* to these functions.

Convert unicodeobject.h macros to static inline functions: * PyUnicode_IS_COMPACT() * PyUnicode_IS_COMPACT_ASCII() * PyUnicode_CHECK_INTERNED() * PyUnicode_IS_READY() Changes: * Reorder functions to declare functions before their first usage. * Remove redundant PyUnicode_Check() assertions. Static inline functions are wrapped into macros which casts pointer types (PyObject*, void*) to prevent introducing new compiler warnings when passing const pointers (ex: PyUnicode_WRITE). PyUnicode_KIND() return type is "unsigned int" rather than "enum PyUnicode_Kind" to prevent introducing new compiler warnings.

Convert unicodeobject.h macros to static inline functions: * PyUnicode_CHECK_INTERNED() * PyUnicode_DATA() * PyUnicode_GET_LENGTH() * PyUnicode_IS_ASCII() * PyUnicode_IS_COMPACT() * PyUnicode_IS_COMPACT_ASCII() * PyUnicode_IS_READY() * _PyUnicode_COMPACT_DATA() * _PyUnicode_NONCOMPACT_DATA() Reorder functions to declare functions before their first usage. Static inline functions are wrapped into macros which casts "PyObject*" with _PyObject_CAST() to prevent introducing new compiler warnings when passing "const PyObject*".

Convert unicodeobject.h macros to static inline functions: * PyUnicode_CHECK_INTERNED() * PyUnicode_DATA(), _PyUnicode_COMPACT_DATA(), _PyUnicode_NONCOMPACT_DATA() * PyUnicode_GET_LENGTH() * PyUnicode_IS_ASCII() * PyUnicode_IS_COMPACT() * PyUnicode_IS_COMPACT_ASCII() * PyUnicode_IS_READY() Reorder functions to declare functions before their first usage. Static inline functions are wrapped into macros which casts "PyObject*" with _PyObject_CAST() to prevent introducing new compiler warnings when passing "const PyObject*".

Convert unicodeobject.h macros to static inline functions: * PyUnicode_MAX_CHAR_VALUE() * PyUnicode_READ() * PyUnicode_READY() * PyUnicode_READ_CHAR() * PyUnicode_WRITE() Move PyUnicode_READY() after _PyUnicode_Ready(), since it uses _PyUnicode_Ready(). Static inline functions are wrapped by macros which casts arguments with _PyObject_CAST() and casts 'kind' arguments to "unsigned int" to prevent introducing new compiler warnings when passing "const PyObject*".

Convert the PyWeakref_GET_OBJECT() macro to a static inline function. The function now checks its argument with assert(PyWeakref_Check(ref_obj)).

Convert the PyWeakref_GET_OBJECT() macro to a static inline function. Add an assertion to check the argument with PyWeakref_Check().

Convert the PyWeakref_GET_OBJECT() macro to a static inline function. Add an assertion to check the argument with PyWeakref_Check(). Add a macro converting the argument to PyObject* to prevent emitting new compiler warning.

Convert macros to static inline functions: * PyTuple_GET_SIZE() * PyTuple_SET_ITEM() * PyList_GET_SIZE() * PyList_SET_ITEM() Add a macro converting arguments to PyTupleObject*, PyListObject* or PyObject* to prevent emitting new compiler warnings.

Convert macros to static inline functions: * PyTuple_GET_SIZE() * PyTuple_SET_ITEM() * PyList_GET_SIZE() * PyList_SET_ITEM() Add a macro converting arguments to PyTupleObject*, PyListObject* or PyObject* to prevent emitting new compiler warnings. According to PEP 670, PyTuple_GET_ITEM() and PyList_GET_ITEM() are left as macros.

Convert the PyWeakref_GET_OBJECT() macro to a static inline function. Add an assertion to check the argument with PyWeakref_Check(). Add a macro converting the argument to PyObject* to prevent emitting new compiler warning.

Convert macros to static inline functions: * PyTuple_GET_SIZE() * PyTuple_SET_ITEM() * PyList_GET_SIZE() * PyList_SET_ITEM() Add a macro converting arguments to PyTupleObject*, PyListObject* or PyObject* to prevent emitting new compiler warnings. According to PEP 670, PyTuple_GET_ITEM() and PyList_GET_ITEM() are left as macros.

Convert unicodeobject.h macros to static inline functions: * PyUnicode_MAX_CHAR_VALUE() * PyUnicode_READ() * PyUnicode_READY() * PyUnicode_READ_CHAR() * PyUnicode_WRITE() Move PyUnicode_READY() after _PyUnicode_Ready(), since it uses _PyUnicode_Ready(). Static inline functions are wrapped by macros which casts arguments with _PyObject_CAST() and casts 'kind' arguments to "unsigned int" to prevent introducing new compiler warnings when passing "const PyObject*".

Convert unicodeobject.h macros to static inline functions: * PyUnicode_AS_DATA() * PyUnicode_AS_UNICODE() * PyUnicode_GET_DATA_SIZE() * PyUnicode_GET_SIZE() Static inline functions are wrapped by macros which casts arguments with _PyObject_CAST() to prevent introducing new compiler warnings when passing "const PyObject*".

In the limited C API version 3.11 and newer, the following functions no longer cast their object pointer argument with _PyObject_CAST() or _PyObject_CAST_CONST(): * Py_REFCNT(), Py_TYPE(), Py_SIZE() * Py_SET_REFCNT(), Py_SET_TYPE(), Py_SET_SIZE() * Py_IS_TYPE() * Py_INCREF(), Py_DECREF() * Py_XINCREF(), Py_XDECREF() * Py_NewRef(), Py_XNewRef() * PyObject_TypeCheck() * PyType_Check() * PyType_CheckExact() Split Py_DECREF() implementation in 3 versions to make the code more readable. Update the xxlimited.c extension, which uses the limited C API version 3.11, to pass PyObject* to these functions.

vstinner · 2022-04-26T22:30:34Z

Merged commits:

commit 4e52c66: PEP 670: Convert unicodeobject.h macros to functions (#91705)
commit 1a2b282: PEP 670: Convert PyWeakref_GET_OBJECT() to function (#91785)
commit 2a5f171: PEP 670: Convert tuple macros to functions (#91786)
commit 128d624: PEP 670: Convert unicodeobject.h macros to functions (#91773)
commit c1474fa: PEP 670: Group deprecated API in unicodeobject.h (#91796)
commit 636ad7b: PEP 670: Convert unicodeobject.h macros to functions (#91799)
commit efe7fd4: Add assertions to unicodeobject.h functions (#91800)
commit 61381d7: PEP 670: Functions don't cast pointers (#91697)
commit 1218061: PEP 670: Amend docs (GH-91813)

See also #91768 which replaced const PyObject* with PyObject* and commit 9e146bb (gh-80527: Deprecate PEP 623 Unicode functions, #91801).

Convert the following macros to static inline functions: * PyByteArray_AS_STRING() * PyByteArray_GET_SIZE() * PyBytes_AS_STRING() * PyBytes_GET_SIZE() Limited C API version 3.11 no longer casts arguments. Add _PyBytes_CAST() and _PyByteArray_CAST() macros.

Convert the following macros to static inline functions: * PyCFunction_GET_CLASS() * PyCFunction_GET_FLAGS() * PyCFunction_GET_FUNCTION() * PyCFunction_GET_SELF() Limited C API version 3.11 no longer casts arguments. Export PyCMethod_GetClass() method.

Convert the following macros to static inline functions: * PyCFunction_GET_CLASS() * PyCFunction_GET_FLAGS() * PyCFunction_GET_FUNCTION() * PyCFunction_GET_SELF() Limited C API version 3.11 no longer casts arguments. Export the PyCMethod_GetClass() function.

Convert the following macros to static inline functions: * PyCFunction_GET_CLASS() * PyCFunction_GET_FLAGS() * PyCFunction_GET_FUNCTION() * PyCFunction_GET_SELF() Limited C API version 3.11 no longer casts arguments.

vstinner added 3.11 expert-C-API labels Oct 15, 2021

vstinner changed the title ~~[meta][C API] Avoid C macro pitfalls and usage of static inline functions~~ [C API] PEP 670: Convert macros to functions in the Python C API Feb 8, 2022

ezio-melotti transferred this issue from another repository Apr 10, 2022

erlend-aasland mentioned this issue Apr 13, 2022

PEP 670: Mark as Accepted python/peps#2522

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Apr 21, 2022

pythongh-89653: PEP 670: Convert PyWeakref_GET_OBJECT() to function

ca96bad

Convert the PyWeakref_GET_OBJECT() macro to a static inline function. Add an assertion to check the argument with PyWeakref_Check().

vstinner added a commit to vstinner/cpython that referenced this issue Apr 21, 2022

pythongh-89653: PEP 670: Group deprecated API

c060a4e

vstinner added a commit to vstinner/cpython that referenced this issue Apr 21, 2022

pythongh-89653: PEP 670: Group deprecated API in unicodeobject.h

dc81080

vstinner added a commit that referenced this issue Apr 21, 2022

gh-89653: PEP 670: Group deprecated API in unicodeobject.h (#91796)

c1474fa

vstinner added a commit to vstinner/cpython that referenced this issue Apr 21, 2022

pythongh-89653: Add assertions to unicodeobject.h functions

dbaf24e

vstinner added a commit that referenced this issue Apr 21, 2022

gh-89653: Add assertions to unicodeobject.h functions (#91800)

efe7fd4

vstinner pushed a commit that referenced this issue Apr 22, 2022

gh-89653: PEP 670: Amend docs (GH-91813)

1218061

python / cpython Public

[C API] PEP 670: Convert macros to functions in the Python C API #89653

[C API] PEP 670: Convert macros to functions in the Python C API #89653

vstinner commented Oct 15, 2021

vstinner commented Oct 15, 2021

malemburg commented Oct 15, 2021

erlend-aasland commented Oct 18, 2021

erlend-aasland commented Oct 18, 2021

vstinner commented Feb 8, 2022

erlend-aasland commented Feb 10, 2022

vstinner commented Feb 11, 2022

vstinner commented Apr 19, 2022

vstinner commented Apr 26, 2022

python / cpython Public

[C API] PEP 670: Convert macros to functions in the Python C API #89653

[C API] PEP 670: Convert macros to functions in the Python C API #89653

Comments

vstinner commented Oct 15, 2021

vstinner commented Oct 15, 2021

malemburg commented Oct 15, 2021

erlend-aasland commented Oct 18, 2021

erlend-aasland commented Oct 18, 2021

vstinner commented Feb 8, 2022

erlend-aasland commented Feb 10, 2022

vstinner commented Feb 11, 2022

vstinner commented Apr 19, 2022

vstinner commented Apr 26, 2022