[C API] PEP 670: Convert macros to functions in the Python C API #89653

vstinner · 2021-10-15T17:43:26Z

BPO	45490
Nosy	@malemburg, @vstinner, @erlend-aasland
PRs	#29728 #31217 #31221
Files	macros-that-reuse-args.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-10-15.17:43:25.813>
labels = ['expert-C-API', '3.11']
title = '[C API]  PEP 670: Convert macros to functions in the Python C API'
updated_at = <Date 2022-02-11.16:01:26.957>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-02-11.16:01:26.957>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['C API']
creation = <Date 2021-10-15.17:43:25.813>
creator = 'vstinner'
dependencies = []
files = ['50616']
hgrepos = []
issue_num = 45490
keywords = ['patch']
message_count = 7.0
messages = ['404038', '404045', '404183', '404185', '412861', '412995', '413079']
nosy_count = 3.0
nosy_names = ['lemburg', 'vstinner', 'erlendaasland']
pr_nums = ['29728', '31217', '31221']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue45490'
versions = ['Python 3.11']

vstinner · 2021-10-15T17:43:25Z

C macros are really cool and useful, but there are a bunch of pitfalls which are better to avoid:
https://gcc.gnu.org/onlinedocs/cpp/Macro-Pitfalls.html

Some macros of the Python C API have been converted to static inline functions over the last years. It went smoothly, I am not aware of any major issue with these conversions.

This meta issue tracks other issues related to macros and static inline functions.

=== Return void ===

One issue is that some macros are treated as an expression and can be reused, whereas it was not intended. For example PyList_SET_ITEM() was implemented as (simplified code):

  #define PyList_SET_ITEM(op, i, v) (op->ob_item[i] = v)

This expression has a value! Two projects used this value by mistake, like:

"if (obj == NULL || PyList_SET_ITEM (l, i, obj) < 0)"

PyList_SET_ITEM() was fixed by casting the expression to void:

  #define PyList_SET_ITEM(op, i, v) ((void)(op->ob_item[i] = v))

=> bpo-30459

=== Abuse macros as an l-value ===

The Py_TYPE() macro could be used to assign a value: "Py_TYPE(obj) = new_type".

The macro was defined as:

  #define Py_TYPE(ob) (ob->ob_type)

It was converted to a static inline function to disallow using it as an l-value and a new Py_SET_TYPE(op, new_type) function was added. These changes give more freedom to other Python implementations to implement "PyObject" and Py_SET_TYPE().

=> bpo-45476 "[C API] Disallow using PyFloat_AS_DOUBLE() as l-value"
=> bpo-39573 PyObject Py_TYPE/Py_SET_TYPE

=== C API: Macros and embedded Python ===

Sadly, only symbols exported by libpython are accessible to other programming languages embedding Python. Macros of the Python C API are simply not available to them. Projects embedding Python have to hardcode constants and copy macros to their own language, with the risk of being outdated when Python macros are updated.

Even some projects written in C cannot use macros, because they only use libpython symbols. The vim text editor embeds Python this way.

Also, macros are simply excluded from the Python stable ABI (PEP-384).

=== Performance of static inline functions ===

In bpo-45116, it seems like _PyEval_EvalFrameDefault() reached Visual Studio thresholds and some static inline functions are no longer inlined (Py_INCREF/Py_DECREF).

I also noticed that when Python is built in debug mode in Visual Studio, static inline functions are not inlined. Well, the compiler is free to not inline in debug mode. I guess that GCC and clang also skip inlining using -Og and/or -O0 optimization levels. Using __forceinline and __attribute__((always_inline)) on static inline functions (Py_INCREF, Py_TYPE) for debug builds was discussed in bpo-45094, but the idea was rejected.

On the other side, sometimes it's better to *disable* inlining on purpose to reduce the stack memory consumption, using the Py_NO_INLINE macro. See recent measurements of the stack memory usage:
https://bugs.python.org/issue45439#msg403768

In #73079, I noticed that converting a static inline function (PyObject_CallOneArg) to a regular function made it faster. I am not really sure, more benchmarks should be run to really what's going on.

=== Advantages of static inline functions ===

It's possible to put a breakpoint on a static inline functions.
Debuggers and profilers are able to get the static inline function names from the machine line, even with inline functions.
Parameters and the return value have well defined types.
Variables have a local scope.
There is no risk of evaluating an expression multiple times.
Regular C code. No need to use "\" character to multi-line statement. No need for "do { ... } while (0)" and other quicks to workaround preprocessor pitfalls. No abuse of (((parenthesis))).

malemburg · 2021-10-15T19:18:45Z

Meta comment :-) ... wouldn't it be better to enable the Github wiki feature for
such collections ?

erlend-aasland · 2021-10-18T13:17:24Z

+1!

See also bpo-43502

erlend-aasland · 2021-10-18T13:18:51Z

Previous discussion on Discourse:

https://discuss.python.org/t/what-to-do-with-unsafe-macros/7771

vstinner · 2022-02-08T19:15:41Z

I will use this issue to track changes related to PEP-670.

erlend-aasland · 2022-02-10T12:50:37Z

I made a list of macros that reuse their argument some time around February/March 2021. (Each macro is squashed into a single line for some reason I can't remember.) See attachment, or check out the gist version:

https://gist.github.com/erlend-aasland/a7ca3cff95b136e272ff5b03447aff21

vstinner · 2022-02-11T16:01:27Z

New changeset e0bcfd0 by Victor Stinner in branch 'main':
bpo-45490: Rename static inline functions (GH-31217)
e0bcfd0

vstinner · 2022-04-19T06:05:35Z

Update: the SC approved PEP 670: https://mail.python.org/archives/list/python-dev@python.org/thread/QQFCJ7LR36RUZSC3WI6WZZMQVQ3ZI4MS/

…ions * Convert unicodeobject.h macros to static inline functions. * Reorder functions to declare functions before their first usage. * Add "kind" variable to PyUnicode_READ_CHAR() and PyUnicode_MAX_CHAR_VALUE() functions to only call PyUnicode_KIND() once. * PyUnicode_KIND() return type is "unsigned int" (not "enum PyUnicode_Kind"). * Simplify PyUnicode_GET_SIZE(). * PyUnicode_READ_CHAR() now uses PyUnicode_1BYTE_DATA(), PyUnicode_2BYTE_DATA() and PyUnicode_4BYTE_DATA(). Assertions: * Remove redundant PyUnicode_Check() assertions. * Add assertions to PyUnicode_WRITE() on the max value. Part of PEP 670 implementation.

…ions * Convert unicodeobject.h macros to static inline functions. * Reorder functions to declare functions before their first usage. * PyUnicode_READ_CHAR() and PyUnicode_MAX_CHAR_VALUE() now only call PyUnicode_KIND() once. * PyUnicode_KIND() return type is "unsigned int" (not "enum PyUnicode_Kind"). * Simplify PyUnicode_GET_SIZE(). * PyUnicode_READ_CHAR() now uses PyUnicode_1BYTE_DATA(), PyUnicode_2BYTE_DATA() and PyUnicode_4BYTE_DATA(). Assertions: * Remove redundant PyUnicode_Check() assertions. * Add assertions to PyUnicode_WRITE() on the max value. Part of PEP 670 implementation.

…ions * Convert unicodeobject.h macros to static inline functions. * Reorder functions to declare functions before their first usage. * PyUnicode_READ_CHAR() and PyUnicode_MAX_CHAR_VALUE() now only call PyUnicode_KIND() once. * Simplify PyUnicode_GET_SIZE(). * PyUnicode_READ_CHAR() now uses PyUnicode_1BYTE_DATA(), PyUnicode_2BYTE_DATA() and PyUnicode_4BYTE_DATA(). * Remove redundant PyUnicode_Check() assertions. PyUnicode_KIND() return type is "unsigned int" rather than "enum PyUnicode_Kind" to prevent introducing new compiler warnings. Part of PEP 670 implementation.

* Convert unicodeobject.h macros to static inline functions. * Reorder functions to declare functions before their first usage. * PyUnicode_READ_CHAR() and PyUnicode_MAX_CHAR_VALUE() now only call PyUnicode_KIND() once. * Simplify PyUnicode_GET_SIZE(). * PyUnicode_READ_CHAR() now uses PyUnicode_1BYTE_DATA(), PyUnicode_2BYTE_DATA() and PyUnicode_4BYTE_DATA(). * Remove redundant PyUnicode_Check() assertions. PyUnicode_KIND() return type is "unsigned int" rather than "enum PyUnicode_Kind" to prevent introducing new compiler warnings.

Convert unicodeobject.h macros to static inline functions: * Reorder functions to declare functions before their first usage. * PyUnicode_READ_CHAR() and PyUnicode_MAX_CHAR_VALUE() now only call PyUnicode_KIND() once. * Simplify PyUnicode_GET_SIZE(). * PyUnicode_READ_CHAR() now uses PyUnicode_1BYTE_DATA(), PyUnicode_2BYTE_DATA() and PyUnicode_4BYTE_DATA(). * Remove redundant PyUnicode_Check() assertions. PyUnicode_KIND() return type is "unsigned int" rather than "enum PyUnicode_Kind" to prevent introducing new compiler warnings.

Convert unicodeobject.h macros to static inline functions: * Reorder functions to declare functions before their first usage. * PyUnicode_READ_CHAR() and PyUnicode_MAX_CHAR_VALUE() now only call PyUnicode_KIND() once. * Simplify PyUnicode_GET_SIZE(). * PyUnicode_READ_CHAR() now uses PyUnicode_1BYTE_DATA(), PyUnicode_2BYTE_DATA() and PyUnicode_4BYTE_DATA(). * Remove redundant PyUnicode_Check() assertions. Static inline functions are wrapped into macros which casts pointer types (PyObject*, void*) to prevent introducing new compiler warnings when passing const pointers (ex: PyUnicode_WRITE). PyUnicode_KIND() return type is "unsigned int" rather than "enum PyUnicode_Kind" to prevent introducing new compiler warnings.

In the limited C API version 3.11 and newer, the following functions no longer cast their object pointer argument with _PyObject_CAST() or _PyObject_CAST_CONST(): * Py_REFCNT(), Py_TYPE(), Py_SIZE() * Py_SET_REFCNT(), Py_SET_TYPE(), Py_SET_SIZE() * Py_IS_TYPE() * Py_INCREF(), Py_DECREF() * Py_XINCREF(), Py_XDECREF() * Py_NewRef(), Py_XNewRef() * PyObject_TypeCheck() * PyType_Check() * PyType_CheckExact() Update the xxlimited.c extension, which uses the limited C API version 3.11, to pass PyObject* to these functions.

In the limited C API version 3.11 and newer, the following functions no longer cast their object pointer argument with _PyObject_CAST() or _PyObject_CAST_CONST(): * Py_REFCNT(), Py_TYPE(), Py_SIZE() * Py_SET_REFCNT(), Py_SET_TYPE(), Py_SET_SIZE() * Py_IS_TYPE() * Py_INCREF(), Py_DECREF() * Py_XINCREF(), Py_XDECREF() * Py_NewRef(), Py_XNewRef() * PyObject_TypeCheck() * PyType_Check() * PyType_CheckExact() Split Py_DECREF() implementation in 3 versions to make the code more readable. Update the xxlimited.c extension, which uses the limited C API version 3.11, to pass PyObject* to these functions.

Convert unicodeobject.h macros to static inline functions: * PyUnicode_IS_COMPACT() * PyUnicode_IS_COMPACT_ASCII() * PyUnicode_CHECK_INTERNED() * PyUnicode_IS_READY() Changes: * Reorder functions to declare functions before their first usage. * Remove redundant PyUnicode_Check() assertions. Static inline functions are wrapped into macros which casts pointer types (PyObject*, void*) to prevent introducing new compiler warnings when passing const pointers (ex: PyUnicode_WRITE). PyUnicode_KIND() return type is "unsigned int" rather than "enum PyUnicode_Kind" to prevent introducing new compiler warnings.

Convert unicodeobject.h macros to static inline functions: * PyUnicode_CHECK_INTERNED() * PyUnicode_DATA() * PyUnicode_GET_LENGTH() * PyUnicode_IS_ASCII() * PyUnicode_IS_COMPACT() * PyUnicode_IS_COMPACT_ASCII() * PyUnicode_IS_READY() * _PyUnicode_COMPACT_DATA() * _PyUnicode_NONCOMPACT_DATA() Reorder functions to declare functions before their first usage. Static inline functions are wrapped into macros which casts "PyObject*" with _PyObject_CAST() to prevent introducing new compiler warnings when passing "const PyObject*".

Convert unicodeobject.h macros to static inline functions: * PyUnicode_CHECK_INTERNED() * PyUnicode_DATA(), _PyUnicode_COMPACT_DATA(), _PyUnicode_NONCOMPACT_DATA() * PyUnicode_GET_LENGTH() * PyUnicode_IS_ASCII() * PyUnicode_IS_COMPACT() * PyUnicode_IS_COMPACT_ASCII() * PyUnicode_IS_READY() Reorder functions to declare functions before their first usage. Static inline functions are wrapped into macros which casts "PyObject*" with _PyObject_CAST() to prevent introducing new compiler warnings when passing "const PyObject*".

…-92654) The limited API version 3.11 no longer casts arguments to expected types of functions of functions: * PyList_GET_SIZE(), PyList_SET_ITEM() * PyTuple_GET_SIZE(), PyTuple_SET_ITEM() * PyWeakref_GET_OBJECT() (cherry picked from commit 7d3b469e475e6e52ce4f0bad7198bb05ead77b1d) Co-authored-by: Victor Stinner <vstinner@python.org>

Use the PyObject* type for parameters of static inline functions: * Py_SIZE(): same parameter type than PyObject_Size() * PyList_GET_SIZE(), PyList_SET_ITEM(): same parameter type than PyList_Size() and PyList_SetItem() * PyTuple_GET_SIZE(), PyTuple_SET_ITEM(): same parameter type than PyTuple_Size() and PyTuple_SetItem().

The limited API version 3.11 no longer casts arguments to expected types of functions of functions: * PyList_GET_SIZE(), PyList_SET_ITEM() * PyTuple_GET_SIZE(), PyTuple_SET_ITEM() * PyWeakref_GET_OBJECT() (cherry picked from commit 7d3b469) Co-authored-by: Victor Stinner <vstinner@python.org>

The limited C API version 3.12 no longer casts the argument.

Use _Py_CAST() and _Py_STATIC_CAST() in macros wrapping static inline functions of unicodeobject.h. Change also the kind type from unsigned int to int: same parameter type than PyUnicode_FromKindAndData().

Use _Py_CAST() and _Py_STATIC_CAST() in macros wrapping static inline functions of unicodeobject.h. Change also the kind type from unsigned int to int: same parameter type than PyUnicode_FromKindAndData(). The limited API version 3.11 no longer casts arguments to expected types.

Use the PyObject* type for parameters of static inline functions: * Py_SIZE(): same parameter type than PyObject_Size() * PyList_GET_SIZE(), PyList_SET_ITEM(): same parameter type than PyList_Size() and PyList_SetItem() * PyTuple_GET_SIZE(), PyTuple_SET_ITEM(): same parameter type than PyTuple_Size() and PyTuple_SetItem().

…92694) Use the PyObject* type for parameters of static inline functions: * Py_SIZE(): same parameter type than PyObject_Size() * PyList_GET_SIZE(), PyList_SET_ITEM(): same parameter type than PyList_Size() and PyList_SetItem() * PyTuple_GET_SIZE(), PyTuple_SET_ITEM(): same parameter type than PyTuple_Size() and PyTuple_SetItem(). (cherry picked from commit 6de78ef) Co-authored-by: Victor Stinner <vstinner@python.org>

The limited C API version 3.12 no longer casts the argument.

Use the PyObject* type for parameters of static inline functions: * Py_SIZE(): same parameter type than PyObject_Size() * PyList_GET_SIZE(), PyList_SET_ITEM(): same parameter type than PyList_Size() and PyList_SetItem() * PyTuple_GET_SIZE(), PyTuple_SET_ITEM(): same parameter type than PyTuple_Size() and PyTuple_SetItem(). (cherry picked from commit 6de78ef) Co-authored-by: Victor Stinner <vstinner@python.org>

Use _Py_CAST() and _Py_STATIC_CAST() in macros wrapping static inline functions of unicodeobject.h. Change also the kind type from unsigned int to int: same parameter type than PyUnicode_FromKindAndData(). The limited API version 3.11 no longer casts arguments to expected types.

) Use _Py_CAST() and _Py_STATIC_CAST() in macros wrapping static inline functions of unicodeobject.h. Change also the kind type from unsigned int to int: same parameter type than PyUnicode_FromKindAndData(). The limited API version 3.11 no longer casts arguments to expected types. (cherry picked from commit d0c9353a79c2003385c83892db5dfd4e443474c9) Co-authored-by: Victor Stinner <vstinner@python.org>

Use _Py_CAST() and _Py_STATIC_CAST() in macros wrapping static inline functions of unicodeobject.h. Change also the kind type from unsigned int to int: same parameter type than PyUnicode_FromKindAndData(). The limited API version 3.11 no longer casts arguments to expected types. (cherry picked from commit d0c9353) Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Victor Stinner <vstinner@python.org>

Use the same type that PyUnicode_FromKindAndData() kind parameter type (public C API): int.

In the limited C API version 3.12, PyUnicode_KIND() is now implemented as a static inline function. Keep the macro for the regular C API and for the limited C API version 3.11 and older to prevent introducing new compiler warnings. Update _decimal.c and stringlib/eq.h for stricter PyUnicode_KIND() even if the limited C API is not used by default.

In the limited C API version 3.12, PyUnicode_KIND() is now implemented as a static inline function. Keep the macro for the regular C API and for the limited C API version 3.11 and older to prevent introducing new compiler warnings. Update _decimal.c and stringlib/eq.h for stricter PyUnicode_KIND().

In the limited C API version 3.12, PyUnicode_KIND() is now implemented as a static inline function. Keep the macro for the regular C API and for the limited C API version 3.11 and older to prevent introducing new compiler warnings. Update _decimal.c and stringlib/eq.h for PyUnicode_KIND().

vstinner · 2022-05-12T00:50:44Z

Bugfixes in the 3.11 branch:

commit 6f92872: PEP 670: Limited API doesn't cast arguments (#92693)
commit a1bef8c: PEP 670: Use PyObject* type for parameters (#92698)
commit 6e1a214: PEP 670: unicodeobject.h uses _Py_CAST() (#92703)

vstinner · 2022-05-12T00:56:13Z

Merged commits:

commit 61381d7: Functions don't cast pointers (#91697)
commit 2eca5da: Convert PyBytes macros to functions (#91990)
commit 08b562a: Convert PyCFunction macros to functions (#92302)
commit ffcc7cd: Convert pycore_gc.h macros to functions (#92649)
commit 897f14d: Convert PyCell macros to functions (#92653)
commit eb88f21: Convert unicodeobject.h macros to functions (#92648)
commit 7d3b469: Limited API doesn't cast arguments (#92654)
commit 6de78ef: Use PyObject* type for parameters (#92694)
commit f9b67ad: Convert PyDict_GET_SIZE() macro to function (#92695)
commit 92f0ed1: Update C API unicode documentation (#92702)
commit d0c9353: unicodeobject.h uses _Py_CAST() (#92696)

vstinner added 3.11 expert-C-API labels Oct 15, 2021

vstinner changed the title ~~[meta][C API] Avoid C macro pitfalls and usage of static inline functions~~ [C API] PEP 670: Convert macros to functions in the Python C API Feb 8, 2022

ezio-melotti transferred this issue from another repository Apr 10, 2022

erlend-aasland mentioned this issue Apr 13, 2022

PEP 670: Mark as Accepted python/peps#2522

Merged

vstinner added a commit that referenced this issue May 11, 2022

gh-89653: Add assertions to unicodeobject.h functions (#92692)

d492f0a

vstinner added a commit to vstinner/cpython that referenced this issue May 11, 2022

pythongh-89653: PEP 670: Convert PyDict_GET_SIZE() macro to function

73fa06d

The limited C API version 3.12 no longer casts the argument.

vstinner added a commit that referenced this issue May 11, 2022

gh-89653: PEP 670: Convert PyDict_GET_SIZE() macro to function (#92695)

f9b67ad

The limited C API version 3.12 no longer casts the argument.

vstinner added a commit to vstinner/cpython that referenced this issue May 11, 2022

pythongh-89653: PEP 670: Update C API unicode documentation

206a09c

vstinner added a commit that referenced this issue May 11, 2022

gh-89653: PEP 670: Update C API unicode documentation (#92702)

92f0ed1

vstinner added a commit to vstinner/cpython that referenced this issue May 12, 2022

pythongh-89653: Use int type for Unicode kind

016d839

Use the same type that PyUnicode_FromKindAndData() kind parameter type (public C API): int.

vstinner added a commit to vstinner/cpython that referenced this issue May 12, 2022

pythongh-89653: Use int type for Unicode kind

4c6a437

Use the same type that PyUnicode_FromKindAndData() kind parameter type (public C API): int.

vstinner added a commit to vstinner/cpython that referenced this issue May 12, 2022

pythongh-89653: PEP 670: Fix Sphinx syntax in Unicode doc

04c4fda

vstinner added a commit that referenced this issue May 12, 2022

gh-89653: PEP 670: Fix Sphinx syntax in Unicode doc (#92707)

1a9645f

python / cpython Public

[C API] PEP 670: Convert macros to functions in the Python C API #89653

[C API] PEP 670: Convert macros to functions in the Python C API #89653

vstinner commented Oct 15, 2021

vstinner commented Oct 15, 2021

malemburg commented Oct 15, 2021

erlend-aasland commented Oct 18, 2021

erlend-aasland commented Oct 18, 2021

vstinner commented Feb 8, 2022

erlend-aasland commented Feb 10, 2022

vstinner commented Feb 11, 2022

vstinner commented Apr 19, 2022

vstinner commented May 12, 2022 •

edited

vstinner commented May 12, 2022

python / cpython Public

[C API] PEP 670: Convert macros to functions in the Python C API #89653

[C API] PEP 670: Convert macros to functions in the Python C API #89653

Comments

vstinner commented Oct 15, 2021

vstinner commented Oct 15, 2021

malemburg commented Oct 15, 2021

erlend-aasland commented Oct 18, 2021

erlend-aasland commented Oct 18, 2021

vstinner commented Feb 8, 2022

erlend-aasland commented Feb 10, 2022

vstinner commented Feb 11, 2022

vstinner commented Apr 19, 2022

vstinner commented May 12, 2022 • edited

vstinner commented May 12, 2022

vstinner commented May 12, 2022 •

edited