Skip to content

PyCode_Type.tp_basictype change in Python 3.11 broke Cython #93585

Closed
@vstinner

Description

@vstinner

PyCode_Type.tp_basicsize was changed in Python 3.11 by the commit 2bde682 (by @brandtbucher):

 PyTypeObject PyCode_Type = {
     PyVarObject_HEAD_INIT(&PyType_Type, 0)
     "code",
-    sizeof(PyCodeObject),
-    0,
+    offsetof(PyCodeObject, co_code_adaptive),
+    sizeof(_Py_CODEUNIT),
     (destructor)code_dealloc,           /* tp_dealloc */
     0,                                  /* tp_vectorcall_offset */
     0,                                  /* tp_getattr */

This change broke the gevent project which uses Cython to access the PyCodeObject structure: cython/cython#4827

Cython expects that PyCode_Type.tp_basicsize = sizeof(PyCodeObject) which was true until Python 3.10, but is false in Python 3.11.

Python defines 8 built-in types which use a PyVarObject. Python is inconsistent on tp_basicsize on these types (current values in the main branch):

  • PyByteArray_Type.tp_basicsize = sizeof(PyByteArrayObject). PyByteArrayObject has no flexible array but a char *ob_bytes; member.
  • PyBytes_Type.tp_basicsize = PyBytesObject_SIZE = (offsetof(PyBytesObject, ob_sval) + 1). PyBytesObject structure ends with the flexible array char ob_sval[1].
  • PyCode_Type.tp_basicsize = offsetof(PyCodeObject, co_code_adaptive). PyCodeObject structure ends with a flexible array: char co_code_adaptive[1].
  • PyLong_Type.tp_basicsize = offsetof(PyLongObject, ob_digit). PyLongObject structure ends with a flexible array: digit ob_digit[1].
  • PyType_Type.tp_basicsize = sizeof(PyHeapTypeObject). PyHeapTypeObject structure has no flexible array.
  • PyList_Type.tp_basicsize = sizeof(PyListObject). PyListObject structure has no flexible array but PyObject **ob_item.
  • PyTuple_Type.tp_basicsize = sizeof(PyTupleObject) - sizeof(PyObject *). PyTupleObject structure ends with a flexible array: PyObject *ob_item[1].
  • PyMemoryView_Type.tp_basicsize = offsetof(PyMemoryViewObject, ob_array). PyMemoryViewObject structure ends with a flexible array: Py_ssize_t ob_array[1].

See also the discussion about the undefined behavior related to flexible arrays like this one: #84301

Metadata

Metadata

Assignees

No one assigned

    Labels

    type-bugAn unexpected behavior, bug, or error

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions