New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[subinterpreters] Static types incorrectly share some objects between interpreters #94673
Comments
The current Footnotes |
Python currently has a common GIL. |
I have the changes ready for this:
|
This is covered by PEP 684. The issue exists for implementation discussion and to anchor PRs. |
This has been discussed elsewhere. Also see #29228. FYI, for this PR (and in most cases) the performance penalty is insignificant. |
This is the first of several precursors to storing tp_subclasses (and tp_weaklist) on the interpreter state for static builtin types. We do the following: * add `_PyStaticType_InitBuiltin()` * add `_Py_TPFLAGS_STATIC_BUILTIN` * set it on all static builtin types in `_PyStaticType_InitBuiltin()` * shuffle some code around to be able to use _PyStaticType_InitBuiltin() * rename `_PyStructSequence_InitType()` to `_PyStructSequence_InitBuiltinWithFlags()` * add `_PyStructSequence_InitBuiltin()`.
Static builtin types are finalized by calling _PyStaticType_Dealloc(). Before this change, we were skipping finalizing such a type if it still had subtypes (i.e. its tp_subclasses hadn't been cleared yet). The problem is that types hold several heap objects, which leak if we skip the type's finalization. This change addresses that. For context, there's an old comment (from e9e3eab) that says the following: // If a type still has subtypes, it cannot be deallocated. // A subtype can inherit attributes and methods of its parent type, // and a type must no longer be used once it's deallocated. However, it isn't clear that is actually still true. Clearing tp_dict should mean it isn't a problem. Furthermore, the only subtypes that might still be around come from extension modules that didn't clean them up when unloaded (i.e. extensions that do not implement multi-phase initialization, AKA PEP 489). Those objects are already leaking, so this change doesn't change anything in that regard. Instead, this change means more objects gets cleaned up that before.
This is the last precursor to storing tp_subclasses (and tp_weaklist) on the interpreter state for static builtin types. Here we add per-type storage on PyInterpreterState, but only for the static builtin types. This involves the following: * add PyInterpreterState.types * move PyInterpreterState.type_cache to it * add a "num_builtins_initialized" field * add a "builtins" field (a static array big enough for all the static builtin types) * add _PyStaticType_GetState() to look up a static builtin type's state * (temporarily) add PyTypeObject.tp_static_builtin_index (to hold the type's index into PyInterpreterState.types.builtins) We will be eliminating tp_static_builtin_index in a later change.
Please get the PEP accepted before merging anything you can't easily revert. |
…95302) * Store tp_weaklist on the interpreter state for static builtin types. * Factor out _PyStaticType_GET_WEAKREFS_LISTPTR(). * Add _PyStaticType_ClearWeakRefs(). * Add a comment about how _PyStaticType_ClearWeakRefs() loops. * Document the change. * Update Doc/whatsnew/3.12.rst * Fix a typo.
While static types (
PyTypeObject
values) don't themselves ever change (oncePyType_Ready()
has run), they do hold mutable data. This means they cannot be safely shared between multiple interpreters without a common GIL (and without state leaking between).Mutable data:
__class__
)__base__
) - set byPyType_Ready()
if not set__bases__
) - always set byPyType_Ready()
__mro__
) - always set byPyType_Ready()
__dict__
) - set byPyType_Ready()
if not set__subclasses__
)(See https://docs.python.org/3/c-api/typeobj.html#tp-slots.)
(Note that
tp_cache
is no longer used.)For the object header, if PEP 683 (immortal objects) is accepted then we can make static types immortal.
For the otherwise immutable objects, we can make sure each is immortal and then we're good. Even
tp_dict
is fine since it gets hidden behindtypes.MappingProxyType
. We'd also need to either make sure each contained item is immortal.For
tp_subclasses
we will need a per-interpreter copy, and do the proper lookup in the__subclasses__
getter. The cache could be stored onPyInterpreterState
or even as a dict intp_subclasses
.For
tp_weaklist
it's a similar story as fortp_subclasses
. Note thattp_weaklist
isn't very important for static types since they are never deallocated.(The above is also discussed in PEP 684.)
CC @kumaraditya303
The text was updated successfully, but these errors were encountered: