New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[subinterpreters] Global C variables are a problem #81057
Comments
We still have a bunch of "global" C variables (static globals, static locals, maybe thread-local storage) in our code-base that break the isolation between interpreters. I added Tools/c-globals/check-c-globals.py a while back to help identify such variables, however more have crept in. I also did not take static locals or thread-locals into account. To address the above, we should do the following:
Separately, we should move fields out of _PyRuntimeState into PyInterpreterState wherever possible. That can also be done at step 2 if it's not too much work. |
Also, Tools/c-globals/ignored-globals.txt is a bit out of date (some vars have been removed, renamed, or moved to another file). That should get cleaned up. It might also make sense to update check-c-globals.py to verify that all variables in ignored-globals.txt actually exist. |
The new test_check_c_globals.ActualChecks test is failing with an "unexpected success" on the bolen-ubuntu buildbot (under Ubuntu 18.04.3). I can reproduce the failure in a manually built tree. |
@db3l, I'll take a look right away. |
New changeset 9def81a by Vinay Sajip in branch 'master': |
Thanks, Vinay! |
FYI, others have been tackling this in separate issues (e.g. Victor, anyone relative to PEP-384). |
And I *am* still working on the c-analyzer + a test to that runs it, so we can keep from adding more globals. :) |
I'm getting the following FutureWarning on a certain regular expression. I think it just needs "[]" to become "\[\]". 0:05:36 load avg: 0.00 [ 53/427] test_check_c_globals |
… Objects Initializer (gh-99389) Up until now we had a single generated initializer macro for all the statically declared global objects in _PyRuntimeState, including several one-offs (e.g. the empty tuple). The one-offs don't need to be generated, but were because we had one big initializer. Having separate initializers for set of generated global objects allows us to generate only the ones we need to. This allows us to add initializers for one-off global objects without having to generate them. #81057
As we consolidate global variables, we find some objects that are almost suitable to add to _PyRuntimeState.global_objects, but have some small/sneaky bit of per-interpreter state (e.g. a weakref list). We're adding PyInterpreterState.static_objects so we can move such objects there. (We'll removed the _not_used field once we've added others.) #81057
The global allocators were stored in 3 static global variables: _PyMem_Raw, _PyMem, and _PyObject. State for the "small block" allocator was stored in another 13. That makes a total of 16 global variables. We are moving all 16 to the _PyRuntimeState struct as part of the work for gh-81057. (If PEP 684 is accepted then we will follow up by moving them all to PyInterpreterState.) #81057
We actually don't move PyImport_Inittab. Instead, we make a copy that we keep on _PyRuntimeState and use only that after Py_Initialize(). We also prevent folks from modifying PyImport_Inittab (the best we can) after that point. #81057
…Global Objects Initializer (pythongh-99389) Up until now we had a single generated initializer macro for all the statically declared global objects in _PyRuntimeState, including several one-offs (e.g. the empty tuple). The one-offs don't need to be generated, but were because we had one big initializer. Having separate initializers for set of generated global objects allows us to generate only the ones we need to. This allows us to add initializers for one-off global objects without having to generate them. python#81057
* main: (8272 commits) Update Windows readme.txt to clarify Visual Studio required versions (pythonGH-99522) pythongh-99460 Emscripten trampolines on optimized METH_O and METH_NOARGS code paths (python#99461) pythongh-92647: [Enum] use final status to determine lookup or create (pythonGH-99500) pythongh-81057: Move Globals in Core Code to _PyRuntimeState (pythongh-99496) Post 3.12.0a2 pythongh-99300: Use Py_NewRef() in Python/Python-ast.c (python#99499) pythongh-93649: Split pytime and datetime tests from _testcapimodule.c (python#99494) pythongh-99370: fix test_zippath_from_non_installed_posix (pythonGH-99483) pythonGH-99205: remove `_static` field from `PyThreadState` and `PyInterpreterState` (pythonGH-99385) pythongh-81057: Move the Remaining Import State Globals to _PyRuntimeState (pythongh-99488) pythongh-87604: Avoid publishing list of active per-interpreter audit hooks via the gc module (pythonGH-99373) pythongh-93649: Split getargs tests from _testcapimodule.c (python#99346) pythongh-81057: Move Global Variables Holding Objects to _PyRuntimeState. (pythongh-99487) pythonGH-98219: reduce sleep time in `asyncio` subprocess test (python#99464) pythonGH-99388: add `loop_factory` parameter to `asyncio.run` (python#99462) pythongh-99300: Use Py_NewRef() in PC/ directory (python#99479) pythongh-99300: Use Py_NewRef() in Doc/ directory (python#99480) pythongh-99300: Use Py_NewRef() in Modules/ directory (python#99473) pythongh-99300: Use Py_NewRef() in Modules/ directory (python#99469) pythongh-99370: Calculate zip path from prefix when in a venv (pythonGH-99371) ...
ericsnowcurrently commentedMay 10, 2019
•
edited by bedevere-bot
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: