We still have a bunch of "global" C variables (static globals, static locals, maybe thread-local storage) in our code-base that break the isolation between interpreters. I added Tools/c-globals/check-c-globals.py a while back to help identify such variables, however more have crept in. I also did not take static locals or thread-locals into account.

To address the above, we should do the following:

1. update check-c-globals.py to identify static locals (and thread-locals)
2. deal with any identified globals
   • move them to _PyRuntimeState (or thread-locals to PyThreadState, etc.)
   • ignore them by adding them to Tools/c-globals/ignored-globals.txt
3. add check-c-globals.py to "make check"
4. (if "make check" isn't already there), ensure check-c-globals.py is run at some point in CI

Separately, we should move fields out of _PyRuntimeState into PyInterpreterState wherever possible. That can also be done at step 2 if it's not too much work.

Also, Tools/c-globals/ignored-globals.txt is a bit out of date (some vars have been removed, renamed, or moved to another file). That should get cleaned up. It might also make sense to update check-c-globals.py to verify that all variables in ignored-globals.txt actually exist.

New changeset ee536b2 by Eric Snow in branch 'master':
bpo-36876: Add a tool that identifies unsupported global C variables. (bpo-15877)
ee536b2

The new test_check_c_globals.ActualChecks test is failing with an "unexpected success" on the bolen-ubuntu buildbot (under Ubuntu 18.04.3). I can reproduce the failure in a manually built tree.

@db3l, I'll take a look right away.

New changeset 64535fc by Eric Snow in branch 'master':
bpo-36876: Skip test_check_c_globals for now. (gh-16017)
64535fc

New changeset 088b63e by Eric Snow in branch 'master':
bpo-36876: Fix the globals checker tool. (gh-16058)
088b63e

New changeset e4c431e by Eric Snow in branch 'master':
bpo-36876: Re-organize the c-analyzer tool code. (gh-16841)
e4c431e

New changeset 9def81a by Vinay Sajip in branch 'master':
bpo-36876: Moved Parser/listnode.c statics to interpreter state. (GH-16328)
9def81a

Thanks, Vinay!

FYI, others have been tackling this in separate issues (e.g. Victor, anyone relative to PEP-384).

And I *am* still working on the c-analyzer + a test to that runs it, so we can keep from adding more globals. :)

More and more C extensions are converted to multiphase initialization API (PEP-489) and their global variables are moved into a new module state.

bpo-39465 will make _Py_IDENTIFIER compatible with subinterpreters.

See also bpo-40521 for caches like free lists and Unicode interned strings.

New changeset 345cd37 by Eric Snow in branch 'master':
bpo-36876: Fix the C analyzer tool. (GH-22841)
345cd37

New changeset 4fe7209 by Eric Snow in branch 'master':
bpo-36876: Small adjustments to the C-analyzer tool. (GH-23045)
4fe7209

New changeset 9f02b47 by Eric Snow in branch 'master':
bpo-36876: [c-analyzer tool] Tighten up the results and output. (GH-23431)
9f02b47

New changeset 7ec59d8 by Eric Snow in branch 'master':
bpo-36876: [c-analyzer tool] Add a "capi" subcommand to the c-analyzer tool. (gh-23918)
7ec59d8

New changeset 5ae9be6 by Eric Snow in branch 'master':
bpo-36876: [c-analyzer tool] Additional CLI updates for "capi" command. (gh-23929)
5ae9be6

I'm getting the following FutureWarning on a certain regular expression. I think it just needs "[]" to become "\[\]".

0:05:36 load avg: 0.00 [ 53/427] test_check_c_globals
...\Tools\c-analyzer\c_common\tables.py:236: FutureWarning: Possible nested set at position 12
_COLSPEC_RE = re.compile(textwrap.dedent(r'''

New changeset 77bab59 by Eric Snow in branch 'main':
bpo-36876: Update the c-analyzer whitelist. (gh-31225)
77bab59

New changeset cb68788 by Eric Snow in branch 'main':
bpo-36876: Minor cleanup to c-analyzer "ignored" data.' (gh-31239)
cb68788

New changeset 80e4f26 by Eric Snow in branch 'main':
bpo-36876: Make sure the c-analyzer is checking all the source files.' (gh-31264)
80e4f26

I still don't understand how moving global variables into _PyRuntime does solve any problem. What are the advantages of moving a variable declared as "static" in a C file into _PyRuntime? I don't get the rationale.

For me, it just sounds more complicated to get _PyRuntime.threads._condattr_monotonic.ptr instead of condattr_monotonic. It's annoying to have to repeat all conditions to decide if a variable should be declared or not in _PyRuntimeState, since they are tons of variables, and many are platform specific. A side effect is that any C file using _PyRuntime should now include headers files required to decide if platform specific variables should be declared or not.

I already noticed this problem when I made global variables per-interpreter: move them into PyInterpreterState. See the long list of direct includes in pycore_interp.h:

#include "pycore_atomic.h"        // _Py_atomic_address
#include "pycore_ast_state.h"     // struct ast_state
#include "pycore_code.h"          // struct callable_cache
#include "pycore_context.h"       // struct _Py_context_state
#include "pycore_dict.h"          // struct _Py_dict_state
#include "pycore_exceptions.h"    // struct _Py_exc_state
#include "pycore_floatobject.h"   // struct _Py_float_state
#include "pycore_genobject.h"     // struct _Py_async_gen_state
#include "pycore_gil.h"           // struct _gil_runtime_state
#include "pycore_gc.h"            // struct _gc_runtime_state
#include "pycore_list.h"          // struct _Py_list_state
#include "pycore_tuple.h"         // struct _Py_tuple_state
#include "pycore_typeobject.h"    // struct type_cache
#include "pycore_unicodeobject.h" // struct _Py_unicode_state
#include "pycore_warnings.h"      // struct _warnings_runtime_state

But they are also many indirect includes, like <windows.h> which is quite big and has many side effects :-(