[WIP] bpo-42671: Make Python finalization deterministic #23826
Conversation
Make the Python finalization more deterministic: * First, clear the __main__ module. * Then, clear modules from the most recently imported to the least recently imported: reversed(sys.modules). * builtins and sys modumes are always cleared last. * Module attributes are set to None from the most recently defined to the least recently defined: reversed(module.__dict__). Changes: * finalize_modules() no longer uses a list of weak references to modules while clearing sys.modules dict. * When -vv command line option is used, the module name is now also logged, not only the attribute name. * test_module.test_module_finalization_at_shutdown(): final_a.x is now None when final_a.c is cleared. * test_sys.test_sys_ignores_cleaning_up_user_data(): the exception is no longer silently ignored. Rename the test to test_sys_cleaning_up_user_data(). * test_threading.test_main_thread_during_shutdown() keeps a reference to threading functions since threading module variables are cleared before RefCycle object is deleted by the garbage collector.
|
|
||
| int verbose = _Py_GetConfig()->verbose; | ||
| for (int step=1; step <= 2; step++) { | ||
| PyObject *reversed = PyObject_CallOneArg((PyObject*)&PyReversed_Type, dict); |
There was a problem hiding this comment.
PyDict_Keys() can be used.
In future we can also add _PyDict_Prev().
There was a problem hiding this comment.
PyDict_Keys() returns keys in their creation order, no? I would like to get them in the reverse order. Do you suggest to iterate on the list backwards?
| PyObject *key, *value; | ||
|
|
||
| int verbose = _Py_GetConfig()->verbose; | ||
| for (int step=1; step <= 2; step++) { |
There was a problem hiding this comment.
I think that it may be better now to clear all objects in the same order, without prioritizing underscored names.
There was a problem hiding this comment.
Oh. I didn't know, so I chose the conservative option: keep the existing "heuristic". I'm fine with making the cleanup simpler :-)
| PyErr_WriteUnraisable(NULL); | ||
| return; | ||
| } | ||
| PyObject *iter = PyObject_GetIter(keys); |
There was a problem hiding this comment.
It is easier to iterate list by index.
There was a problem hiding this comment.
Oh, you're right. My first attempt avoided the creation of a list, but then I get annoying RuntimeError when the dict was modified while we iterate on it. Deleting a variable can have many side effects, it's way safer to copy the list of keys.
| PyErr_Clear(); | ||
| } | ||
|
|
||
| PyObject *reversed = PyObject_CallOneArg((PyObject*)&PyReversed_Type, modules); |
There was a problem hiding this comment.
PyMapping_Keys() + iterating by index.
|
I will update my PR later to take Serhiy's comment in account. |
|
This PR is stale because it has been open for 30 days with no activity. |
Make the Python finalization more deterministic:
recently imported: reversed(sys.modules).
least recently defined: reversed(module.dict).
Changes:
modules while clearing sys.modules dict.
logged, not only the attribute name.
now None when final_a.c is cleared.
no longer silently ignored. Rename the test to
test_sys_cleaning_up_user_data().
to threading functions since threading module variables are cleared
before RefCycle object is deleted by the garbage collector.
https://bugs.python.org/issue42671