New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gh-101659: Isolate "obmalloc" State to Each Interpreter #101660
base: main
Are you sure you want to change the base?
gh-101659: Isolate "obmalloc" State to Each Interpreter #101660
Conversation
FYI, the current CI failures are due to existing code. Basically, that code (in FWIW, the failure is exactly what we should expect to happen when per-interpreter data breaks isolation. I actually spent a while trying to figure out what I was doing wrong in my branch before realizing that everything was working as it should. UPDATE: I've opened gh-101758 to address this. |
Given what I've determined via gh-101758, I'll probably need to make non-isolated interpreters share the obmalloc state with the main interpreter. |
I'm tabling this until we've isolated all non-static objects. |
… init extensions.
FYI, as with the per-interpreter ref total, I'm probably going to add some code to preserve the per-interpreter allocated blocks when each subinterpreter is destroyed, so any leaks may be handled appropriately when the runtime is finalized. I may also report the per-interpreter count at interpreter finalization. We'll see. |
…h-102661) It doesn't make sense to use multi-phase init for these modules. Using a per-interpreter "m_copy" (instead of PyModuleDef.m_base.m_copy) makes this work okay. (This came up while working on gh-101660.) Note that we might instead end up disallowing re-load for sys/builtins since they are so special. #102660
This is strictly about moving the "obmalloc" runtime state from
_PyRuntimeState
toPyInterpreterState
. Doing so improves isolation between interpreters, specifically most of the memory (incl. objects) allocated for each interpreter's use. This is important for a per-interpreter GIL, but such isolation is valuable even without it.Benchmarking indicates the performance impact is negligible. (I'll re-run soon to double-check.)