Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maybe Drop "channels" from _xxsubinterpreters #101524

Open
ericsnowcurrently opened this issue Feb 2, 2023 · 1 comment
Open

Maybe Drop "channels" from _xxsubinterpreters #101524

ericsnowcurrently opened this issue Feb 2, 2023 · 1 comment
Labels
3.12 expert-subinterpreters extension-modules C modules in the Modules dir type-feature A feature request or enhancement

Comments

@ericsnowcurrently
Copy link
Member

ericsnowcurrently commented Feb 2, 2023

The _xxsubinterpreters module is essentially the low-level implementation of PEP 554. However, we added it a while back for testing purposes, especially to further exercise the runtime relative to subinterpreters. Since then, I've removed "channels" from PEP 554. So it may make sense to drop that part of the implementation. That part of Modules/_xxsubinterpretersmodule.c is much more code and certainly much more complex than the basic functionality the PEP now describes.

Linked PRs

@ericsnowcurrently ericsnowcurrently added type-feature A feature request or enhancement extension-modules C modules in the Modules dir expert-subinterpreters 3.12 labels Feb 2, 2023
ericsnowcurrently added a commit that referenced this issue Feb 4, 2023
This is step 1 in potentially dropping all the "channel"-related code. Channels have already been removed from PEP 554.

#101524
@rhettinger
Copy link
Contributor

This is a bummer. It was one of the few demonstrable benefits of subinterpreters. It seems that we're left with only the most ancient and awkward forms of communcation, shared file descriptors and pipes. Almost nothing in CSP can be readily expressed with those low level tools.

Side note: The PEP seems significantly understate the impact of isolating modules, "This situation is limited to modules that use C globals (or use libraries that use C globals) to store internal state.". As Erland Aasland sweeps through standard library making edits, the cost is becoming clear. Even simple stateless modules like bisect get significant churn. More interesting but stateless modules like itertools have about half the lines in the module changed (see gh-101277 for just one-fourth of the edits). The impact is massive. Along the way, the edits negatively impact performance which is annoying because almost the entire point of writing C accelerator code is to fix performance issues on critical code paths. We're losing freelists, ability of a type to reference its own methods, the _Py_IDENTIFIER() optimization, singletons, etc.

So far, this massive effort seems like all cost and no benefit. Gaining Go-like or CSP-like channels would have been the one win. Otherwise, all we left with is the questionable benefit of "looking like one process to the O/S". It isn't clear at all that that is something we want to pay for.

The cost/benefit proposition of PEP 554 has changed considerably since the beginning. The code churn is huge and is taking many man-months of time. Meanwhile the proposed benefits are down to "looks like a single process" and "ability to experiment with new concurrency models".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.12 expert-subinterpreters extension-modules C modules in the Modules dir type-feature A feature request or enhancement
Projects
Status: No status
Development

No branches or pull requests

2 participants