[subinterpreters] PEP 554 implementation: add interpreters module #76785
Comments
|
In the interest of getting something landed for 3.7, so we can start using it in tests, I'm putting up a patch for a low-level interpreters module. In some ways this is a precursor for issue bpo-30439, which will add a proper public stdlib module in 3.8. The module I'm adding draws from the ideas in PEP-554 (particularly for channels). Consequently, this will also give us an opportunity to try out some of the semantics from the PEP to give us better ideas for 3.8. I expect to have some follow-on patches to facilitate simpler use in tests. This patch is big enough already. :) |
ericsnowcurrently
added
3.7 (EOL)
|
@ned, it may be a little tight to land this given the time left before beta 1. However, this is meant as a tool for us to use in the test suite (particularly to test the subinterpreter C-API). So I'm arguing that, if necessary, it would still be okay to land this after the feature freeze. (I'm still hoping to get this in before the cutoff.) What do you think? |
|
FYI, there are a few things I need to clean up in the PR. However, I expect that those changes will be minor relative to the the whole patch, so I wanted to get the ball rolling on a review. :) |
|
@eric, given the breadth of change introduced in the PR (including adding a new extension), I think it would be best if at all possible to get it in for beta 1 if we can resolve the review comments in time. If necessary and if there are no objections from other core developers, I would be willing to consider making an exception and allowing it into beta 2 as long as it remains a private interface. If it looks like it won't be in releasable shape by then, I think you should hold off for 3.8; doing otherwise would be unfair to others and to our downstream beta users / testers, for example, even if it is private, adding a new extension and setup.py changes potentially affect downstream packagers. |
|
Sounds good, Ned. Thanks for taking a look. I should have everything finished up by Friday, so I'm hopeful for landing the change before the deadline. I may have a few minor tweaks to make after that, but I'll discuss that with you before making any changes if that happens. |
|
I've merged the patch without Windows support, which shouldn't be a problem given the purpose of the extension module. I've also added a PR for get the module building under Windows. I'd like to get that resolved ASAP. |
|
Eric, looks like some buildbots are unhappy, for instance: |
|
Yeah, I'm looking into it. Also, I noticed some refleaks that I'll be sorting out. |
|
On 4 of the buildbots: ====================================================================== Traceback (most recent call last):
File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 890, in test_drop_multiple_times
interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)====================================================================== Traceback (most recent call last):
File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 848, in test_drop_single_user
interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)====================================================================== Traceback (most recent call last):
File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 957, in test_drop_used_multiple_times_by_single_user
interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)====================================================================== Traceback (most recent call last):
File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 899, in test_drop_with_unused_items
interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3) |
|
On the PPC64 AIX 3.x buildbot: ====================================================================== Traceback (most recent call last):
File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/test__xxsubinterpreters.py", line 784, in test_repr
self.assertEqual(repr(cid), 'ChannelID(10)')
AssertionError: 'ChannelID(0)' != 'ChannelID(10)'
- ChannelID(0)
+ ChannelID(10)
? + |
|
I just put up a PR that should fix the 4 buildbots. |
|
The buildbots should be happier now. I'll keep an eye on them. |
|
A couple defects reported by coverity: ** CID 1428758: Integer handling issues (CONSTANT_EXPRESSION_RESULT) /Modules/_xxsubinterpretersmodule.c: 45 in _coerce_id() 1217 } |
|
I've landed a PR that fixes all the memory leaks in the module. It also fixes the 2 defects reported by coverity. The only thing left here is to get the module building under Windows. |
|
FYI, out of 2389 source lines in the C extension, 1563 are the channel-related code. That means the non-channel code is 826 lines (about a third). That non-channel code does not depend on the channel code at all and I considered splitting the source out, but figured there wasn't enough benefit. However, I might revisit the matter later when I circle back to PEP-554. :) |
|
Eric, it looks like your recent commit introduced a refleak. We need to fix it before beta2. ~/d/p/cpython (master $) » ./python.exe -m test -R3:3 test_multiprocessing_fork 1 test failed: And just before it: ~/d/p/cpython ((bd09335…) $) » ./python.exe -m test -R3:3 test_multiprocessing_fork Total duration: 9 min 12 sec |
…l C-API (pythongh-111715)" This reverts commit d4426e8.
…ythongh-111715) I added _Py_excinfo to the internal API (and added its functions in Python/errors.c) in pythongh-111530 (9322ce9). Since then I've had a nagging sense that I should have added the type and functions in its own PR. While I do plan on using _Py_excinfo outside crossinterp.c very soon (see pythongh-111572/pythongh-111573), I'd still feel more comfortable if the _Py_excinfo stuff went in as its own PR. Hence, here we are. (FWIW, I may combine that with pythongh-111572, which I may, in turn, combine with pythongh-111573. We'll see.)
The new function corresponds to the existing (public) PyType_GetName() and PyType_GetQualName().
This involves a number of changes for PEP 734.
gh-112982 broke test_threading on one of the s390 buildbots (Fedora Clang Installed). Apparently ImportError is raised (rather than ModuleNotFoundError) for the name part of "from" imports. This fixes that by catching ImportError in test_threading.py.
This brings the module (along with the associated extension modules) mostly in sync with PEP 734. There are only a few small things to wrap up.
This is one of the last pieces to get test.support.interpreters in sync with PEP 734.
…h-113034) When an exception is uncaught in Interpreter.exec_sync(), it helps to show that exception's error display if uncaught in the calling interpreter. We do so here by generating a TracebackException in the subinterpreter and passing it between interpreters using pickle.
…ter Exceptions (gh-113036) We need the TracebackException of uncaught exceptions for a single purpose: the error display. Thus we only need to pass the formatted error display between interpreters. Passing a pickled TracebackException is overkill.
In gh-112982 I made some changes to .github/CODEOWNERS. Later, @ezio-melotti pointed out that some of those changes were unnecessary.
…GH-112982) This is so that we can run python -m test.test_interpreters. As such it backports that aspect of commit 86a77f4, where it is implemnented by a package __main__.py.
…GH-112982) This is so that we can run python -m test.test_interpreters. As such it backports that aspect of commit 86a77f4, where it is implemented by a package __main__.py.
…GH-112982) (pythonGH-113470) This is so that we can run python -m test.test_interpreters. As such it backports that aspect of commit 86a77f4, where it is implemented by a package __main__.py. (cherry picked from commit bdad5c3) Co-authored-by: Jeff Allen <ja.py@farowl.co.uk>
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs
The text was updated successfully, but these errors were encountered: