Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash During Subinterpreter Finalization #105699

Closed
ericsnowcurrently opened this issue Jun 12, 2023 · 9 comments
Closed

Crash During Subinterpreter Finalization #105699

ericsnowcurrently opened this issue Jun 12, 2023 · 9 comments
Assignees
Labels
3.12 bugs and security fixes 3.13 new features, bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-subinterpreters type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@ericsnowcurrently
Copy link
Member

ericsnowcurrently commented Jun 12, 2023

There's an isolation leak somewhere. It may be just in the _xxsubinterpreters module, but I suspect it's not.

See #99114 (comment).

Reproducers:

FYI, I see crashes on this fairly infrequently.

Linked PRs

@ericsnowcurrently ericsnowcurrently added interpreter-core (Objects, Python, Grammar, and Parser dirs) type-crash A hard crash of the interpreter, possibly with a core dump topic-subinterpreters 3.12 bugs and security fixes 3.13 new features, bugs and security fixes labels Jun 12, 2023
@ericsnowcurrently
Copy link
Member Author

ericsnowcurrently commented Jun 12, 2023

Probably the same thing:

(AMD64 Arch Linux TraceRefs 3.12)

test_one (test.test__xxsubinterpreters.DestroyTests.test_one) ... ok
Objects/object.c:2211: _Py_ForgetReference: Assertion failed: invalid object chain
Enable tracemalloc to get the memory block allocation traceback
object address  : 0x7f6bf962d300
object refcount : 0
object type     : 0x55a320cd18c0
object type name: bytes
object repr     : <refcnt 0 at 0x7f6bf962d300>
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized
Current thread 0x00007f6c09e2e740 (most recent call first):
  <no Python frame>
Debug memory block at address p=0x7f6c09d55e90: API '�'
    18302063728033398269 bytes originally requested
    The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
        at p-7: 0xdd *** OUCH
        at p-6: 0xdd *** OUCH
        at p-5: 0xdd *** OUCH
        at p-4: 0xdd *** OUCH
        at p-3: 0xdd *** OUCH
        at p-2: 0xdd *** OUCH
        at p-1: 0xdd *** OUCH
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
    The 8 pad bytes at tail=0xfdfe7d6a07d35c8d are Fatal Python error: Segmentation fault
Current thread 0x00007f6c09e2e740 (most recent call first):
  <no Python frame>
Extension modules: _testcapi, _xxsubinterpreters, _xxinterpchannels (total: 3)
make: *** [Makefile:2015: buildbottest] Segmentation fault (core dumped)

@ericsnowcurrently
Copy link
Member Author

maybe related: #105690

ericsnowcurrently added a commit that referenced this issue Jun 14, 2023
This fixes a race during import. The existing _PyRuntimeState.imports.pkgcontext is shared between interpreters, and occasionally this would cause a crash when multiple interpreters were importing extensions modules at the same time.  To solve this we add a thread-local variable for the value.  We also leave the existing state (and infrequent race) in place for platforms that do not support thread-local variables.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 14, 2023
…-105740)

This fixes a race during import. The existing _PyRuntimeState.imports.pkgcontext is shared between interpreters, and occasionally this would cause a crash when multiple interpreters were importing extensions modules at the same time.  To solve this we add a thread-local variable for the value.  We also leave the existing state (and infrequent race) in place for platforms that do not support thread-local variables.
(cherry picked from commit b87d288)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
ericsnowcurrently added a commit that referenced this issue Jun 14, 2023
…) (gh-105765)

This fixes a race during import. The existing _PyRuntimeState.imports.pkgcontext is shared between interpreters, and occasionally this would cause a crash when multiple interpreters were importing extensions modules at the same time.  To solve this we add a thread-local variable for the value.  We also leave the existing state (and infrequent race) in place for platforms that do not support thread-local variables.
(cherry picked from commit b87d288)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
TheShermanTanker added a commit to TheShermanTanker/cpython that referenced this issue Jun 14, 2023
commit 74c2422
Author: Guido van Rossum <guido@python.org>
Date:   Wed Jun 14 08:19:24 2023 -0700

    Update DSL docs for cases generator (python#105753)

    * Clarify things around goto error/ERROR_IF a bit
    * Remove docs for super-instructions
    * Add pseudo; fix heading markup

commit 1d857da
Author: Mark Shannon <mark@hotpy.org>
Date:   Wed Jun 14 16:15:08 2023 +0100

    pythonGH-77273: Better bytecodes for f-strings (pythonGH-6132)

commit 307bcea
Author: Victor Stinner <vstinner@python.org>
Date:   Wed Jun 14 17:00:40 2023 +0200

    xmlrpc.client uses datetime.datetime.isoformat() (python#105741)

    Reimplement _iso8601_format() using the datetime isoformat() method.
    Ignore the timezone.

    Co-Authored-by: Paul Ganssle <1377457+pganssle@users.noreply.github.com>

commit 7b1f0f2
Author: Alex Waygood <Alex.Waygood@Gmail.com>
Date:   Wed Jun 14 15:58:41 2023 +0100

    pythongh-105570: Deprecate unusual ways of creating empty TypedDicts (python#105780)

    Deprecate two methods of creating typing.TypedDict classes with 0 fields using the functional syntax: `TD = TypedDict("TD")` and `TD = TypedDict("TD", None)`. Both will be disallowed in Python 3.15. To create a TypedDict class with 0 fields, either use `class TD(TypedDict): pass` or `TD = TypedDict("TD", {})`.

commit d32e8d6
Author: TATHAGATA ROY <royzen9495@gmail.com>
Date:   Wed Jun 14 19:51:30 2023 +0530

    pythongh-105196: Fix indentations of section headings in C API docs (python#105672)

commit 4a113e2
Author: Alex Waygood <Alex.Waygood@Gmail.com>
Date:   Wed Jun 14 15:19:27 2023 +0100

    Typing docs: move the deprecated stuff below the non-deprecated stuff (python#105781)

commit 7199584
Author: Mark Shannon <mark@hotpy.org>
Date:   Wed Jun 14 13:46:37 2023 +0100

    pythonGH-100987: Allow objects other than code objects as the "executable" of an internal frame. (pythonGH-105727)

    * Add table describing possible executable classes for out-of-process debuggers.

    * Remove shim code object creation code as it is no longer needed.

    * Make lltrace a bit more robust w.r.t. non-standard frames.

commit ad56340
Author: Alex Waygood <Alex.Waygood@Gmail.com>
Date:   Wed Jun 14 13:38:49 2023 +0100

    pythongh-105566: Deprecate unusual ways of creating `typing.NamedTuple` classes (python#105609)

    Deprecate creating a typing.NamedTuple class using keyword arguments to denote the fields (`NT = NamedTuple("NT", x=int, y=str)`). This will be disallowed in Python 3.15. Use the class-based syntax or the functional syntax instead.

    Two methods of creating `NamedTuple` classes with 0 fields using the functional syntax are also deprecated, and will be disallowed in Python 3.15: `NT = NamedTuple("NT")` and `NT = NamedTuple("NT", None)`. To create a `NamedTuple` class with 0 fields, either use `class NT(NamedTuple): pass` or `NT = NamedTuple("NT", [])`.

commit fc8037d
Author: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Date:   Wed Jun 14 05:35:06 2023 -0700

    pythongh-104873: Add typing.get_protocol_members and typing.is_protocol (python#104878)

    Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

commit ba516e7
Author: Kirill Podoprigora <kirill.bast9@mail.ru>
Date:   Wed Jun 14 15:17:12 2023 +0300

    pythongh-102541: Hide traceback in help prompt (pythongh-102614)

commit 0316063
Author: Victor Stinner <vstinner@python.org>
Date:   Wed Jun 14 12:30:22 2023 +0200

    gdb libpython.py: Remove compatibility code (python#105739)

    Remove compatibility code for Python 2 and early Python 3 versions.

    * Remove os_fsencode() reimplementation: use os.fsencode() directly.
      os.fsencode() was added to Python 3.2.
    * Remove references to Python 2 and "Python 3": just say "Python".
    * Remove outdated u'' string format: use '' instead.

commit e5d45b7
Author: Nikita Sobolev <mail@sobolevn.me>
Date:   Wed Jun 14 13:29:16 2023 +0300

    pythongh-105745: Fix open method of webbrowser.Konqueror (python#105746)

commit 67f69db
Author: Nikita Sobolev <mail@sobolevn.me>
Date:   Wed Jun 14 13:26:20 2023 +0300

    pythongh-105687: Remove deprecated objects from `re` module (python#105688)

commit fb655e0
Author: Victor Stinner <vstinner@python.org>
Date:   Wed Jun 14 12:12:25 2023 +0200

    _ctypes callbacks.c uses _Py_COMP_DIAG_IGNORE_DEPR_DECLS (python#105732)

    Replace #pragma with _Py_COMP_DIAG_PUSH,
    _Py_COMP_DIAG_IGNORE_DEPR_DECLS and _Py_COMP_DIAG_POP to ease Python
    maintenance. Also add a comment explaining why callbacks.c ignores a
    deprecation warning.

commit 5cdd5ba
Author: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Date:   Wed Jun 14 00:26:48 2023 -0700

    tarfile: Fix positional-only syntax in docs (pythonGH-105770)

    The syntax used in the current docs (a / before any args) is invalid.

    I think the right approach is for the arguments to arbitrary
    filter functions to be treated as positional-only, meaning that users
    can supply filter functions with any names for the argument. tarfile.py
    only calls the filter function with positional arguments.

commit 6199fe3
Author: Eddie Elizondo <eduardo.elizondorueda@gmail.com>
Date:   Wed Jun 14 00:33:32 2023 -0400

    pythongh-105587: Remove assertion from `_PyStaticObject_CheckRefcnt` (python#105638)

commit dab5a3e
Author: Barney Gale <barney.gale@gmail.com>
Date:   Wed Jun 14 05:06:58 2023 +0100

    pythonGH-89812: Clean up pathlib tests. (python#104829)

    Clean up pathlib tests.

    Merge `PurePathTest` into `_BasePurePathTest`, and `PathTest` into
    `_BasePathTest`.

commit b95de96
Author: Victor Stinner <vstinner@python.org>
Date:   Wed Jun 14 05:34:11 2023 +0200

    pythongh-105751: test_ctypes avoids "from ctypes import *" (python#105768)

    Using "import *" prevents linters like pyflakes to detect undefined names
    (usually missing imports).

    Replace c_voidp with c_void_p.

commit 381a1dc
Author: Victor Stinner <vstinner@python.org>
Date:   Wed Jun 14 04:47:01 2023 +0200

    pythongh-105751: test_ctypes.test_numbers uses top level imports (python#105762)

    Moroever, c_ulonglong and c_bool are always available.

commit ac7b551
Author: Victor Stinner <vstinner@python.org>
Date:   Wed Jun 14 04:46:47 2023 +0200

    pythongh-105751: test_ctypes gets Windows attrs from ctypes (python#105758)

    test_ctypes now gets attributes specific to Windows from the ctypes
    module, rather than relying on "from ctypes import *".

    Attributes:

    * ctypes.FormatError
    * ctypes.WINFUNCTYPE
    * ctypes.WinError
    * ctypes.WinDLL
    * ctypes.windll
    * ctypes.oledll
    * ctypes.get_last_error()
    * ctypes.set_last_error()

commit b87d288
Author: Eric Snow <ericsnowcurrently@gmail.com>
Date:   Tue Jun 13 18:58:23 2023 -0600

    pythongh-105699: Use a Thread-Local Variable for PKGCONTEXT (pythongh-105740)

    This fixes a race during import. The existing _PyRuntimeState.imports.pkgcontext is shared between interpreters, and occasionally this would cause a crash when multiple interpreters were importing extensions modules at the same time.  To solve this we add a thread-local variable for the value.  We also leave the existing state (and infrequent race) in place for platforms that do not support thread-local variables.

commit fcf0647
Author: Eric Snow <ericsnowcurrently@gmail.com>
Date:   Tue Jun 13 18:39:18 2023 -0600

    pythongh-104812: Skip Pending Calls Tests if No Threading (pythongh-105761)

    This fixes the WASM buildbots.

commit b542972
Author: Victor Stinner <vstinner@python.org>
Date:   Wed Jun 14 02:33:00 2023 +0200

    pythongh-105387: Limited C API implements Py_INCREF() as func (python#105388)

    In the limited C API version 3.12, Py_INCREF() and Py_DECREF()
    functions are now implemented as opaque function calls to hide
    implementation details.

commit f3266c0
Author: zentarim <33746047+zentarim@users.noreply.github.com>
Date:   Wed Jun 14 02:45:47 2023 +0300

    pythonGH-104554: Add RTSPS support to `urllib/parse.py` (python#104605)

    * pythonGH-104554: Add RTSPS support to `urllib/parse.py`

    RTSPS is the permanent scheme defined in
    https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml
    alongside RTSP and RTSPU schemes.

    * 📜🤖 Added by blurb_it.

    ---------

    Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>

commit 4cefe3c
Author: Steve Dower <steve.dower@python.org>
Date:   Wed Jun 14 00:00:16 2023 +0100

    pythongh-105436: Ignore unrelated errors when checking empty env (pythonGH-105742)

commit 457a459
Author: Victor Stinner <vstinner@python.org>
Date:   Wed Jun 14 00:32:12 2023 +0200

    pythongh-98040: Fix importbench: use types.ModuleType() (python#105743)

    Replace removed imp.new_module(name) with types.ModuleType(name).

commit 757b402
Author: Eric Snow <ericsnowcurrently@gmail.com>
Date:   Tue Jun 13 15:02:19 2023 -0600

    pythongh-104812: Run Pending Calls in any Thread (pythongh-104813)

    For a while now, pending calls only run in the main thread (in the main interpreter).  This PR changes things to allow any thread run a pending call, unless the pending call was explicitly added for the main thread to run.

commit 4e80082
Author: Łukasz Langa <lukasz@langa.pl>
Date:   Tue Jun 13 20:44:27 2023 +0000

    pythongh-102613: Bump recursion limit to fix running test_pathlib under Coverage (python#105744)

commit be2779c
Author: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
Date:   Tue Jun 13 21:42:03 2023 +0100

    pythongh-105481: add flags to each instr in the opcode metadata table, to replace opcode.hasarg/hasname/hasconst (python#105482)

commit 2211454
Author: Victor Stinner <vstinner@python.org>
Date:   Tue Jun 13 20:16:26 2023 +0200

    pythongh-105733: Deprecate ctypes SetPointerType() and ARRAY() (python#105734)

commit b97e14a
Author: Eric Snow <ericsnowcurrently@gmail.com>
Date:   Tue Jun 13 11:08:32 2023 -0600

    pythongh-105603: Change the PyInterpreterConfig.own gil Field (pythongh-105620)

    We are changing it to be more flexible that a strict bool can be for possible future expanded used cases.

commit abfbab6
Author: Lysandros Nikolaou <lisandrosnik@gmail.com>
Date:   Tue Jun 13 17:18:11 2023 +0200

    pythongh-105718: Fix buffer allocation in tokenizer with readline (python#105728)

commit d0f1afd
Author: Victor Stinner <vstinner@python.org>
Date:   Tue Jun 13 13:49:36 2023 +0200

    pythongh-105373: Remove PyArg_Parse() deprecation (python#105394)

    There is no plan to deprecate PyArg_Parse().

    The deprecation was added as a comment in the C API documentation in
    2007 by commit 85eb8c1.

commit ed8217b
Author: Lysandros Nikolaou <lisandrosnik@gmail.com>
Date:   Tue Jun 13 12:39:29 2023 +0200

    pythongh-105713: Document that tokenize raises when mixing tabs/spaces (python#105723)

    * pythongh-105713: Document that tokenize raises when mixing tabs/spaces
    * Update Doc/whatsnew/3.12.rst

    Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

commit c3d2d64
Author: Mark Shannon <mark@hotpy.org>
Date:   Tue Jun 13 10:34:27 2023 +0100

    Fix magic number (pythonGH-105722)

commit 09ffa69
Author: Mark Shannon <mark@hotpy.org>
Date:   Tue Jun 13 09:51:05 2023 +0100

    pythonGH-105678: Split MAKE_FUNCTION into MAKE_FUNCTION and SET_FUNCTION_ATTRIBUTE (pythonGH-105680)

commit 217589d
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Tue Jun 13 10:38:01 2023 +0200

    pythongh-105375: Improve error handling in _Unpickler_SetInputStream() (python#105667)

    Prevent exceptions from possibly being overwritten in case of multiple
    failures.

commit 840d02f
Author: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Date:   Tue Jun 13 11:36:40 2023 +0530

    pythonGH-105684: Require `asyncio.Task` implementations to support `set_name` method (python#105685)

commit 829ac13
Author: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Date:   Tue Jun 13 11:11:34 2023 +0530

    pythonGH-104787: use bitfields in `_asyncio` (python#104788)

commit 8da9d1b
Author: Guido van Rossum <guido@python.org>
Date:   Mon Jun 12 14:55:15 2023 -0700

    pythongh-105540: Fix code generator tests (python#105707)

    This involves expanding PEEK, POKE and JUMPBY macros,
    and removing super and register tests (those features no longer exist).

commit ca3cc4b
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Mon Jun 12 23:35:07 2023 +0200

    pythongh-105375: Explicitly initialise all {Pickler,Unpickler}Object fields (python#105686)

    All fields must be explicitly initialised to prevent manipulation of
    uninitialised fields in dealloc.

    Align initialisation order with the layout of the object structs.

commit f0fb782
Author: Jay <74105438+weijay0804@users.noreply.github.com>
Date:   Tue Jun 13 04:29:02 2023 +0800

    pythongh-105331: Change `asyncio.sleep` to raise ``ValueError` for nan (python#105641)

    Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>
    Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>

commit 9544948
Author: Guido van Rossum <guido@python.org>
Date:   Mon Jun 12 11:19:04 2023 -0700

    Remove support for legacy bytecode instructions (python#105705)

    (A legacy instruction is of the form `instr(FOOBAR)`,
    i.e. missing the `(... -- ...)` stack/cache effect annotation.)

commit b9e7dc7
Author: Guido van Rossum <guido@python.org>
Date:   Mon Jun 12 10:47:08 2023 -0700

    pythongh-105229: Remove syntactic support for super-instructions (python#105703)

    It will not be used again.

commit 4f7d3b6
Author: Dora203 <66343334+sku2000@users.noreply.github.com>
Date:   Tue Jun 13 00:14:55 2023 +0800

    pythongh-105436: The environment block should end with two null wchar_t values (pythonGH-105495)

commit 2b90796
Author: Petr Viktorin <encukou@gmail.com>
Date:   Mon Jun 12 17:45:49 2023 +0200

    pythongh-103968: PyType_FromMetaclass: Allow metaclasses with tp_new=NULL (pythonGH-105386)

commit 58f0bda
Author: Steve Dower <steve.dower@python.org>
Date:   Mon Jun 12 16:09:14 2023 +0100

    ARM64 clamping bug also exists in MSVC 14.35 (pythonGH-105679)

commit a8d69fe
Author: Nikita Sobolev <mail@sobolevn.me>
Date:   Mon Jun 12 11:47:56 2023 +0300

    pythongh-105673: Fix uninitialized warning in sysmodule.c (python#105674)

    In sys_add_xoption(), 'value' may be uninitialized for some error paths.

commit 58f5227
Author: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
Date:   Sun Jun 11 22:31:59 2023 +0100

    pythongh-105481: add pseudo-instructions to the bytecodes DSL (python#105506)

commit 20a56d8
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Sun Jun 11 22:18:46 2023 +0200

    pythongh-105375: Harden pyexpat initialisation (python#105606)

    Add proper error handling to add_errors_module() to prevent exceptions
    from possibly being overwritten.

commit 41cddc2
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Sun Jun 11 22:02:49 2023 +0200

    pythongh-105375: Improve error handling in the sys extension module (python#105611)

    In _PySys_AddXOptionWithError() and sys_add_xoption(),
    bail on first error to prevent exceptions from possibly being
    overwritten.

commit e8998e4
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Sun Jun 11 21:46:19 2023 +0200

    pythongh-105375: Improve error handling in _ctypes (python#105593)

    Prevent repeated PyLong_FromVoidPtr() from possibly overwriting the
    current exception.

commit 555be81
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Sun Jun 11 21:29:19 2023 +0200

    pythongh-105375: Improve error handling in PyUnicode_BuildEncodingMap() (python#105491)

    Bail on first error to prevent exceptions from possibly being overwritten.

commit 567d6ae
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Sun Jun 11 21:23:28 2023 +0200

    pythongh-105375: Improve PyErr_WarnExplicit() error handling (python#105610)

    Bail on first error to prevent exceptions from possibly being
    overwritten.

commit 3f7c081
Author: Samet YASLAN <sametyaslan@gmail.com>
Date:   Sun Jun 11 20:51:21 2023 +0200

    bpo-44185: Added close() to mock_open __exit__ (python#26902)

commit 18d16e9
Author: Tomas R <tomas.roun8@gmail.com>
Date:   Sun Jun 11 17:50:34 2023 +0200

    pythongh-102676: Add more convenience properties to `dis.Instruction` (python#103969)

    Adds start_offset, cache_offset, end_offset, baseopcode,
    baseopname, jump_target and oparg to dis.Instruction.

    Also slightly improves the disassembly output by allowing
    opnames to overflow into the space reserved for opargs.

commit 845e593
Author: litlighilit <97860435+litlighilit@users.noreply.github.com>
Date:   Sun Jun 11 22:06:42 2023 +0800

    Fix typo in configparser module docstring (python#105652)

    "zc.buildbot" -> "zc.buildout"

commit d4fa529
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Sun Jun 11 12:20:43 2023 +0200

    pythongh-105375: Improve error handling in the builtins extension module (python#105585)

commit c932f72
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Sun Jun 11 12:06:06 2023 +0200

    pythongh-105375: Improve _decimal error handling (python#105605)

    Fix a bug where an exception could end up being overwritten.

commit 16d4968
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Sun Jun 11 12:03:09 2023 +0200

    pythongh-105375: Harden _datetime initialisation (python#105604)

    Improve error handling so init bails on the first exception.

commit 35cff54
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Sun Jun 11 11:58:08 2023 +0200

    pythongh-105375: Improve array.array exception handling (python#105594)

    Fix a bug where 'tp_richcompare' could end up overwriting an exception.

commit 01f4230
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Sun Jun 11 11:56:32 2023 +0200

    pythongh-105375: Harden _ssl initialisation (python#105599)

    Add proper error handling to prevent reference leaks and overwritten
    exceptions.

commit cc87948
Author: Hugo van Kemenade <hugovk@users.noreply.github.com>
Date:   Sun Jun 11 12:17:35 2023 +0300

    pythongh-80480: Emit DeprecationWarning for array's 'u' type code (python#95760)

commit 3a314f7
Author: Gregory P. Smith <greg@krypto.org>
Date:   Sat Jun 10 12:09:20 2023 -0700

    pythongh-99108: Mention HACL\* in the hashlib docs. (python#105634)

commit 0d1d6ab
Author: Gregory P. Smith <greg@krypto.org>
Date:   Sat Jun 10 11:49:06 2023 -0700

    Cleanup and clarify our hashlib docs. (python#105624)

    Clarify and improve our hashlib docs. Now with 50% less mess!

commit d636d7d
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Fri Jun 9 23:53:33 2023 +0200

    pythongh-105375: Harden error handling in `_testcapi/heaptype.c` (python#105608)

    Bail on first error in heapctypesubclasswithfinalizer_finalize()

commit 33c92c4
Author: Nikita Sobolev <mail@sobolevn.me>
Date:   Sat Jun 10 00:48:54 2023 +0300

    pythongh-105375: Improve error handling in `zoneinfo` module (python#105586)

    Fix bugs where exceptions could end up being overwritten
    because of deferred error handling.

    Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

commit 91441bf
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Fri Jun 9 23:14:02 2023 +0200

    Docs: fix formatting in 2023-06-09-12-59-18 NEWS item (python#105607)

commit b047fa5
Author: Pablo Galindo Salgado <Pablogsal@gmail.com>
Date:   Fri Jun 9 21:39:01 2023 +0100

    pythongh-105549: Tokenize separately NUMBER and NAME tokens and allow 0-prefixed literals (python#105555)

commit 00b599a
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Fri Jun 9 22:35:03 2023 +0200

    pythongh-105375: Improve error handling in _elementtree (python#105591)

    Fix bugs where exceptions could end up being overwritten.

commit f668f73
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Fri Jun 9 22:07:47 2023 +0200

    pythongh-105375: Improve posix error handling (python#105592)

    Fix a bug where an IndexError could end up being overwritten.

commit eede1d2
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Fri Jun 9 21:57:25 2023 +0200

    pythongh-105375: Improve errnomodule error handling (python#105590)

    Bail immediately if an exception is set, to prevent exceptions from
    being overwritten.

commit 89aac6f
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Fri Jun 9 19:09:53 2023 +0200

    pythongh-105375: Improve _pickle error handling (python#105475)

    Error handling was deferred in some cases, which could potentially lead
    to exceptions being overwritten.

commit 6c832dd
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Fri Jun 9 18:55:53 2023 +0200

    pythongh-105375: Improve error handling in compiler_enter_scope() (python#105494)

commit d7f46bc
Author: Pablo Galindo Salgado <Pablogsal@gmail.com>
Date:   Fri Jun 9 17:01:26 2023 +0100

    pythongh-105564: Don't include artificial newlines in the line attribute of tokens (python#105565)

commit 1dd267a
Author: Pablo Galindo Salgado <Pablogsal@gmail.com>
Date:   Fri Jun 9 16:59:37 2023 +0100

    Clarify the supported cases in the tokenize module (python#105569)

commit 59f009e
Author: Ethan Furman <ethan@stoneleaf.us>
Date:   Fri Jun 9 08:56:05 2023 -0700

    pythongh-105497: [Enum] Fix Flag inversion when alias/mask members exist. (pythonGH-105542)

    When inverting a Flag member (or boundary STRICT), only consider other canonical flags; when inverting an IntFlag member (or boundary KEEP), also consider aliases.

commit 8e75592
Author: Alex Waygood <Alex.Waygood@Gmail.com>
Date:   Fri Jun 9 16:08:57 2023 +0100

    Miscellaneous improvements to the typing docs (python#105529)

    Mostly, these are changes so that we use shorter sentences and shorter paragraphs. In particular, I've tried to make the first sentence introducing each object in the typing API short and declarative.

commit b8fa7bd
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Fri Jun 9 15:36:59 2023 +0200

    pythongh-105557: Remove duplicate sqlite3 test method (python#105558)

    test_func_return_too_large_int() was defined twice.
    Keep only the redefined method, as that also checks the tracebacks.

commit 9bf8d82
Author: Thomas Grainger <tagrain@gmail.com>
Date:   Fri Jun 9 14:29:09 2023 +0100

    pythongh-94924: support `inspect.iscoroutinefunction` in `create_autospec(async_def)` (python#94962)

    * support inspect.iscoroutinefunction in create_autospec(async_def)

    * test create_autospec with inspect.iscoroutine and inspect.iscoroutinefunction

    * test when create_autospec functions check their signature

commit 0f885ff
Author: Victor Stinner <vstinner@python.org>
Date:   Fri Jun 9 14:50:31 2023 +0200

    pythongh-105407: Remove unused imports (python#105554)
@ericsnowcurrently
Copy link
Member Author

ericsnowcurrently commented Jul 19, 2023

After gh-106899, I'm only seeing 3 very infrequent crashers:

  1. failing assertion in _xxinterpreterchannelsmodule.c (not user-facing)
  2. bogus interned string state
  3. something bad in start_thread() (pthread)
(stack track for that last one)
Thread 1 (Thread 0x7f1c41efb700 (LWP 27875)):
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007f1c4a6bd7f1 in __GI_abort () at abort.c:79
#2  0x00007f1c4a706837 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f1c4a833a7b "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007f1c4a70d8ba in malloc_printerr (str=str@entry=0x7f1c4a831c8e "free(): invalid size") at malloc.c:5342
#4  0x00007f1c4a818a6c in _int_free (have_lock=0, p=0x7f1c28001490, av=0x7f1c28000020) at malloc.c:4171
#5  __GI___libc_free (mem=0x7f1c280014a0) at malloc.c:3134
#6  tcache_thread_shutdown () at malloc.c:2979
#7  arena_thread_freeres () at arena.c:950
#8  0x00007f1c4a819562 in __libc_thread_freeres () at thread-freeres.c:29
#9  0x00007f1c4b21a700 in start_thread (arg=0x7f1c41efb700) at pthread_create.c:476
#10 0x00007f1c4a79e61f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

@ericsnowcurrently
Copy link
Member Author

With the 3 PRs I have up I don't see any more crashes (other than in _xxinterpchannels).

ericsnowcurrently added a commit that referenced this issue Jul 21, 2023
A static (process-global) str object must only have its "interned" state cleared when no longer interned in any interpreters.  They are the only ones that can be shared by interpreters so we don't have to worry about any other str objects.

We trigger clearing the state with the main interpreter, since no other interpreters may exist at that point and _PyUnicode_ClearInterned() is only called during interpreter finalization.

We do not address here the fact that a string will only be interned in the first interpreter that interns it.  In any subsequent interpreters str.state.interned is already set so _PyUnicode_InternInPlace() will skip it.  That needs to be addressed separately from fixing the crasher.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 21, 2023
A static (process-global) str object must only have its "interned" state cleared when no longer interned in any interpreters.  They are the only ones that can be shared by interpreters so we don't have to worry about any other str objects.

We trigger clearing the state with the main interpreter, since no other interpreters may exist at that point and _PyUnicode_ClearInterned() is only called during interpreter finalization.

We do not address here the fact that a string will only be interned in the first interpreter that interns it.  In any subsequent interpreters str.state.interned is already set so _PyUnicode_InternInPlace() will skip it.  That needs to be addressed separately from fixing the crasher.
(cherry picked from commit 87e7cb0)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
ericsnowcurrently added a commit that referenced this issue Jul 21, 2023
…106923)

There was a slight race in _Py_ClearFileSystemEncoding() (when called from _Py_SetFileSystemEncoding()), between freeing the value and setting the variable to NULL, which occasionally caused crashes when multiple isolated interpreters were used.  (Notably, I saw at least 10 different, seemingly unrelated spooky-action-at-a-distance, ways this crashed. Yay, free threading!)  We avoid the problem by only setting the global variables with the main interpreter (i.e. runtime init).
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 21, 2023
pythongh-106923)

There was a slight race in _Py_ClearFileSystemEncoding() (when called from _Py_SetFileSystemEncoding()), between freeing the value and setting the variable to NULL, which occasionally caused crashes when multiple isolated interpreters were used.  (Notably, I saw at least 10 different, seemingly unrelated spooky-action-at-a-distance, ways this crashed. Yay, free threading!)  We avoid the problem by only setting the global variables with the main interpreter (i.e. runtime init).
(cherry picked from commit 0ba07b2)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
Yhg1s pushed a commit that referenced this issue Jul 21, 2023
…le (gh-106923) (#106964)

gh-105699: Fix a Crasher Related to a Deprecated Global Variable (gh-106923)

There was a slight race in _Py_ClearFileSystemEncoding() (when called from _Py_SetFileSystemEncoding()), between freeing the value and setting the variable to NULL, which occasionally caused crashes when multiple isolated interpreters were used.  (Notably, I saw at least 10 different, seemingly unrelated spooky-action-at-a-distance, ways this crashed. Yay, free threading!)  We avoid the problem by only setting the global variables with the main interpreter (i.e. runtime init).
(cherry picked from commit 0ba07b2)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
Yhg1s pushed a commit that referenced this issue Jul 21, 2023
gh-105699: Fix an Interned Strings Crasher (gh-106930)

A static (process-global) str object must only have its "interned" state cleared when no longer interned in any interpreters.  They are the only ones that can be shared by interpreters so we don't have to worry about any other str objects.

We trigger clearing the state with the main interpreter, since no other interpreters may exist at that point and _PyUnicode_ClearInterned() is only called during interpreter finalization.

We do not address here the fact that a string will only be interned in the first interpreter that interns it.  In any subsequent interpreters str.state.interned is already set so _PyUnicode_InternInPlace() will skip it.  That needs to be addressed separately from fixing the crasher.
(cherry picked from commit 87e7cb0)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 22, 2023
…ythonGH-106966)

(cherry picked from commit adda43d)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
ericsnowcurrently added a commit that referenced this issue Jul 25, 2023
…H-106966) (gh-107012)

gh-105699: Add some stress tests for subinterpreter creation (GH-106966)
(cherry picked from commit adda43d)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
@ericsnowcurrently
Copy link
Member Author

This can be closed once gh-106974 lands.

@ericsnowcurrently
Copy link
Member Author

The stress tests I added are still crashing occasionally. I suspect most of the problem lies with the _xxsubinterpreters module, but I'm going to check. In the meantime I'm going to disable the two tests to cut down on noise as we approach 3.12rc1.

CC @Yhg1s

@ericsnowcurrently
Copy link
Member Author

Notably, I can reproduce a crash on the 3.12 branch but not on main. (./python -m test test_interpreters -m test_create_many_threaded)

ericsnowcurrently added a commit that referenced this issue Jul 27, 2023
The two tests are crashing periodically in CI and on buildbots.  I suspect the problem is in the _xxsubinterpreters module. 
 Regardless, I'm disabling the tests temporarily, to reduce the noise as we approach 3.12rc1.  I'll be investigating the crashes separately.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 27, 2023
The two tests are crashing periodically in CI and on buildbots.  I suspect the problem is in the _xxsubinterpreters module.
 Regardless, I'm disabling the tests temporarily, to reduce the noise as we approach 3.12rc1.  I'll be investigating the crashes separately.
(cherry picked from commit 4f67921)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
ericsnowcurrently added a commit that referenced this issue Jul 27, 2023
…h-107357)

gh-105699: Disable the Interpreters Stress Tests (gh-107354)

The two tests are crashing periodically in CI and on buildbots.  I suspect the problem is in the _xxsubinterpreters module.
 Regardless, I'm disabling the tests temporarily, to reduce the noise as we approach 3.12rc1.  I'll be investigating the crashes separately.
(cherry picked from commit 4f67921)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
ericsnowcurrently added a commit that referenced this issue Jul 28, 2023
This fixes a crasher due to a race condition, triggered infrequently when two isolated (own GIL) subinterpreters simultaneously initialize their sys or builtins modules.  The crash happened due the combination of the "detached" thread state we were using and the "last holder" logic we use for the GIL.  It turns out it's tricky to use the same thread state for different threads.  Who could have guessed?

We solve the problem by eliminating the one object we were still sharing between interpreters.  We replace it with a low-level hashtable, using the "raw" allocator to avoid tying it to the main interpreter.

We also remove the accommodations for "detached" thread states, which were a dubious idea to start with.
ericsnowcurrently added a commit to ericsnowcurrently/cpython that referenced this issue Jul 28, 2023
…hongh-106974)

This fixes a crasher due to a race condition, triggered infrequently when two isolated (own GIL) subinterpreters simultaneously initialize their sys or builtins modules.  The crash happened due the combination of the "detached" thread state we were using and the "last holder" logic we use for the GIL.  It turns out it's tricky to use the same thread state for different threads.  Who could have guessed?

We solve the problem by eliminating the one object we were still sharing between interpreters.  We replace it with a low-level hashtable, using the "raw" allocator to avoid tying it to the main interpreter.

We also remove the accommodations for "detached" thread states, which were a dubious idea to start with.

(cherry picked from commit 8ba4df9)
ericsnowcurrently added a commit that referenced this issue Jul 28, 2023
…-106974) (gh-107412)

gh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache (gh-106974)

This fixes a crasher due to a race condition, triggered infrequently when two isolated (own GIL) subinterpreters simultaneously initialize their sys or builtins modules.  The crash happened due the combination of the "detached" thread state we were using and the "last holder" logic we use for the GIL.  It turns out it's tricky to use the same thread state for different threads.  Who could have guessed?

We solve the problem by eliminating the one object we were still sharing between interpreters.  We replace it with a low-level hashtable, using the "raw" allocator to avoid tying it to the main interpreter.

We also remove the accommodations for "detached" thread states, which were a dubious idea to start with.

(cherry picked from commit 8ba4df9)
@ericsnowcurrently
Copy link
Member Author

I've disable the new stress tests as they were causing crashes regularly that I'm not convinced indicate bugs (outside _xxsubinterpreters). Once that's sorted out and I've re-enabled the tests, we can close this issue.

ericsnowcurrently added a commit that referenced this issue Aug 8, 2023
We had disabled them due to crashes they exposed, which have since been fixed.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Aug 8, 2023
…thongh-107572)

We had disabled them due to crashes they exposed, which have since been fixed.
(cherry picked from commit f9e3ff1)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
@ericsnowcurrently
Copy link
Member Author

I'm calling this good.

ericsnowcurrently added a commit that referenced this issue Nov 27, 2023
…h-107572) (#107783)

We had disabled them due to crashes they exposed, which have since been fixed.
(cherry picked from commit f9e3ff1)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: T. Wouters <thomas@python.org>
ericsnowcurrently added a commit to ericsnowcurrently/cpython that referenced this issue Nov 27, 2023
ericsnowcurrently added a commit that referenced this issue Nov 27, 2023
…rs Stress Tests" (gh-112474)

Revert "[3.12] gh-105699: Re-enable the Multiple-Interpreters Stress Tests (gh-107572) (#107783)"

This reverts commit a4aac7d.

The stress tests are still failing on FreeBSD.
ericsnowcurrently added a commit to ericsnowcurrently/cpython that referenced this issue Nov 28, 2023
…sts (pythongh-107572) (python#107783)

We had disabled them due to crashes they exposed, which have since been fixed.
(cherry picked from commit f9e3ff1)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: T. Wouters <thomas@python.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.12 bugs and security fixes 3.13 new features, bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-subinterpreters type-crash A hard crash of the interpreter, possibly with a core dump
Projects
Status: Done
Development

No branches or pull requests

1 participant