gh-99741: Clean Up the _xxsubinterpreters Module #99940

ericsnowcurrently · 2022-12-02T01:29:56Z

This cleanup up resolves a few subtle bugs and makes the implementation for multi-phase init much cleaner.

Issue: Implement Multi-Phase Init for _xxsubinterpreters #99741

bedevere-bot · 2022-12-02T01:45:01Z

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit f5e53ef 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

bedevere-bot · 2022-12-02T19:17:09Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot wasm32-emscripten node (pthreads) 3.x has failed when building commit 0547a98.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1050/builds/955) and take a look at the build logs.
Check if the failure is related to this commit (0547a98) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1050/builds/955

Failed tests:

test_capi

Summary of the results of the build (if available):

== Tests result: FAILURE ==

338 tests OK.

10 slowest tests:

test_tokenize: 1 min 15 sec
test_unparse: 51.8 sec
test_io: 44.7 sec
test_lib2to3: 36.3 sec
test_zipfile: 26.2 sec
test_unicodedata: 25.8 sec
test_pickle: 23.5 sec
test_statistics: 19.8 sec
test_decimal: 19.5 sec
test_argparse: 18.7 sec

1 test failed:
test_capi

94 tests skipped:
test__xxsubinterpreters test_asyncgen test_asyncio
test_check_c_globals test_clinic test_cmd_line
test_concurrent_futures test_contextlib_async test_ctypes
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_doctest
test_docxmlrpc test_dtrace test_embed test_epoll test_faulthandler
test_fcntl test_file_eintr test_fork1 test_ftplib test_gdb
test_grp test_httplib test_httpservers test_idle test_imaplib
test_interpreters test_ioctl test_kqueue test_launcher test_lzma
test_mmap test_msilib test_multiprocessing_fork
test_multiprocessing_forkserver test_multiprocessing_main_handling
test_multiprocessing_spawn test_nis test_openpty test_ossaudiodev
test_pdb test_peg_generator test_perf_profiler test_poll
test_poplib test_pty test_pwd test_readline test_regrtest
test_repl test_resource test_select test_selectors test_smtplib
test_smtpnet test_socket test_socketserver test_spwd test_ssl
test_stable_abi_ctypes test_startfile test_subprocess
test_sys_settrace test_syslog test_tcl test_telnetlib test_tix
test_tkinter test_tools test_ttk test_ttk_textonly test_turtle
test_urllib2 test_urllib2_localnet test_urllib2net test_urllibnet
test_venv test_wait3 test_wait4 test_webbrowser test_winconsoleio
test_winreg test_winsound test_wmi test_wsgiref test_xmlrpc
test_xmlrpc_net test_xxlimited test_zipfile64
test_zipimport_support test_zoneinfo
0:22:53 load avg: 7.49
0:22:53 load avg: 7.49 Re-running failed tests is not supported with --python host runner option.

Total duration: 22 min 53 sec

Click to see traceback logs

remote: Enumerating objects: 23, done.        
remote: Counting objects:   4% (1/23)        
remote: Counting objects:   8% (2/23)        
remote: Counting objects:  13% (3/23)        
remote: Counting objects:  17% (4/23)        
remote: Counting objects:  21% (5/23)        
remote: Counting objects:  26% (6/23)        
remote: Counting objects:  30% (7/23)        
remote: Counting objects:  34% (8/23)        
remote: Counting objects:  39% (9/23)        
remote: Counting objects:  43% (10/23)        
remote: Counting objects:  47% (11/23)        
remote: Counting objects:  52% (12/23)        
remote: Counting objects:  56% (13/23)        
remote: Counting objects:  60% (14/23)        
remote: Counting objects:  65% (15/23)        
remote: Counting objects:  69% (16/23)        
remote: Counting objects:  73% (17/23)        
remote: Counting objects:  78% (18/23)        
remote: Counting objects:  82% (19/23)        
remote: Counting objects:  86% (20/23)        
remote: Counting objects:  91% (21/23)        
remote: Counting objects:  95% (22/23)        
remote: Counting objects: 100% (23/23)        
remote: Counting objects: 100% (23/23), done.        
remote: Compressing objects:  16% (1/6)        
remote: Compressing objects:  33% (2/6)        
remote: Compressing objects:  50% (3/6)        
remote: Compressing objects:  66% (4/6)        
remote: Compressing objects:  83% (5/6)        
remote: Compressing objects: 100% (6/6)        
remote: Compressing objects: 100% (6/6), done.        
remote: Total 12 (delta 11), reused 6 (delta 6), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to '0547a981ae413248b21a6bb0cb62dda7d236fe45'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 0547a981ae gh-99741: Clean Up the _xxsubinterpreters Module (gh-99940)
Switched to and reset branch 'main'

../../Objects/obmalloc.c:776:1: warning: always_inline function might not be inlinable [-Wattributes]
  776 | arena_map_get(pymem_block *p, int create)
      | ^~~~~~~~~~~~~

configure: ../../configure --prefix $(PWD)/target/host --with-pydebug --without-pydebug --with-emscripten-target=node --disable-wasm-dynamic-linking --enable-wasm-pthreads --build=x86_64-pc-linux-gnu --host=wasm32-unknown-emscripten --with-build-python=../build/python
configure: WARNING: using cross tools not prefixed with host triplet
mcc: error: no input files

make: make -j2 all
../../Python/initconfig.c:2347:27: warning: format specifies type 'wint_t' (aka 'int') but the argument has type 'wint_t' (aka 'unsigned int') [-Wformat]
    printf(usage_envvars, (wint_t)DELIM, (wint_t)DELIM, PYTHONHOMEHELP);
           ~~~~~~~~~~~~~  ^~~~~~~~~~~~~
../../Python/initconfig.c:138:18: note: format string is defined here
"PYTHONPATH   : '%lc'-separated list of directories prefixed to the\n"
                 ^~~
                 %c
../../Python/initconfig.c:2347:42: warning: format specifies type 'wint_t' (aka 'int') but the argument has type 'wint_t' (aka 'unsigned int') [-Wformat]
    printf(usage_envvars, (wint_t)DELIM, (wint_t)DELIM, PYTHONHOMEHELP);
           ~~~~~~~~~~~~~                 ^~~~~~~~~~~~~
../../Python/initconfig.c:141:58: note: format string is defined here
"PYTHONHOME   : alternate <prefix> directory (or <prefix>%lc<exec_prefix>).\n"
                                                         ^~~
                                                         %c
2 warnings generated.
../../Python/pytime.c:297:10: warning: implicit conversion from 'long long' to 'double' changes value from 9223372036854775807 to 9223372036854775808 [-Wimplicit-const-int-float-conversion]
    if (!_Py_InIntegralTypeRange(time_t, intpart)) {
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../Include/internal/pycore_pymath.h:72:45: note: expanded from macro '_Py_InIntegralTypeRange'
    (_Py_IntegralTypeMin(type) <= v && v <= _Py_IntegralTypeMax(type))
                                         ~~ ^~~~~~~~~~~~~~~~~~~~~~~~~
../../Include/internal/pycore_pymath.h:61:88: note: expanded from macro '_Py_IntegralTypeMax'
    (_Py_IS_TYPE_SIGNED(type) ? (((((type)1 << (sizeof(type)*CHAR_BIT - 2)) - 1) << 1) + 1) : ~(type)0)
                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~
../../Python/pytime.c:352:14: warning: implicit conversion from 'long long' to 'double' changes value from 9223372036854775807 to 9223372036854775808 [-Wimplicit-const-int-float-conversion]
        if (!_Py_InIntegralTypeRange(time_t, intpart)) {
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../Include/internal/pycore_pymath.h:72:45: note: expanded from macro '_Py_InIntegralTypeRange'
    (_Py_IntegralTypeMin(type) <= v && v <= _Py_IntegralTypeMax(type))
                                         ~~ ^~~~~~~~~~~~~~~~~~~~~~~~~
../../Include/internal/pycore_pymath.h:61:88: note: expanded from macro '_Py_IntegralTypeMax'
    (_Py_IS_TYPE_SIGNED(type) ? (((((type)1 << (sizeof(type)*CHAR_BIT - 2)) - 1) << 1) + 1) : ~(type)0)
                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~
../../Python/pytime.c:518:10: warning: implicit conversion from 'long long' to 'double' changes value from 9223372036854775807 to 9223372036854775808 [-Wimplicit-const-int-float-conversion]
    if (!_Py_InIntegralTypeRange(_PyTime_t, d)) {
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../Include/internal/pycore_pymath.h:72:45: note: expanded from macro '_Py_InIntegralTypeRange'
    (_Py_IntegralTypeMin(type) <= v && v <= _Py_IntegralTypeMax(type))
                                         ~~ ^~~~~~~~~~~~~~~~~~~~~~~~~
../../Include/internal/pycore_pymath.h:61:88: note: expanded from macro '_Py_IntegralTypeMax'
    (_Py_IS_TYPE_SIGNED(type) ? (((((type)1 << (sizeof(type)*CHAR_BIT - 2)) - 1) << 1) + 1) : ~(type)0)
                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~
3 warnings generated.
../../Modules/expat/xmlparse.c:3116:9: warning: code will never be executed [-Wunreachable-code]
        parser->m_characterDataHandler(parser->m_handlerArg, parser->m_dataBuf,
        ^~~~~~
../../Modules/expat/xmlparse.c:3115:16: note: silence by adding parentheses to mark code as explicitly dead
      else if (0 && parser->m_characterDataHandler)
               ^
               /* DISABLES CODE */ ( )
../../Modules/expat/xmlparse.c:4059:9: warning: code will never be executed [-Wunreachable-code]
        parser->m_characterDataHandler(parser->m_handlerArg, parser->m_dataBuf,
        ^~~~~~
../../Modules/expat/xmlparse.c:4058:16: note: silence by adding parentheses to mark code as explicitly dead
      else if (0 && parser->m_characterDataHandler)
               ^
               /* DISABLES CODE */ ( )
../../Modules/expat/xmlparse.c:7703:11: warning: format specifies type 'int' but the argument has type 'ptrdiff_t' (aka 'long') [-Wformat]
          bytesMore, (account == XML_ACCOUNT_DIRECT) ? "DIR" : "EXP",
          ^~~~~~~~~
3 warnings generated.
../../Modules/socketmodule.c:4031:33: warning: comparison of integers of different signs: 'unsigned long' and 'long' [-Wsign-compare]
         cmsgh != NULL; cmsgh = CMSG_NXTHDR(&msg, cmsgh)) {
                                ^~~~~~~~~~~~~~~~~~~~~~~~
/opt/buildbot/.emscripten_cache/sysroot/include/sys/socket.h:356:44: note: expanded from macro 'CMSG_NXTHDR'
        __CMSG_LEN(cmsg) + sizeof(struct cmsghdr) >= __MHDR_END(mhdr) - (unsigned char *)(cmsg) \
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../Modules/socketmodule.c:4084:33: warning: comparison of integers of different signs: 'unsigned long' and 'long' [-Wsign-compare]
         cmsgh != NULL; cmsgh = CMSG_NXTHDR(&msg, cmsgh)) {
                                ^~~~~~~~~~~~~~~~~~~~~~~~
/opt/buildbot/.emscripten_cache/sysroot/include/sys/socket.h:356:44: note: expanded from macro 'CMSG_NXTHDR'
        __CMSG_LEN(cmsg) + sizeof(struct cmsghdr) >= __MHDR_END(mhdr) - (unsigned char *)(cmsg) \
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../Modules/socketmodule.c:4708:54: warning: comparison of integers of different signs: 'unsigned long' and 'long' [-Wsign-compare]
            cmsgh = (i == 0) ? CMSG_FIRSTHDR(&msg) : CMSG_NXTHDR(&msg, cmsgh);
                                                     ^~~~~~~~~~~~~~~~~~~~~~~~
/opt/buildbot/.emscripten_cache/sysroot/include/sys/socket.h:356:44: note: expanded from macro 'CMSG_NXTHDR'
        __CMSG_LEN(cmsg) + sizeof(struct cmsghdr) >= __MHDR_END(mhdr) - (unsigned char *)(cmsg) \
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3 warnings generated.
../../Modules/_sqlite/connection.c:2198:19: warning: result of comparison of constant 9223372036854775807 with expression of type 'Py_ssize_t' (aka 'long') is always false [-Wtautological-constant-out-of-range-compare]
    if (data->len > 9223372036854775807) {  // (1 << 63) - 1
        ~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~
1 warning generated.
mcc: warning: USE_PTHREADS + ALLOW_MEMORY_GROWTH may run non-wasm code slowly, see https://github.com/WebAssembly/design/issues/1271 [-Wpthreads-mem-growth]

make: *** [Makefile:1860: buildbottest] Error 2

Cannot open file '/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/build/build_oot/host/test-results.xml' for upload

…ngh-99940)" This reverts commit 0547a98.

ericsnowcurrently added the skip news label Dec 2, 2022

bedevere-bot mentioned this pull request Dec 2, 2022

Implement Multi-Phase Init for _xxsubinterpreters #99741

Closed

bedevere-bot added the awaiting core review label Dec 2, 2022

ericsnowcurrently added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Dec 2, 2022

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Dec 2, 2022

ericsnowcurrently added 18 commits Dec 2, 2022

Use a C11 struct initializer.

3fa77fb

Clean up PyInit__xxsubinterpreters().

f2f00aa

Clean up channel_exceptions_init() and interp_exceptions_init().

bb84f38

Add _globals_fini() and only do init/fini once.

45e1f25

Factor out module_exec().

0efa456

Fix add_new_exception().

bd841f2

Do not rely on exception globals in low-level functions.

36e1239

Import _xxsubinterpreters in a subinterpreter if necessary.

51645f7

Ensure that the XID data gets released.

db57245

Return an error code from _PyCrossInterpreterData_Release().

4f9baf0

Clean up for ChannelIDType.

27d9b35

Always decrement _globals.module_count.

c4aa646

Decref the module when appropriate.

5cb73fa

Add some asserts.

5af58e8

Clear the registered type during module cleanup.

f555fbe

Add a dedicated ChannelIDTests.test_shareable().

ad000ad

Add a TODO comment.

60a78ab

Update a TODO comment.

012d61c

ericsnowcurrently force-pushed the clean-up-_xxsubinterpreters branch from f5e53ef to 012d61c Compare Dec 2, 2022

ericsnowcurrently merged commit 0547a98 into python:main Dec 2, 2022
13 checks passed

bedevere-bot removed the awaiting core review label Dec 2, 2022

ericsnowcurrently deleted the clean-up-_xxsubinterpreters branch Dec 2, 2022

sobolevn mentioned this pull request Dec 4, 2022

New warnings: 'function': conversion from 'int64_t' to 'int', possible loss of data #99984

Closed

ramvikrams added a commit to ramvikrams/cpython that referenced this pull request Dec 5, 2022

Revert "pythongh-99741: Clean Up the _xxsubinterpreters Module (pytho…

79983f0

…ngh-99940)" This reverts commit 0547a98.

ericsnowcurrently mentioned this pull request Dec 5, 2022

gh-99984: Fix Compiler Warnings #100036

Merged

gh-99741: Clean Up the _xxsubinterpreters Module #99940

gh-99741: Clean Up the _xxsubinterpreters Module #99940

ericsnowcurrently commented Dec 2, 2022 •

edited by bedevere-bot

bedevere-bot commented Dec 2, 2022

bedevere-bot commented Dec 2, 2022

gh-99741: Clean Up the _xxsubinterpreters Module #99940

gh-99741: Clean Up the _xxsubinterpreters Module #99940

Conversation

ericsnowcurrently commented Dec 2, 2022 • edited by bedevere-bot

bedevere-bot commented Dec 2, 2022

bedevere-bot commented Dec 2, 2022

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

ericsnowcurrently commented Dec 2, 2022 •

edited by bedevere-bot