bpo-39573: Py_TYPE becomes a static inline function #28128

vstinner · 2021-09-02T15:46:05Z

Convert the Py_TYPE() and Py_SIZE() macros to static inline
functions. The Py_SET_TYPE() and Py_SET_SIZE() functions must now be
used to set an object type and size.

https://bugs.python.org/issue39573

vstinner · 2021-09-02T15:47:32Z

I already pushed this change in June: f3fa63e

But it was reverted since it broke a Windows debug buildbot and I failed to find time to analyze why: https://bugs.python.org/issue39573#msg395205

bedevere-bot · 2021-09-02T16:20:37Z

🤖 New build scheduled with the buildbot fleet by @vstinner for commit d57df22 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

vstinner · 2021-09-03T11:21:45Z

buildbot/AMD64 Windows10 PR failed:

Py_DEBUG: Yes (sys.gettotalrefcount() present)
...
Windows fatal exception: stack overflow
Current thread 0x000012f8 (most recent call first):
  File "D:\buildarea\pull_request.bolen-windows10\build\lib\test\test_exceptions.py", line 1313 in test_recursion_in_except_handler

buildbot/AMD64 Windows10 Pro PR: failed

Py_DEBUG: Yes (sys.gettotalrefcount() present)
...
Windows fatal exception: stack overflow
Current thread 0x000022e8 (most recent call first):
  File "C:\buildbot.python.org\pull_request.kloth-win64\build\lib\test\test_exceptions.py", line 1313 in test_recursion_in_except_handler

buildbot/AMD64 Arch Linux Asan Debug PR:

(...)
  File "/buildbot/buildarea/pull_request.pablogsal-arch-x86_64.asan_debug/build/Lib/subprocess.py", line 966, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/pull_request.pablogsal-arch-x86_64.asan_debug/build/Lib/subprocess.py", line 1775, in _execute_child
    self.pid = _posixsubprocess.fork_exec(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: subprocess not supported for isolated subinterpreters

Hum, usually this error comes from an incremental build: a C structure changed, the compiler reused old object files (.o).

buildbot/x86 Gentoo Non-Debug with X PR:

0:36:34 load avg: 8.91 [239/427/1] test_peg_generator crashed (Exit code 1)
Timeout (0:25:00)!
Thread 0xb7bc1700 (most recent call first):
  File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1896 in _try_wait
  File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1938 in _wait
  File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1204 in wait
  File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.nondebug/build/Lib/distutils/spawn.py", line 80 in spawn
  File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.nondebug/build/Lib/distutils/ccompiler.py", line 910 in spawn
  File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.nondebug/build/Lib/distutils/unixccompiler.py", line 117 in _compile
  File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.nondebug/build/Lib/distutils/ccompiler.py", line 574 in compile
  File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 529 in build_extension
  File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 474 in _build_extensions_serial
  File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 449 in build_extensions
  File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 340 in run
  File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.nondebug/build/Tools/peg_generator/pegen/build.py", line 93 in compile_c_extension
  File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.nondebug/build/Tools/peg_generator/pegen/testutil.py", line 105 in generate_parser_c_extension
  File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 93 in build_extension
  File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 96 in run_test
  File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 257 in test_return_stmt_noexpr_action
(...)

It seems like the timeout is just too low for this slow buildbot. On a previous build, the test took 21 min which is close to the timeout of 25 min:

https://buildbot.python.org/all/#/builders/344/builds/151

10 slowest tests:
- test_peg_generator: 21 min 46 sec

Fidget-Spinner · 2021-09-03T12:05:03Z

For the test_exceptions bug, there's an issue at https://bugs.python.org/issue44348. Some time ago I proposed that we need to increase stack size on windows.

vstinner · 2021-09-03T13:15:22Z

I created python/buildmaster-config#262 to increase "/x86 Gentoo Non-Debug with X" buildbot timeout: python/buildmaster-config@eeff117

@Fidget-Spinner: "For the test_exceptions bug, there's an issue at https://bugs.python.org/issue44348. Some time ago I proposed that we need to increase stack size on windows."

Right, I'm aware of that. But I was optimistic and hoped that the bug gone magically :-)

For test_exceptions, my notes on the stack size and stack overflow on a recursion error on Windows: https://pythondev.readthedocs.io/unstable_tests.html#unlimited-recursion

vstinner · 2021-09-03T13:55:35Z

I created https://bugs.python.org/issue45094 for the test_exceptions stack overflow.

vstinner · 2021-09-03T23:28:16Z

I converted this PR to a draft until https://bugs.python.org/issue45094 is fixed.

I pushed changes to test if Py_ALWAYS_INLINE (PR #28141) would fix the test_exceptions on Windows when Python is built in debug mode.

vstinner · 2021-09-03T23:47:16Z

Oh, it seems like adding __declspec(noinline) didn't fix test_exceptions, or my PR has an issue, or __declspec(noinline) is ignored when Python is built in debug mode.

Windows fatal exception: stack overflow
0:06:07 load avg: 8.17 [236/427/1] test_exceptions crashed (Exit code 3221225725)

Current thread 0x00000348 (most recent call first):
  File "D:\a\cpython\cpython\lib\test\test_exceptions.py", line 1313 in test_recursion_in_except_handler

vstinner · 2021-09-06T15:35:53Z

New approach: bpo-45115: Enable optimiaztions on Windows debug build. I included PR #28181 in this PR, to see if PR #28181 fix test_exceptions.

vstinner · 2021-09-07T16:47:31Z

I pushed a change to run the test suite with a debug mode on Windows (commit " [WIP] GitHub Action: Windows uses a debug build"): tests now pass thanks to my test_exceptions fix!

Convert the Py_TYPE() and Py_SIZE() macros to static inline functions. The Py_SET_TYPE() and Py_SET_SIZE() functions must now be used to set an object type and size.

vstinner · 2021-09-07T16:48:36Z

I removed the debug commits of this PR (since tests pass) and I removed the Draft status.

Don't use _PyObject_CAST() which doesn't exist in old Python versions.

bedevere-bot · 2021-09-07T17:06:04Z

🤖 New build scheduled with the buildbot fleet by @vstinner for commit d411467 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

vstinner · 2021-09-07T19:05:35Z

buildbot/AMD64 Arch Linux Asan Debug PR

The compile step failed with "RuntimeError: subprocess not supported for isolated subinterpreters". This issue is unrelated to this PR.

the-knights-who-say-ni added the CLA signed label Sep 2, 2021

bedevere-bot added the awaiting core review label Sep 2, 2021

vstinner added the 🔨 test-with-buildbots label Sep 2, 2021

bedevere-bot removed the 🔨 test-with-buildbots label Sep 2, 2021

vstinner marked this pull request as draft Sep 3, 2021

vstinner mentioned this pull request Sep 3, 2021

bpo-44348: Use debug builds for windows CI #26627

Closed

vstinner force-pushed the vstinner:py_type_func branch from f330ea1 to 413c72e Sep 3, 2021

vstinner mentioned this pull request Sep 4, 2021

bpo-45094: Add Py_ALWAYS_INLINE macro #28141

Closed

vstinner force-pushed the vstinner:py_type_func branch from 413c72e to 515b51f Sep 6, 2021

vstinner mentioned this pull request Sep 6, 2021

bpo-45115: Enable optimiaztions on Windows debug build #28181

Closed

vstinner force-pushed the vstinner:py_type_func branch from 515b51f to 709c77b Sep 7, 2021

bpo-39573: Py_TYPE becomes a static inline function

cb33989

Convert the Py_TYPE() and Py_SIZE() macros to static inline functions. The Py_SET_TYPE() and Py_SET_SIZE() functions must now be used to set an object type and size.

vstinner force-pushed the vstinner:py_type_func branch from 709c77b to cb33989 Sep 7, 2021

vstinner marked this pull request as ready for review Sep 7, 2021

Fix the doc

d411467

Don't use _PyObject_CAST() which doesn't exist in old Python versions.

vstinner added the 🔨 test-with-buildbots label Sep 7, 2021

bedevere-bot removed the 🔨 test-with-buildbots label Sep 7, 2021

bedevere-bot removed the awaiting core review label Sep 8, 2021

vstinner deleted the vstinner:py_type_func branch Sep 8, 2021

python / cpython Public

bpo-39573: Py_TYPE becomes a static inline function #28128

bpo-39573: Py_TYPE becomes a static inline function #28128

vstinner commented Sep 2, 2021 •

edited by bedevere-bot

vstinner commented Sep 2, 2021

bedevere-bot commented Sep 2, 2021

vstinner commented Sep 3, 2021

Fidget-Spinner commented Sep 3, 2021

vstinner commented Sep 3, 2021

vstinner commented Sep 3, 2021

vstinner commented Sep 3, 2021

vstinner commented Sep 3, 2021

vstinner commented Sep 6, 2021 •

edited by bedevere-bot

vstinner commented Sep 7, 2021

vstinner commented Sep 7, 2021

bedevere-bot commented Sep 7, 2021

vstinner commented Sep 7, 2021

python / cpython Public

Sponsor python/cpython

bpo-39573: Py_TYPE becomes a static inline function #28128

bpo-39573: Py_TYPE becomes a static inline function #28128

Conversation

vstinner commented Sep 2, 2021 • edited by bedevere-bot

vstinner commented Sep 2, 2021

bedevere-bot commented Sep 2, 2021

vstinner commented Sep 3, 2021

Fidget-Spinner commented Sep 3, 2021

vstinner commented Sep 3, 2021

vstinner commented Sep 3, 2021

vstinner commented Sep 3, 2021

vstinner commented Sep 3, 2021

vstinner commented Sep 6, 2021 • edited by bedevere-bot

vstinner commented Sep 7, 2021

vstinner commented Sep 7, 2021

bedevere-bot commented Sep 7, 2021

vstinner commented Sep 7, 2021

vstinner commented Sep 2, 2021 •

edited by bedevere-bot

vstinner commented Sep 6, 2021 •

edited by bedevere-bot