bpo-43950: Add option to opt-out of PEP-657 #27023

ammaraskar · 2021-07-04T19:32:47Z

To be merged after #26958

This adds the opt-out option and disables the compiler's column information and traceback printing.

https://bugs.python.org/issue43950

Lib/test/test_zipimport.py

Objects/codeobject.c

Lib/test/test_code.py

Python/sysmodule.c

pablogsal · 2021-07-05T16:30:37Z

Btw, do we have a test that checks that the flag works?

ammaraskar · 2021-07-05T16:30:53Z

Btw, do we have a test that checks that the flag works?

Not yet :)

ammaraskar · 2021-07-05T18:20:29Z

@pablogsal After making these changes, I am a little worried about the sys.flags.no_debug_ranges conditionals, I fear that without it being automatically tested somewhere we could end up with code-rot in the test suite. Would it be possible to run a buildbot with -X no_debug_ranges just to ensure that no bugs in the opted-out code paths pop up.

pablogsal · 2021-07-05T18:46:56Z

Would it be possible to run a buildbot with -X no_debug_ranges just to ensure that no bugs in the opted-out code paths pop up.

Hmmm, is possible, but I would prefer to explicitly test that the flag + env work in some specific tests using a subprocess and that would only left the skip() as conditional. Is true that that could break, but adding an entire buildbot is a bit of a big deal, although we can explore that.

ammaraskar · 2021-07-05T19:13:09Z

Btw, do we have a test that checks that the flag works?

Just added some, please take a look.

Hmmm, is possible, but I would prefer to explicitly test that the flag + env work in some specific tests using a subprocess

For example running test_traceback.py with the flag set? That would certainly work too, I like that approach.

The last remaining component for this PR is to make marshal.c replace the tables with None if the config option is set and test that.

pablogsal · 2021-07-05T19:17:29Z

The last remaining component for this PR is to make marshal.c replace the tables with None if the config option is set and test that.

Maybe we should do the logic in the code object constructor and that way we don't need to deal with marshal.c and the compiler.

ammaraskar · 2021-07-05T21:42:44Z

Maybe we should do the logic in the code object constructor and that way we don't need to deal with marshal.c and the compiler.

Great idea, did this in the common code path in codeobject.c :)
That makes the code a lot simpler.

Added tests to make sure de-marshaling with the flag results in None for the endlinetable and columntable.

Hmmm, is possible, but I would prefer to explicitly test that the flag + env work in some specific tests using a subprocess

Going back to this, do you know if there is an example of such a test that already exists in the suite? Where should it go? A new test_traceback_no_debug_range.py or similar?

pablogsal · 2021-07-07T13:21:37Z

We need more rebasing :(

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com> Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com> Co-authored-by: Ammar Askar <ammar@ammaraskar.com>

ammaraskar · 2021-07-07T14:13:44Z

Rebased!

bedevere-bot · 2021-07-07T14:14:00Z

🤖 New build scheduled with the buildbot fleet by @ammaraskar for commit d34f7a9 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

ammaraskar · 2021-07-07T15:40:37Z

buildbot/AMD64 Ubuntu Shared PR has a weird failure...

/srv/buildbot/buildarea/pull_request.bolen-ubuntu/build/python: error while loading shared libraries: libpython3.11d.so.1.0: cannot open shared object file: No such file or directory

I wonder if we need a @requires_* on it or if it's just a buildbot problem?

ammaraskar · 2021-07-07T16:43:51Z

Nope, no @requires annotation is needed to use assert_python_ok and assert_python_failure. This seems to just be a problem with the buildbot.

ammaraskar · 2021-07-07T19:02:34Z

@pablogsal all lights are green, anything else for this one?

pablogsal · 2021-07-07T19:07:27Z

@pablogsal all lights are green, anything else for this one?

Nop. Great job! 👌

bedevere-bot · 2021-07-07T19:27:38Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 FreeBSD Shared 3.x has failed when building commit 4823d9a.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/483/builds/469) and take a look at the build logs.
Check if the failure is related to this commit (4823d9a) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/483/builds/469

Failed tests:

test_marshal
test_traceback
test_code

Failed subtests:

test_endline_and_columntable_none_when_no_debug_ranges_env - test.test_code.CodeTest
test_endline_and_columntable_none_when_no_debug_ranges - test.test_code.CodeTest
test_no_caret_with_no_debug_ranges_flag - test.test_traceback.TracebackCases
test_no_columntable_and_endlinetable_with_no_debug_ranges - test.test_marshal.CodeTestCase
test_no_caret_with_no_debug_ranges_flag_python_traceback - test.test_traceback.TracebackCases

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

404 tests OK.

10 slowest tests:

test_smtpnet: 6 min 6 sec
test_nntplib: 3 min 45 sec
test_concurrent_futures: 3 min 33 sec
test_multiprocessing_spawn: 2 min 2 sec
test_multiprocessing_forkserver: 2 min 2 sec
test_unparse: 2 min 2 sec
test_urllib2: 2 min 1 sec
test_tokenize: 1 min 42 sec
test_multiprocessing_fork: 1 min 34 sec
test_asyncio: 1 min 23 sec

3 tests failed:
test_code test_marshal test_traceback

20 tests skipped:
test_dbm_gnu test_devpoll test_epoll test_idle test_ioctl
test_msilib test_ossaudiodev test_readline test_spwd
test_startfile test_tcl test_tix test_tk test_ttk_guionly
test_ttk_textonly test_turtle test_winconsoleio test_winreg
test_winsound test_zipfile64

3 re-run tests:
test_code test_marshal test_traceback

Total duration: 14 min 59 sec

Click to see traceback logs

TracebackErrorCaretTests) ... ok


TracebackCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_traceback.py", line 110, in test_no_caret_with_no_debug_ranges_flag_python_traceback
    _, _, stderr = assert_python_ok(
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '@test_77265_tmpæ']


TracebackCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_traceback.py", line 110, in test_no_caret_with_no_debug_ranges_flag_python_traceback
    _, _, stderr = assert_python_ok(
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '@test_80109_tmpæ']


Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_marshal.py", line 150, in test_no_columntable_and_endlinetable_with_no_debug_ranges
    assert_python_ok('-X', 'no_debug_ranges',
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '-c', "\nimport sys\nimport marshal\nwith open(sys.argv[1], 'rb') as f:\n    co = marshal.load(f)\n\n    assert co.co_endlinetable is None\n    assert co.co_columntable is None\n", '@test_77265_tmpæ']


TracebackFormatTests) ... ok


TracebackCases) ... ok


Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_code.py", line 398, in test_endline_and_columntable_none_when_no_debug_ranges_env
    assert_python_ok('-c', code, PYTHONNODEBUGRANGES='1', __cleanenv=True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/python', '-X', 'faulthandler', '-c', '\ndef f():\n    pass\n\nassert f.__code__.co_endlinetable is None\nassert f.__code__.co_columntable is None\n']


TracebackErrorLocationCaretTests) ... ok


TracebackException) ... ok


TracebackCases) ... FAIL


Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_code.py", line 387, in test_endline_and_columntable_none_when_no_debug_ranges
    assert_python_ok('-X', 'no_debug_ranges', '-c', code, __cleanenv=True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '-c', '\ndef f():\n    pass\n\nassert f.__code__.co_endlinetable is None\nassert f.__code__.co_columntable is None\n']


TracebackCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_traceback.py", line 90, in test_no_caret_with_no_debug_ranges_flag
    self.assertEqual(len(lines), 4)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 1 != 4


Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_marshal.py", line 150, in test_no_columntable_and_endlinetable_with_no_debug_ranges
    assert_python_ok('-X', 'no_debug_ranges',
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '-c', "\nimport sys\nimport marshal\nwith open(sys.argv[1], 'rb') as f:\n    co = marshal.load(f)\n\n    assert co.co_endlinetable is None\n    assert co.co_columntable is None\n", '@test_82013_tmpæ']

bedevere-bot · 2021-07-07T19:39:08Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Ubuntu Shared 3.x has failed when building commit 4823d9a.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/506/builds/502) and take a look at the build logs.
Check if the failure is related to this commit (4823d9a) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/506/builds/502

Failed tests:

test_marshal
test_traceback
test_code

Failed subtests:

test_endline_and_columntable_none_when_no_debug_ranges_env - test.test_code.CodeTest
test_endline_and_columntable_none_when_no_debug_ranges - test.test_code.CodeTest
test_no_caret_with_no_debug_ranges_flag - test.test_traceback.TracebackCases
test_no_columntable_and_endlinetable_with_no_debug_ranges - test.test_marshal.CodeTestCase
test_no_caret_with_no_debug_ranges_flag_python_traceback - test.test_traceback.TracebackCases

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

407 tests OK.

10 slowest tests:

test_concurrent_futures: 4 min 6 sec
test_gdb: 3 min 30 sec
test_peg_generator: 3 min 6 sec
test_capi: 2 min 42 sec
test_multiprocessing_spawn: 2 min 34 sec
test_asyncio: 1 min 56 sec
test_unparse: 1 min 48 sec
test_lib2to3: 1 min 33 sec
test_multiprocessing_forkserver: 1 min 28 sec
test_tokenize: 1 min 16 sec

3 tests failed:
test_code test_marshal test_traceback

17 tests skipped:
test_devpoll test_idle test_ioctl test_kqueue test_msilib
test_ossaudiodev test_startfile test_tcl test_tix test_tk
test_ttk_guionly test_ttk_textonly test_turtle test_winconsoleio
test_winreg test_winsound test_zipfile64

3 re-run tests:
test_code test_marshal test_traceback

Total duration: 26 min 47 sec

Click to see traceback logs

TracebackErrorCaretTests) ... ok


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_marshal.py", line 150, in test_no_columntable_and_endlinetable_with_no_debug_ranges
    assert_python_ok('-X', 'no_debug_ranges',
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 127
command line: ['/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '-c', "\nimport sys\nimport marshal\nwith open(sys.argv[1], 'rb') as f:\n    co = marshal.load(f)\n\n    assert co.co_endlinetable is None\n    assert co.co_columntable is None\n", '@test_15164_tmpæ']


TracebackCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_traceback.py", line 110, in test_no_caret_with_no_debug_ranges_flag_python_traceback
    _, _, stderr = assert_python_ok(
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 127
command line: ['/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '@test_15685_tmpæ']


TracebackCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_traceback.py", line 90, in test_no_caret_with_no_debug_ranges_flag
    self.assertEqual(len(lines), 4)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 1 != 4


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_marshal.py", line 150, in test_no_columntable_and_endlinetable_with_no_debug_ranges
    assert_python_ok('-X', 'no_debug_ranges',
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 127
command line: ['/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '-c', "\nimport sys\nimport marshal\nwith open(sys.argv[1], 'rb') as f:\n    co = marshal.load(f)\n\n    assert co.co_endlinetable is None\n    assert co.co_columntable is None\n", '@test_29458_tmpæ']


TracebackFormatTests) ... ok


TracebackCases) ... ok


TracebackCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_traceback.py", line 110, in test_no_caret_with_no_debug_ranges_flag_python_traceback
    _, _, stderr = assert_python_ok(
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 127
command line: ['/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '@test_29458_tmpæ']


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_code.py", line 398, in test_endline_and_columntable_none_when_no_debug_ranges_env
    assert_python_ok('-c', code, PYTHONNODEBUGRANGES='1', __cleanenv=True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 127
command line: ['/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python', '-X', 'faulthandler', '-c', '\ndef f():\n    pass\n\nassert f.__code__.co_endlinetable is None\nassert f.__code__.co_columntable is None\n']


TracebackErrorLocationCaretTests) ... ok


TracebackException) ... ok


TracebackCases) ... FAIL


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_code.py", line 387, in test_endline_and_columntable_none_when_no_debug_ranges
    assert_python_ok('-X', 'no_debug_ranges', '-c', code, __cleanenv=True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 127
command line: ['/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '-c', '\ndef f():\n    pass\n\nassert f.__code__.co_endlinetable is None\nassert f.__code__.co_columntable is None\n']

pablogsal · 2021-07-07T20:03:04Z

@ammaraskar @isidentical Hummm, have we bumped the magic number? This buildbot failure seems quite related. I'm in some meetings right now and cannot check but may need to revert:(

isidentical · 2021-07-07T20:06:20Z

AFAIK we did. Checking the failures now...

ammaraskar · 2021-07-07T20:10:44Z

The errors on both are:

/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python: error while loading shared libraries: libpython3.11d.so.1.0: cannot open shared object file: No such file or directory

stderr:
---
ld-elf.so.1: Shared object "libpython3.11d.so.1.0" not found, required by "python"
---

when running assert_python_ok and assert_python_failure and both buildbots have "Shared" in the name.

ammaraskar · 2021-07-07T20:12:08Z

The confusing thing is, it only happens with the new tests, existing assert_python_ok tests are working just fine.

isidentical · 2021-07-07T20:14:46Z

To reproduce locally;

$ git clean -fxd
$ ./configure --with-pydebug --enable-shared -C
$ make -j
$ LD_LIBRARY_PATH=$PWD ./python -m test -v test_traceback

isidentical · 2021-07-07T20:17:06Z

We don't preserve the original environment, so shared builds fail. See here:

cpython/Lib/test/support/script_helper.py

Lines 117 to 121 in bb3e0c2

    
           else: 
        
               # Need to preserve the original environment, for in-place testing of 
        
               # shared library builds. 
        
               env = os.environ.copy()

isidentical · 2021-07-07T20:17:43Z

I can confirm that setting __cleanenv to False fixes the issue

ammaraskar · 2021-07-07T20:19:27Z

Aah, let's remove __cleanenv in that case. I set it primarily because I thought it would avoid propagation of the testing Interpreter's config.

isidentical · 2021-07-07T20:20:24Z

Also see this example usage of how it is used (it creates a new env based on the existing one) if you want to isolate these;

    def get_hash(self, repr_, seed=None):
        env = os.environ.copy()
        env['__cleanenv'] = True  # signal to assert_python not to do a copy
                                  # of os.environ on its own
        if seed is not None:
            env['PYTHONHASHSEED'] = str(seed)
        else:
            env.pop('PYTHONHASHSEED', None)
        out = assert_python_ok(
            '-c', self.get_hash_command(repr_),
            **env)
        stdout = out[1].strip()
        return int(stdout)

the-knights-who-say-ni added the CLA signed label Jul 4, 2021

bedevere-bot added the awaiting review label Jul 4, 2021

ammaraskar added the skip news label Jul 4, 2021

ammaraskar force-pushed the colnotab:pep657-config branch 8 times, most recently from db3cbe3 to 6b3c6d7 Jul 4, 2021

ammaraskar reviewed Jul 5, 2021

View changes

Lib/test/test_zipimport.py Outdated Show resolved Hide resolved

ammaraskar reviewed Jul 5, 2021

View changes

Objects/codeobject.c Show resolved Hide resolved

ammaraskar force-pushed the colnotab:pep657-config branch 2 times, most recently from 1a555a1 to 0268bfe Jul 5, 2021

isidentical reviewed Jul 5, 2021

View changes

Lib/test/test_code.py Outdated Show resolved Hide resolved

ammaraskar mentioned this pull request Jul 5, 2021

bpo-43950: Print columns in tracebacks (PEP 657) #26958

Merged

pablogsal reviewed Jul 5, 2021

View changes

Python/sysmodule.c Outdated Show resolved Hide resolved

ammaraskar force-pushed the colnotab:pep657-config branch from ed65e7e to d789c9a Jul 5, 2021

ammaraskar marked this pull request as ready for review Jul 5, 2021

ammaraskar requested review from brettcannon, encukou and ericsnowcurrently as code owners Jul 5, 2021

isidentical approved these changes Jul 6, 2021

View changes

bedevere-bot added awaiting merge and removed awaiting review labels Jul 6, 2021

bpo-43950: Add option to opt-out of PEP-657

d34f7a9

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com> Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com> Co-authored-by: Ammar Askar <ammar@ammaraskar.com>

ammaraskar force-pushed the colnotab:pep657-config branch from caa72dd to d34f7a9 Jul 7, 2021

ammaraskar added the 🔨 test-with-buildbots label Jul 7, 2021

bedevere-bot removed the 🔨 test-with-buildbots label Jul 7, 2021

pablogsal approved these changes Jul 7, 2021

View changes

bedevere-bot removed the awaiting merge label Jul 7, 2021

ammaraskar mentioned this pull request Jul 7, 2021

Remove __cleanenv from PEP-657 tests #27060

Merged

python / cpython

bpo-43950: Add option to opt-out of PEP-657 #27023

bpo-43950: Add option to opt-out of PEP-657 #27023

ammaraskar commented Jul 4, 2021 •

edited by bedevere-bot

pablogsal commented Jul 5, 2021

ammaraskar commented Jul 5, 2021

ammaraskar commented Jul 5, 2021 •

edited

pablogsal commented Jul 5, 2021

ammaraskar commented Jul 5, 2021

pablogsal commented Jul 5, 2021

ammaraskar commented Jul 5, 2021 •

edited

pablogsal commented Jul 7, 2021

ammaraskar commented Jul 7, 2021

bedevere-bot commented Jul 7, 2021

ammaraskar commented Jul 7, 2021

ammaraskar commented Jul 7, 2021

ammaraskar commented Jul 7, 2021

pablogsal commented Jul 7, 2021

bedevere-bot commented Jul 7, 2021

bedevere-bot commented Jul 7, 2021

pablogsal commented Jul 7, 2021

isidentical commented Jul 7, 2021

ammaraskar commented Jul 7, 2021

ammaraskar commented Jul 7, 2021

isidentical commented Jul 7, 2021

isidentical commented Jul 7, 2021

isidentical commented Jul 7, 2021

ammaraskar commented Jul 7, 2021

isidentical commented Jul 7, 2021

python / cpython

Sponsor python/cpython

bpo-43950: Add option to opt-out of PEP-657 #27023

bpo-43950: Add option to opt-out of PEP-657 #27023

Conversation

ammaraskar commented Jul 4, 2021 • edited by bedevere-bot

pablogsal commented Jul 5, 2021

ammaraskar commented Jul 5, 2021

ammaraskar commented Jul 5, 2021 • edited

pablogsal commented Jul 5, 2021

ammaraskar commented Jul 5, 2021

pablogsal commented Jul 5, 2021

ammaraskar commented Jul 5, 2021 • edited

pablogsal commented Jul 7, 2021

ammaraskar commented Jul 7, 2021

bedevere-bot commented Jul 7, 2021

ammaraskar commented Jul 7, 2021

ammaraskar commented Jul 7, 2021

ammaraskar commented Jul 7, 2021

pablogsal commented Jul 7, 2021

bedevere-bot commented Jul 7, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented Jul 7, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

pablogsal commented Jul 7, 2021

isidentical commented Jul 7, 2021

ammaraskar commented Jul 7, 2021

ammaraskar commented Jul 7, 2021

isidentical commented Jul 7, 2021

isidentical commented Jul 7, 2021

isidentical commented Jul 7, 2021

ammaraskar commented Jul 7, 2021

isidentical commented Jul 7, 2021

ammaraskar commented Jul 4, 2021 •

edited by bedevere-bot

ammaraskar commented Jul 5, 2021 •

edited

ammaraskar commented Jul 5, 2021 •

edited