bpo-43049: Use io.IncrementalNewlineDecoder for doctest newline conversion #24359

pdonis · 2021-01-28T02:13:45Z

This PR is a follow-up to PR #17385 for bpo-1812 (now merged). During discussion on that issue the suggestion was made to investigate whether any resources already provided by the io module might be available to do the newline conversion in doctest that was corrected in that PR. It turns out there is: the io.IncrementalNewlineDecoder object does exactly the conversion that is needed. This PR implements using that object in place of the custom newline conversion that was done in the previous PR. The only change is in the relevant code in the doctest module.

https://bugs.python.org/issue43049

merwok · 2021-01-28T02:25:53Z

No new test is needed because existing tests already cover this code, is that correct?

pdonis · 2021-01-28T02:40:58Z

No new test is needed because existing tests already cover this code, is that correct?

Yes, that's correct.

github-actions · 2021-02-28T00:26:27Z

This PR is stale because it has been open for 30 days with no activity.

pdonis · 2021-02-28T01:51:08Z

Just saw the stale notification, bumping to get an update on merging this PR.

zware · 2021-03-02T17:04:35Z

I was leaving this for anyone more familiar with either doctest or io to review, but with neither forthcoming I'll go ahead and merge :)

bedevere-bot · 2021-03-02T18:17:47Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 RHEL8 3.x has failed when building commit b36349a.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/529/builds/691) and take a look at the build logs.
Check if the failure is related to this commit (b36349a) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/529/builds/691

Failed tests:

test_asyncio

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 4, done.        
remote: Counting objects:  25% (1/4)        
remote: Counting objects:  50% (2/4)        
remote: Counting objects:  75% (3/4)        
remote: Counting objects: 100% (4/4)        
remote: Counting objects: 100% (4/4), done.        
remote: Compressing objects:  33% (1/3)        
remote: Compressing objects:  66% (2/3)        
remote: Compressing objects: 100% (3/3)        
remote: Compressing objects: 100% (3/3), done.        
remote: Total 4 (delta 1), reused 1 (delta 1), pack-reused 0        
From https://github.com/python/cpython
 * branch                  master     -> FETCH_HEAD
Reset branch 'master'

/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/./setup.py:33: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  from distutils import log
/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Modules/_zoneinfo.c:2611:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
 module_free()
 ^~~~~~~~~~~
/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Modules/_xxtestfuzz/fuzzer.c:86:12: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
 static int init_struct_unpack() {
            ^~~~~~~~~~~~~~~~~~
/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Modules/_xxtestfuzz/fuzzer.c:149:12: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
 static int init_json_loads() {
            ^~~~~~~~~~~~~~~
/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Modules/_xxtestfuzz/fuzzer.c:195:12: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
 static int init_sre_compile() {
            ^~~~~~~~~~~~~~~~
/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Modules/_xxtestfuzz/fuzzer.c:287:12: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
 static int init_sre_match() {
            ^~~~~~~~~~~~~~
/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Modules/_xxtestfuzz/fuzzer.c:342:12: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
 static int init_csv_reader() {
            ^~~~~~~~~~~~~~~

/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/./setup.py:33: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  from distutils import log

/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/./setup.py:33: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  from distutils import log
test_tk skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_flock (__main__.FNTLEINTRTest) ... ok
test_lockf (__main__.FNTLEINTRTest) ... ok
test_read (__main__.OSEINTRTest) ... ok
test_wait (__main__.OSEINTRTest) ... ok
test_wait3 (__main__.OSEINTRTest) ... ok
test_wait4 (__main__.OSEINTRTest) ... ok
test_waitpid (__main__.OSEINTRTest) ... ok
test_write (__main__.OSEINTRTest) ... ok
test_devpoll (__main__.SelectEINTRTest) ... skipped 'need select.devpoll'
test_epoll (__main__.SelectEINTRTest) ... ok
test_kqueue (__main__.SelectEINTRTest) ... skipped 'need select.kqueue'
test_poll (__main__.SelectEINTRTest) ... ok
test_select (__main__.SelectEINTRTest) ... ok
test_sigtimedwait (__main__.SignalEINTRTest) ... ok
test_sigwaitinfo (__main__.SignalEINTRTest) ... ok
test_accept (__main__.SocketEINTRTest) ... ok
test_open (__main__.SocketEINTRTest) ... ok
test_os_open (__main__.SocketEINTRTest) ... ok
test_recv (__main__.SocketEINTRTest) ... ok
test_recvmsg (__main__.SocketEINTRTest) ... ok
test_send (__main__.SocketEINTRTest) ... ok
test_sendall (__main__.SocketEINTRTest) ... ok
test_sendmsg (__main__.SocketEINTRTest) ... ok
test_sleep (__main__.TimeEINTRTest) ... ok

----------------------------------------------------------------------
Ran 24 tests in 7.704s

OK (skipped=2)
test_tix skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_badargs (__main__.GeneralTest) ... ok
test_bound_methods (__main__.GeneralTest) ... ok
test_clear (__main__.GeneralTest) ... ok
test_exit (__main__.GeneralTest) ... ok
test_order (__main__.GeneralTest) ... ok
test_raise (__main__.GeneralTest) ... ok
test_raise_unnormalized (__main__.GeneralTest) ... ok
test_stress (__main__.GeneralTest) ... ok
test_unregister (__main__.GeneralTest) ... ok

----------------------------------------------------------------------
Ran 9 tests in 0.003s

OK
test_ioctl skipped -- Unable to open /dev/tty
test_msilib skipped -- No module named '_msi'
test_ttk_guionly skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_winreg skipped -- No module named 'winreg'
test_winconsoleio skipped -- test only relevant on win32
test_startfile skipped -- object <module 'os' from '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/os.py'> has no attribute 'startfile'
/tmp/tmpk842em4u.py:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  from distutils.util import byte_compile
/tmp/tmpwe9ow8i9.py:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  from distutils.util import byte_compile
test_ossaudiodev skipped -- [Errno 2] No such file or directory: '/dev/dsp'
test_kqueue skipped -- test works only on BSD
test_winsound skipped -- No module named 'winsound'
test_zipfile64 skipped -- test requires loads of disk-space bytes and a long time to run
test_devpoll skipped -- test works only on Solaris OS family
<string>:2: PendingDeprecationWarning: lib2to3 package is deprecated and may not be able to parse Python 3.10+
Timeout (0:15:00)!
Thread 0x0000ffff8e934970 (most recent call first):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/selectors.py", line 469 in select
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/asyncio/base_events.py", line 1845 in _run_once
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/asyncio/base_events.py", line 595 in run_forever
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/asyncio/base_events.py", line 628 in run_until_complete
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/test_asyncio/test_subprocess.py", line 442 in test_cancel_make_subprocess_transport_exec
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/case.py", line 549 in _callTestMethod
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/case.py", line 592 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/case.py", line 652 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/runner.py", line 176 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/support/__init__.py", line 959 in _run_suite
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/support/__init__.py", line 1082 in run_unittest
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/runtest.py", line 211 in _test_module
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/runtest.py", line 236 in _runtest_inner2
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/runtest.py", line 272 in _runtest_inner
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/runtest.py", line 142 in _runtest
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/runtest.py", line 195 in runtest
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/runtest_mp.py", line 81 in run_tests_worker
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/main.py", line 659 in _main
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/main.py", line 639 in main
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/main.py", line 717 in main
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/regrtest.py", line 43 in _main
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/regrtest.py", line 47 in <module>
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/runpy.py", line 87 in _run_code
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/runpy.py", line 197 in _run_module_as_main
Timeout (0:15:00)!
Thread 0x0000ffff9c0d4970 (most recent call first):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/selectors.py", line 469 in select
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/asyncio/base_events.py", line 1845 in _run_once
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/asyncio/base_events.py", line 595 in run_forever
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/asyncio/base_events.py", line 628 in run_until_complete
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/test_asyncio/test_subprocess.py", line 442 in test_cancel_make_subprocess_transport_exec
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/case.py", line 549 in _callTestMethod
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/case.py", line 592 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/case.py", line 652 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/unittest/runner.py", line 176 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/support/__init__.py", line 959 in _run_suite
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/support/__init__.py", line 1082 in run_unittest
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/runtest.py", line 211 in _test_module
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/runtest.py", line 236 in _runtest_inner2
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/runtest.py", line 272 in _runtest_inner
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/runtest.py", line 155 in _runtest
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/runtest.py", line 195 in runtest
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/main.py", line 319 in rerun_failed_tests
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/main.py", line 696 in _main
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/main.py", line 639 in main
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/libregrtest/main.py", line 717 in main
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/__main__.py", line 2 in <module>
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/runpy.py", line 87 in _run_code
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/runpy.py", line 197 in _run_module_as_main
make: *** [Makefile:1236: buildbottest] Error 1

Cannot open file '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/test-results.xml' for upload

zware · 2021-03-03T02:50:05Z

I'm going to declare that buildbot failure Not Our Fault™ :). I also got an email about a Ubuntu check failure, which also appears unrelated to this change.

Use io.IncrementalNewlineDecoder for doctest newline conversion.

6487496

the-knights-who-say-ni added the CLA signed label Jan 28, 2021

bedevere-bot added the awaiting review label Jan 28, 2021

merwok changed the title ~~Use io.IncrementalNewlineDecoder for doctest newline conversion.~~ bpo-43049: Use io.IncrementalNewlineDecoder for doctest newline conversion Jan 28, 2021

zware added the skip news label Jan 28, 2021

zware approved these changes Jan 28, 2021

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting review labels Jan 28, 2021

github-actions bot added the stale Stale PR or inactive for long period of time. label Feb 28, 2021

github-actions bot removed the stale Stale PR or inactive for long period of time. label Mar 1, 2021

zware merged commit b36349a into python:master Mar 2, 2021

bedevere-bot removed the awaiting merge label Mar 2, 2021

sthagen mentioned this pull request Mar 2, 2021

bpo-43049: Use io.IncrementalNewlineDecoder for doctest newline conve… sthagen/python-cpython#412

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

bpo-43049: Use io.IncrementalNewlineDecoder for doctest newline conversion #24359

bpo-43049: Use io.IncrementalNewlineDecoder for doctest newline conversion #24359

Uh oh!

pdonis commented Jan 28, 2021 •

edited by merwok

Loading

Uh oh!

merwok commented Jan 28, 2021

Uh oh!

pdonis commented Jan 28, 2021

Uh oh!

github-actions bot commented Feb 28, 2021

Uh oh!

pdonis commented Feb 28, 2021

Uh oh!

zware commented Mar 2, 2021

Uh oh!

bedevere-bot commented Mar 2, 2021

Uh oh!

zware commented Mar 3, 2021

Uh oh!

Uh oh!

Uh oh!

bpo-43049: Use io.IncrementalNewlineDecoder for doctest newline conversion #24359

bpo-43049: Use io.IncrementalNewlineDecoder for doctest newline conversion #24359

Uh oh!

Conversation

pdonis commented Jan 28, 2021 • edited by merwok Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

merwok commented Jan 28, 2021

Uh oh!

pdonis commented Jan 28, 2021

Uh oh!

github-actions bot commented Feb 28, 2021

Uh oh!

pdonis commented Feb 28, 2021

Uh oh!

zware commented Mar 2, 2021

Uh oh!

bedevere-bot commented Mar 2, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

zware commented Mar 3, 2021

Uh oh!

Uh oh!

pdonis commented Jan 28, 2021 •

edited by merwok

Loading