New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bpo-43510: Implement PEP 597 opt-in EncodingWarning. #19481

Merged

methane merged 51 commits into python:master from methane:open-encoding

Mar 29, 2021

Member

methane commented Apr 12, 2020 •

edited

Add -X warn_default_encoding and PYTHONWARNDEFAULTENCODING.
Add EncodingWarning
Add io.text_encoding()
open(), TextIOWrapper() emits EncodingWarning when encoding is omitted and warn_default_encoding is enabled.
_pyio.TextIOWrapper() uses UTF-8 as fallback default encoding used when failed to import locale module. (used during building Python)
bz2, configparser, gzip, lzma, pathlib, tempfile modules use io.text_encoding().
Add documents
Add what's new entry

methane added the DO-NOT-MERGE label

the-knights-who-say-ni added the CLA signed label

bedevere-bot added the awaiting core review label

methane force-pushed the open-encoding branch 4 times, most recently from c37295e to 6f72b38 Compare

methane added 3 commits


          Raise a warning when encoding is omitted

a3c014b

The warning is raised only in dev mode.

pathlib uses the new `io.text_encoding` helper function.
Other libraries will follow.


          add test

050bd1b


          wrap encoding=None with text_encoding.

939f4a0

methane force-pushed the open-encoding branch from e44dc5b to 939f4a0 Compare


          Add io.LOCALE_ENCODING = "locale"

3c99777

methane changed the title ~~PEP 597: Raise a warning when encoding is omitted~~ PEP 597: Emit EncodingWarning when encoding is omitted

methane added 2 commits


          Add EncodingWarning.

Copied from python#13938


          Add sys.warn_default_encoding

c5c556c

methane force-pushed the open-encoding branch from db537b9 to c5c556c Compare

methane added 11 commits


          shorten option names

d9a08c2


          EncodingWarning extends Warning

772648e


          make clinic

1a8e305


          fix test

20966cd


          remove wrong test case

2b80f42


          fix exception_hierarchy.txt

760308c


          Make sys.flags.encoding_warning int

a95dff2


          Fix text_embed.

31fb411


          Fix test_pickle

096a0a3


          configparser: use io.text_encoding()

99fc938


          Rename option names

6fdbcbc

sys.flags.encoding_warning -> warn_default_encoding
-X warn_encoding -> warn_default_encoding
PYTHONWARNENCODING -> PYTHONWARNDEFAULTENCODING

methane changed the title ~~PEP 597: Emit EncodingWarning when encoding is omitted~~ bpo-43510: Add encoding="locale" and optional EncodingWarning

methane removed the DO-NOT-MERGE label


          Merge remote-tracking branch 'upstream/master' into open-encoding

3f362bc

methane and others added 10 commits


          Update Doc/library/io.rst

737059e

Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>


          Update Doc/library/io.rst

6a62211

Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>


          Update Doc/library/io.rst

54c7dc6

Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>


          Update Doc/library/io.rst

5b2830b

Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>


          Apply suggestions from code review

14f2a6e

Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>


          Move EncodingWarnings

06e2a32


          fix comment

27d49d2


          fix text_encoding() docstring

80f4644


          update what's new

6ad0e7f


          fix doc build

73b27f1

CAM-Gerlach reviewed

View changes

Lib/_pyio.py Outdated Show resolved Hide resolved

Member

CAM-Gerlach commented Mar 23, 2021

Thanks @methane ! I've replied to your comments with updated suggestions to reflect your feedback.


          Update Doc/library/io.rst

c149d65

Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>

methane reviewed

View changes

Lib/_pyio.py Outdated Show resolved Hide resolved


          Apply suggestions from code review

4eb7655

Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>

methane reviewed

View changes

Doc/whatsnew/3.10.rst Outdated Show resolved Hide resolved


          Apply suggestions from code review

e3bce76

Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>

CAM-Gerlach reviewed

View changes

Doc/library/io.rst Outdated Show resolved Hide resolved


          Update Doc/library/io.rst

c089fd7

Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>

Member Author

methane commented Mar 25, 2021

I will merge this PR next week if there are no objections.

methane merged commit 4827483 into python:master

11 checks passed

methane deleted the open-encoding branch

bedevere-bot removed the awaiting core review label

bedevere-bot commented Mar 29, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL7 LTO + PGO 3.x has failed when building commit 4827483.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/96/builds/910) and take a look at the build logs.
Check if the failure is related to this commit (4827483) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/96/builds/910

Failed tests:

test_tools
test_ftplib

Failed subtests:

test_storlines - test.test_ftplib.TestFTPClass
test_reindent_file_with_bad_encoding - test.test_tools.test_reindent.ReindentTests

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

412 tests OK.

10 slowest tests:

test_concurrent_futures: 2 min 38 sec
test_shelve: 2 min 30 sec
test_mailbox: 1 min 57 sec
test_dbm: 1 min 52 sec
test_tarfile: 1 min 36 sec
test_multiprocessing_spawn: 1 min 34 sec
test_tokenize: 1 min 33 sec
test_dbm_gnu: 1 min 28 sec
test_unparse: 1 min 28 sec
test_asyncio: 1 min 17 sec

1 test failed:
test_tools

14 tests skipped:
test_devpoll test_gdb test_ioctl test_kqueue test_msilib
test_ossaudiodev test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

2 re-run tests:
test_ftplib test_tools

Total duration: 6 min 19 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/test_ftplib.py", line 623, in test_storlines
    self.check_data(self.server.handler_instance.last_received_data, RETR_DATA)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/test_ftplib.py", line 494, in check_data
    self.assertEqual(len(received), len(expected))
AssertionError: 12019 != 12018


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/test_tools/test_reindent.py", line 29, in test_reindent_file_with_bad_encoding
    rc, out, err = assert_python_ok(self.script, '-r', bad_coding_path)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/python', '-X', 'faulthandler', '-I', '/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Tools/scripts/reindent.py', '-r', '/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/bad_coding.py']

bedevere-bot commented Mar 29, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL7 LTO 3.x has failed when building commit 4827483.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/507/builds/896) and take a look at the build logs.
Check if the failure is related to this commit (4827483) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/507/builds/896

Failed tests:

test_tools

Failed subtests:

test_reindent_file_with_bad_encoding - test.test_tools.test_reindent.ReindentTests

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

412 tests OK.

10 slowest tests:

test_mailbox: 4 min 33 sec
test_peg_generator: 3 min 37 sec
test_concurrent_futures: 2 min 56 sec
test_shelve: 2 min 27 sec
test_multiprocessing_spawn: 1 min 50 sec
test_tokenize: 1 min 49 sec
test_unparse: 1 min 37 sec
test_lib2to3: 1 min 25 sec
test_asyncio: 1 min 21 sec
test_multiprocessing_forkserver: 1 min 9 sec

1 test failed:
test_tools

14 tests skipped:
test_devpoll test_gdb test_ioctl test_kqueue test_msilib
test_ossaudiodev test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

1 re-run test:
test_tools

Total duration: 7 min 15 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Lib/test/test_tools/test_reindent.py", line 29, in test_reindent_file_with_bad_encoding
    rc, out, err = assert_python_ok(self.script, '-r', bad_coding_path)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/python', '-X', 'faulthandler', '-I', '/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Tools/scripts/reindent.py', '-r', '/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Lib/test/bad_coding.py']

bedevere-bot commented Mar 29, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows10 3.x has failed when building commit 4827483.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/146/builds/1014) and take a look at the build logs.
Check if the failure is related to this commit (4827483) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/146/builds/1014

Failed tests:

test_locale

Failed subtests:

test_getsetlocale_issue1813 - test.test_locale.TestMiscellaneous

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

395 tests OK.

10 slowest tests:

test_mmap: 9 min 37 sec
test_largefile: 5 min 57 sec
test_io: 5 min 39 sec
test_unparse: 3 min 12 sec
test_multiprocessing_spawn: 2 min 16 sec
test_concurrent_futures: 2 min 3 sec
test_tokenize: 1 min 46 sec
test_capi: 1 min 46 sec
test_peg_generator: 1 min 39 sec
test_asyncio: 1 min 31 sec

1 test failed:
test_locale

31 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_spwd
test_syslog test_threadsignals test_wait3 test_wait4
test_xxlimited test_xxtestfuzz test_zipfile64

1 re-run test:
test_locale

Total duration: 19 min 35 sec

Click to see traceback logs

Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_locale.py", line 567, in test_getsetlocale_issue1813
    locale.setlocale(locale.LC_CTYPE, loc)
  File "D:\buildarea\3.x.bolen-windows10\build\lib\locale.py", line 610, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

Member Author

methane commented Mar 29, 2021 •

edited by bedevere-bot

test_locale fail is unrelating to this PR. See bpo-37945.

Member

CAM-Gerlach commented Mar 29, 2021

Thanks @methane ! Excited to see this in Python 3.10.

alexdelorenzo mentioned this pull request

Python 3.10 alpha 7 introduces new changes to pathlib alexdelorenzo/aiopath#4

Open

kreathon pushed a commit to kreathon/cpython that referenced this issue


          bpo-43510: Implement PEP 597 opt-in EncodingWarning. (pythonGH-19481)

37adcf2

See [PEP 597](https://www.python.org/dev/peps/pep-0597/).

* Add `-X warn_default_encoding` and `PYTHONWARNDEFAULTENCODING`.
* Add EncodingWarning
* Add io.text_encoding()
* open(), TextIOWrapper() emits EncodingWarning when encoding is omitted and warn_default_encoding is enabled.
* _pyio.TextIOWrapper() uses UTF-8 as fallback default encoding used when failed to import locale module. (used during building Python)
* bz2, configparser, gzip, lzma, pathlib, tempfile modules use io.text_encoding().
* What's new entry

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment