bpo-47000: Add `locale.getencoding()` #32068

methane · 2022-03-23T04:52:57Z

This API is a part of PEP 686.

https://bugs.python.org/issue47000

Doc/library/locale.rst

vstinner

In some places, "utf-8" is used, in some other places "UTF-8" is used. I suggest to use lower case "utf-8" everywhere: doc, C code, _Py_STR(), etc. You may have to update getpreferredencoding() doc.

Doc/glossary.rst

Doc/library/locale.rst

Doc/whatsnew/3.11.rst

Lib/locale.py

Modules/_io/textio.c

Co-authored-by: Victor Stinner <vstinner@python.org>

vstinner · 2022-04-07T13:09:53Z

Doc/library/locale.rst

   argument are ignored.

   The :ref:`Python preinitialization <c-preinit>` configures the LC_CTYPE
   locale. See also the :term:`filesystem encoding and error handler`.

   .. versionchanged:: 3.7
-      The function now always returns ``UTF-8`` on Android or if the
+      The function now always returns ``"UTF-8"`` on Android or if the


Suggested change

The function now always returns ``"UTF-8"`` on Android or if the

The function now always returns ``"utf-8"`` on Android or if the

vstinner · 2022-04-07T13:09:53Z

Doc/library/locale.rst

      :ref:`Python UTF-8 Mode <utf8-mode>` is enabled.

+   .. versionchanged:: 3.11
+      The function now returns ``"utf-8"`` instead of ``"UTF-8"`` on Android
+      or if the :ref:`Python UTF-8 Mode <utf8-mode>` is enabled.


I am sure that the upper case versus lower case is worth to be mentioned. But I'm fine with keeping if you consider that it's worth it.

vstinner · 2022-04-07T13:09:53Z

Doc/library/locale.rst

@@ -327,16 +327,37 @@ The :mod:`locale` module defines the following exception and functions:
   is not necessary or desired, *do_setlocale* should be set to ``False``.

   On Android or if the :ref:`Python UTF-8 Mode <utf8-mode>` is enabled, always
-   return ``'UTF-8'``, the :term:`locale encoding` and the *do_setlocale*
+   return ``'utf-8'``, the :term:`locale encoding` and the *do_setlocale*
   argument are ignored.

   The :ref:`Python preinitialization <c-preinit>` configures the LC_CTYPE
   locale. See also the :term:`filesystem encoding and error handler`.


I suggest to copy/paste this paragraph to getencoding() doc.

vstinner · 2022-04-07T13:09:53Z

Doc/whatsnew/3.11.rst

+------
+
+* Add :func:`locale.getencoding` to get the current locale encoding. It is similar to
+  ``locale.getpreferredencoding(False)`` but ignores :ref:`UTF-8 Mode <utf8-mode>`.


I like repeating "Python" to remind that it's unrelated to the Unix locale, but really something specific to Python which ignores the locale.

Suggested change

``locale.getpreferredencoding(False)`` but ignores :ref:`UTF-8 Mode <utf8-mode>`.

``locale.getpreferredencoding(False)`` but ignores the :ref:`Python UTF-8 Mode <utf8-mode>`.

vstinner · 2022-04-07T13:09:53Z

Misc/NEWS.d/next/Library/2022-04-06-11-54-53.bpo-47000.2nmAR1.rst

@@ -0,0 +1 @@
+Add :func:`locale.getencoding`.


You can copy/paste the Doc/whatsnew/3.11.rst entry entry.

vstinner · 2022-04-07T13:09:53Z

Doc/glossary.rst

-      encoding.
+      On Android and VxWorks, Python uses ``"utf-8"`` as the locale encoding.
+
+      ``locale.getencoding()`` can be used to get the locale encoding.

      Python uses the :term:`filesystem encoding and error handler` to convert
      between Unicode filenames and bytes filenames.


IMO it would be helpful to mention here the Python UTF-8 Mode which ignores the locale encoding and always uses UTF-8. What do you think?

I don't think so. This paragraph doesn't describe where the locale encoding is used.
This paragraph describes what the locale encoding is.
With this pull request, locale encoding is locale encoding even in UTF-8 mode.

On the other hand, following this paragraph looks unnecessary:

Python uses the :term:`filesystem encoding and error handler` to convert between Unicode filenames and bytes filenames.

I will replace it with See also :term:`filesystem encoding and error handler`.

vstinner

LGTM. Thanks.

IMO it's important that the current is complete and explains well this function, since it's a complex topic. If the doc is unclear, people will misuse it.

Thanks for adding "See also the filesystem encoding and error handler" in the glossary, that's good!

I'm not sure why the PR is still a draft.

bedevere-bot added the awaiting core review label Mar 23, 2022

the-knights-who-say-ni added the CLA signed label Mar 23, 2022

methane force-pushed the locale-get-encoding branch from c99ef77 to 8fc120e Compare Mar 31, 2022

methane added 4 commits Apr 4, 2022

bpo-47000: Add locale.get_encoding()

cd4bccc

get_encoding() -> getencoding()

aaa8956

fix TextIOWrapper.__init__

104206a

get_encoding -> getencoding

8b50eb1

methane force-pushed the locale-get-encoding branch from 04fd5b0 to 8b50eb1 Compare Apr 4, 2022

methane changed the title ~~bpo-47000: Add locale.get_encoding()~~ bpo-47000: Add locale.getencoding() Apr 4, 2022

methane changed the title ~~bpo-47000: Add locale.getencoding()~~ bpo-47000: Add locale.getencoding() Apr 4, 2022

Use _Py_STR for "utf-8"

a0204c2

methane force-pushed the locale-get-encoding branch from ffc9ba1 to a0204c2 Compare Apr 4, 2022

methane added 2 commits Apr 6, 2022

Update the doc.

08413f1

Add news

f61b622

methane requested a review from vstinner Apr 6, 2022

vstinner reviewed Apr 6, 2022

View changes

Doc/library/locale.rst Outdated Show resolved Hide resolved

methane added 2 commits Apr 6, 2022

Apply proposed changes.

1c60e5b

Fix doc error.

7397493

vstinner reviewed Apr 6, 2022

View changes

methane and others added 2 commits Apr 7, 2022

Apply suggestions from code review

44730dd

Co-authored-by: Victor Stinner <vstinner@python.org>

UTF-8 -> utf-8

7720b10

methane force-pushed the locale-get-encoding branch from 39e99bc to 7720b10 Compare Apr 7, 2022

Fix test_utf8_mode

104008d

vstinner reviewed Apr 7, 2022

View changes

Update based on suggestions

a8e50bd

vstinner approved these changes Apr 8, 2022

View changes

bedevere-bot added awaiting merge and removed awaiting core review labels Apr 8, 2022

methane marked this pull request as ready for review Apr 9, 2022

methane merged commit 6773203 into python:main Apr 9, 2022
12 checks passed

methane deleted the locale-get-encoding branch Apr 9, 2022

bedevere-bot removed the awaiting merge label Apr 9, 2022

python / cpython Public

bpo-47000: Add `locale.getencoding()` #32068

bpo-47000: Add `locale.getencoding()` #32068

methane commented Mar 23, 2022 •

edited

vstinner left a comment

vstinner Apr 7, 2022

vstinner Apr 7, 2022

vstinner Apr 7, 2022

vstinner Apr 7, 2022

vstinner Apr 7, 2022

vstinner Apr 7, 2022

methane Apr 8, 2022

vstinner left a comment

	The function now always returns ``"UTF-8"`` on Android or if the
	The function now always returns ``"utf-8"`` on Android or if the

	``locale.getpreferredencoding(False)`` but ignores :ref:`UTF-8 Mode <utf8-mode>`.
	``locale.getpreferredencoding(False)`` but ignores the :ref:`Python UTF-8 Mode <utf8-mode>`.

python / cpython Public

bpo-47000: Add locale.getencoding() #32068

bpo-47000: Add locale.getencoding() #32068

Conversation

methane commented Mar 23, 2022 • edited

vstinner left a comment

vstinner Apr 7, 2022

Choose a reason for hiding this comment

vstinner Apr 7, 2022

Choose a reason for hiding this comment

vstinner Apr 7, 2022

Choose a reason for hiding this comment

vstinner Apr 7, 2022

Choose a reason for hiding this comment

vstinner Apr 7, 2022

Choose a reason for hiding this comment

vstinner Apr 7, 2022

Choose a reason for hiding this comment

methane Apr 8, 2022

Choose a reason for hiding this comment

vstinner left a comment

bpo-47000: Add `locale.getencoding()` #32068

bpo-47000: Add `locale.getencoding()` #32068

methane commented Mar 23, 2022 •

edited