ElementTree should use UTF-8 for xml declaration. #91810

methane · 2022-04-22T07:23:56Z

Feature or enhancement

Currently, ElementTree.tostring(root, encoding="unicode", xml_declaration=True) uses locale encoding.

I think ElementTree should use UTF-8, instead of locale encoding.

Example:

$ LANG=ja_JP.eucJP ./python.exe
Python 3.11.0a7+ (heads/bytes-alloc-dirty:7fbc7f6128, Apr 19 2022, 16:53:54) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import xml.etree.ElementTree as ET
>>> et = ET.fromstring("<t>hello</t>")
>>> ET.tostring(et, encoding="unicode", xml_declaration=True)
"<?xml version='1.0' encoding='eucJP'?>\n<t>hello</t>"

Code:

cpython/Lib/xml/etree/ElementTree.py

Lines 732 to 742 in bcf14ae

    
           with _get_writer(file_or_filename, enc_lower) as write: 
        
               if method == "xml" and (xml_declaration or 
        
                       (xml_declaration is None and 
        
                        enc_lower not in ("utf-8", "us-ascii", "unicode"))): 
        
                   declared_encoding = encoding 
        
                   if enc_lower == "unicode": 
        
                       # Retrieve the default encoding for the xml declaration 
        
                       import locale 
        
                       declared_encoding = locale.getpreferredencoding() 
        
                   write("<?xml version='1.0' encoding='%s'?>\n" % ( 
        
                       declared_encoding,))

Pitch

UTF-8 is the most common encoding for XML.
Locale encoding name (e.g. cp932 or eucJP) would be different from XML encoding name recommended by w3c (e.g. Shift_JIS or EUC-JP).

The text was updated successfully, but these errors were encountered:

serhiy-storchaka · 2022-04-22T19:28:19Z

Look at dump(). It writes an element to stdout, which usually uses the locale encoding. I think it is the rationale of using the locale encoding here. You need to change dump() to use the stdout's encoding explicitly.

serhiy-storchaka · 2022-04-22T19:30:57Z

Or maybe change write() to get the encoding from the output string if available.

methane · 2022-04-23T01:33:35Z

dump don't use xml_declaration=True. So this issue doesn't affect it.

methane · 2022-04-25T05:28:37Z

@scoder would you give us an advice? (you are listed as etree expert in expert index).

There is no correct behavior, because output is Unicode and etree don't know what is real output encoding.

There are some cases that current behavior is better (e.g. using default encoding (e.g. open(filename, 'w')).
On the other hand, encoding="cp932" (in Japanese Windows) is non-portable (encoding="Shift_JIS" should be used), and UTF-8 is the most recommended encoding for XML.

I have two ideas:

a. Make UTF-8 default. This is simplest.
b. Keep using locale.getpreferredencoding() and wait PEP 686 accepted. (But it should be replaced with locale.getpreferredencoding(False) anyway.)

serhiy-storchaka · 2022-04-25T05:47:10Z

Adding encoding="UTF-8" and using cp932 to encode the content would be even worse.

Maybe add a simple mapping from Python encodings to XML encodings (for example we need to write "ascii" as "us-ascii")? Later we can discuss adding a public API for this.

serhiy-storchaka · 2022-04-25T05:50:10Z

I proposed to get the default encoding from the file object if available. #91812 (comment)

methane · 2022-04-25T05:55:53Z

Adding encoding="UTF-8" and using cp932 to encode the content would be even worse.

Of course, we should recommend to use UTF-8.
Note that encoding='cp932' and using UTF-8 is possible bug for now already.
Any default value may cause bug. There is no one correct default. But UTF-8 may be the best for now.

Maybe add a simple mapping from Python encodings to XML encodings (for example we need to write "ascii" as "us-ascii")? Later we can discuss adding a public API for this.

We may not know Python encoding because output is Unicode (e.g. Unicode string or StringIO).
Such idea works only when output is TextIOWrapper. (And there are no guarantee that TextIOWrapper.encoding is really the final encoding.)

If we want to support arbitrary encoding, we should add another option like xml_declaration_encoding="Shift_JIS".
But this is not strict necessary.
User can chose xml_declaration=False and prepend <?xml version="1.0" encoding="Shift_JIS" ?> manually when they really need to use encoding other than UTF-8.

…ML declaration ElementTree method write() and function tostring() now use the text file's encoding ("UTF-8" if not available) instead of locale encoding in XML declaration when encoding="unicode" is specified.

serhiy-storchaka · 2022-04-25T07:31:52Z

Note that encoding='cp932' and using UTF-8 is possible bug for now already.

Yes, it is a bug, and #91903 fixes it.

…91989)

pythonGH-91989) (cherry picked from commit f60b4c3) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…II data (pythonGH-91989). (cherry picked from commit f60b4c3) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…91989) (cherry picked from commit f60b4c3) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

GH-91989). (GH-91994) (cherry picked from commit f60b4c3)

serhiy-storchaka · 2022-04-29T05:45:23Z

The difference of #91903 from #91812:

If pass a file path, it declares the default file encoding (used to encode the content) instead of UTF-8. After implementing PEP 686 the result will be the same, but older versions need a fix.
It also correctly escapes non-encodable characters.
If pass an open text file, it declares the file encoding instead of UTF-8. PEP 686 will not fix this.

What is common in #91812 and #91903 and different from the current code:

If pass StringIO or custom stream without the encoding attribute, they declare UTF-8 instead of the locale encoding.

serhiy-storchaka · 2022-05-10T17:59:12Z

What do you think about this @methane?

…ML declaration (pythonGH-91903) ElementTree method write() and function tostring() now use the text file's encoding ("UTF-8" if not available) instead of locale encoding in XML declaration when encoding="unicode" is specified. (cherry picked from commit 707839b) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…laration (GH-91903) ElementTree method write() and function tostring() now use the text file's encoding ("UTF-8" if not available) instead of locale encoding in XML declaration when encoding="unicode" is specified.

…XML declaration (GH-91903) (GH-92663) ElementTree method write() and function tostring() now use the text file's encoding ("UTF-8" if not available) instead of locale encoding in XML declaration when encoding="unicode" is specified. (cherry picked from commit 707839b) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Automerge-Triggered-By: GH:serhiy-storchaka

…XML declaration (GH-91903) (GH-92664) ElementTree method write() and function tostring() now use the text file's encoding ("UTF-8" if not available) instead of locale encoding in XML declaration when encoding="unicode" is specified. (cherry picked from commit 707839b) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Automerge-Triggered-By: GH:serhiy-storchaka

…ML declaration (GH-91903) (GH-92665) ElementTree method write() and function tostring() now use the text file's encoding ("UTF-8" if not available) instead of locale encoding in XML declaration when encoding="unicode" is specified. (cherry picked from commit 707839b) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Automerge-Triggered-By: GH:serhiy-storchaka

…ncoding='unicode' Suppress writing an XML declaration in open files in ElementTree.write() with encoding='unicode' and xml_declaration=None.

…II data (pythonGH-91989). (pythonGH-91994) (cherry picked from commit f60b4c3)

…t in XML declaration (pythonGH-91903) (pythonGH-92665) ElementTree method write() and function tostring() now use the text file's encoding ("UTF-8" if not available) instead of locale encoding in XML declaration when encoding="unicode" is specified. (cherry picked from commit 707839b) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Automerge-Triggered-By: GH:serhiy-storchaka

…g='unicode' (GH-93426) Suppress writing an XML declaration in open files in ElementTree.write() with encoding='unicode' and xml_declaration=None. If file patch is passed to ElementTree.write() with encoding='unicode', always open a new file in UTF-8.

…ncoding='unicode' (pythonGH-93426) Suppress writing an XML declaration in open files in ElementTree.write() with encoding='unicode' and xml_declaration=None. If file patch is passed to ElementTree.write() with encoding='unicode', always open a new file in UTF-8. (cherry picked from commit d7db9dc3cc5b44d0b4ce000571fecf58089a01ec) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…ncoding='unicode' (pythonGH-93426) Suppress writing an XML declaration in open files in ElementTree.write() with encoding='unicode' and xml_declaration=None. If file patch is passed to ElementTree.write() with encoding='unicode', always open a new file in UTF-8. (cherry picked from commit d7db9dc) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

methane added type-feature expert-XML labels Apr 22, 2022

methane mentioned this issue Apr 22, 2022

gh-91810: ElementTree: Change default encoding in XML declaration to UTF-8 #91812

Closed

serhiy-storchaka mentioned this issue Apr 25, 2022

gh-91810: ElementTree: Use text file's encoding by default in XML declaration #91903

Merged

serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Apr 27, 2022

pythongh-91810: Expand ElementTree.write() tests to use non-ASCII data

68008bd

serhiy-storchaka mentioned this issue Apr 27, 2022

gh-91810: Expand ElementTree.write() tests to use non-ASCII data #91989

Merged

serhiy-storchaka added a commit that referenced this issue Apr 27, 2022

gh-91810: Expand ElementTree.write() tests to use non-ASCII data (GH-…

f60b4c3

…91989)

miss-islington pushed a commit to miss-islington/cpython that referenced this issue Apr 27, 2022

pythongh-91810: Expand ElementTree.write() tests to use non-ASCII data (

74fec20

pythonGH-91989) (cherry picked from commit f60b4c3) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

miss-islington added a commit that referenced this issue Apr 27, 2022

gh-91810: Expand ElementTree.write() tests to use non-ASCII data (GH-…

777d478

…91989) (cherry picked from commit f60b4c3) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

serhiy-storchaka added a commit that referenced this issue Apr 27, 2022

[3.9] gh-91810: Expand ElementTree.write() tests to use non-ASCII data (

993bb16

GH-91989). (GH-91994) (cherry picked from commit f60b4c3)

serhiy-storchaka closed this as completed in #91903 May 11, 2022

serhiy-storchaka mentioned this issue Jun 2, 2022

gh-91810: Fix regression with writing an XML declaration with encoding='unicode' #93426

Merged

hello-adam pushed a commit to hello-adam/cpython that referenced this issue Jun 2, 2022

[3.9] pythongh-91810: Expand ElementTree.write() tests to use non-ASC…

8d2b4c0

…II data (pythonGH-91989). (pythonGH-91994) (cherry picked from commit f60b4c3)

MarkusPiotrowski mentioned this issue Jun 10, 2022

CI issues @github-actions CI / test_windows (3.9) (pull_request) Failing after 5m — test_windows (3.9) biopython/biopython#3944

Open

ElementTree should use UTF-8 for xml declaration. #91810

ElementTree should use UTF-8 for xml declaration. #91810

methane commented Apr 22, 2022

serhiy-storchaka commented Apr 22, 2022

serhiy-storchaka commented Apr 22, 2022

methane commented Apr 23, 2022

methane commented Apr 25, 2022

serhiy-storchaka commented Apr 25, 2022

serhiy-storchaka commented Apr 25, 2022

methane commented Apr 25, 2022 •

edited

serhiy-storchaka commented Apr 25, 2022 •

edited

serhiy-storchaka commented Apr 29, 2022

serhiy-storchaka commented May 10, 2022

ElementTree should use UTF-8 for xml declaration. #91810

ElementTree should use UTF-8 for xml declaration. #91810

Comments

methane commented Apr 22, 2022

serhiy-storchaka commented Apr 22, 2022

serhiy-storchaka commented Apr 22, 2022

methane commented Apr 23, 2022

methane commented Apr 25, 2022

serhiy-storchaka commented Apr 25, 2022

serhiy-storchaka commented Apr 25, 2022

methane commented Apr 25, 2022 • edited

serhiy-storchaka commented Apr 25, 2022 • edited

serhiy-storchaka commented Apr 29, 2022

serhiy-storchaka commented May 10, 2022

methane commented Apr 25, 2022 •

edited

serhiy-storchaka commented Apr 25, 2022 •

edited