bpo-46811: Make test suite support Expat >=2.4.5 #31453

hartwork · 2022-02-20T20:05:56Z

https://bugs.python.org/issue46811

Happy to adjust and discuss.
Please check the commit messages for why I'm dropping that one test.

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/

mgorny · 2022-02-20T21:26:17Z

Thank you for doing this. While I'm not an expert on these tests, your work seems well-thought.

ambv · 2022-02-21T13:49:56Z

Lib/test/test_xml_etree.py

-        self.assertEqual(e.tag, '{${stuff}}localname')
-        t = ET.ElementTree(e)
-        self.assertEqual(ET.tostring(e), b'<ns0:localname xmlns:ns0="${stuff}" />')
-


I understand the other changes. For this one, can you explain why this needs to be removed?

Hi @ambv, have you checked the commit message at dd7da01 ?

Indeed I have not. I have now. This will sadly be a breaking change. Was the elevated strictness here security-related as well?

This will sadly be a breaking change.

Could you elaborate? I should note that xmlns has also been about URIs.

Was the elevated strictness here security-related as well?

Yes, please see https://github.com/libexpat/libexpat/pull/561/files#diff-d1bcab18f24ba66b34aeb2e156f7fde58ef3de1a165514b0fccf0d04c26838f8R3758-R3767 . This allowed code execution through Expat in another application.

Could you elaborate?

While the use was incorrect per spec, clearly parsing what seems to be XML template files was a use case that existed in the wild when BPO-3151 was filed. The curly-brace and dollar sign suggest some ZOPE-related template (or JBOSS, or JavaScript, or the Sun Java System Web Server, etc. etc.). Whatever this usage was, it will now break with expat 2.4.5+

But since this is security-related, there's nothing we can do other than move on.

But since this is security-related, there's nothing we can do other than move on.

@ambv I notice now that (while Expat doesn't do full validation), moving the namespace separator in ElementTree off current } could make this work longer. A space or a newline would be other options, for instance.

@ambv it's here:

cpython/Modules/_elementtree.c

Line 3660 in 2cae938

self->parser = EXPAT(ParserCreate_MM)(encoding, &ExpatMemoryHandler, "}");

But that may need a closer look, it could be a breaking change too.

miss-islington · 2022-02-21T14:48:35Z

Thanks @hartwork for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7, 3.8, 3.9, 3.10.
🐍🍒⛏🤖

miss-islington · 2022-02-21T14:48:39Z

Sorry @hartwork and @ambv, I had trouble checking out the 3.10 backport branch.
Please backport using cherry_picker on command line.
cherry_picker 2cae93832f46b245847bdc252456ddf7742ef45e 3.10

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <sebastian@pipping.org>

bedevere-bot · 2022-02-21T14:48:59Z

GH-31469 is a backport of this pull request to the 3.9 branch.

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <sebastian@pipping.org>

bedevere-bot · 2022-02-21T14:49:06Z

GH-31470 is a backport of this pull request to the 3.8 branch.

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <sebastian@pipping.org>

bedevere-bot · 2022-02-21T14:49:11Z

GH-31471 is a backport of this pull request to the 3.7 branch.

miss-islington · 2022-02-21T15:03:22Z

Thanks @hartwork for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <sebastian@pipping.org>

bedevere-bot · 2022-02-21T15:03:43Z

GH-31472 is a backport of this pull request to the 3.10 branch.

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <sebastian@pipping.org>

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <sebastian@pipping.org> Co-authored-by: Sebastian Pipping <sebastian@pipping.org>

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <sebastian@pipping.org>

…thonGH-31469) Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <sebastian@pipping.org> Co-authored-by: Sebastian Pipping <sebastian@pipping.org>

…thonGH-31472) Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <sebastian@pipping.org>

…thonGH-31469) Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <sebastian@pipping.org> Co-authored-by: Sebastian Pipping <sebastian@pipping.org>

Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (rebased for 2.7.18 by Michał Górny)

…thonGH-31469) Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <sebastian@pipping.org> Co-authored-by: Sebastian Pipping <sebastian@pipping.org>

Upstream PR: python/cpython#31453

…thonGH-31469) Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <sebastian@pipping.org> Co-authored-by: Sebastian Pipping <sebastian@pipping.org>

…thonGH-31472) Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <sebastian@pipping.org>

…thonGH-31469) Curly brackets were never allowed in namespace URIs according to RFC 3986, and so-called namespace-validating XML parsers have the right to reject them a invalid URIs. libexpat >=2.4.5 has become strcter in that regard due to related security issues; with ET.XML instantiating a namespace-aware parser under the hood, this test has no future in CPython. References: - https://datatracker.ietf.org/doc/html/rfc3968 - https://www.w3.org/TR/xml-names/ Also, test_minidom.py: Support Expat >=2.4.5 (cherry picked from commit 2cae938) Co-authored-by: Sebastian Pipping <sebastian@pipping.org> Co-authored-by: Sebastian Pipping <sebastian@pipping.org>

bedevere-bot added the awaiting review label Feb 20, 2022

the-knights-who-say-ni added the CLA signed label Feb 20, 2022

bedevere-bot added the tests label Feb 20, 2022

hartwork force-pushed the bpo-46811 branch from b261ec6 to 37f6622 Compare Feb 20, 2022

hartwork added 2 commits Feb 20, 2022

test_minidom.py: Support Expat >=2.4.5

aa7523f

Add blurp file for bpo-46811 to section "Library"

2dcb305

hartwork force-pushed the bpo-46811 branch from 37f6622 to 2dcb305 Compare Feb 20, 2022

ambv reviewed Feb 21, 2022

View changes

ambv added needs backport to 3.7 needs backport to 3.8 needs backport to 3.9 needs backport to 3.10 labels Feb 21, 2022

ambv merged commit 2cae938 into python:main Feb 21, 2022
12 checks passed

bedevere-bot removed the awaiting review label Feb 21, 2022

miss-islington assigned ambv Feb 21, 2022

bedevere-bot removed the needs backport to 3.9 label Feb 21, 2022

bedevere-bot removed the needs backport to 3.8 label Feb 21, 2022

bedevere-bot removed the needs backport to 3.7 label Feb 21, 2022

ambv added needs backport to 3.10 and removed needs backport to 3.10 labels Feb 21, 2022

bedevere-bot removed the needs backport to 3.10 label Feb 21, 2022

hartwork deleted the bpo-46811 branch Feb 21, 2022

algitbot pushed a commit to alpinelinux/aports that referenced this issue Mar 30, 2022

main/python3: fix testsuite with expat update

0d137a9

Upstream PR: python/cpython#31453

hartwork mannequin mentioned this pull request Apr 18, 2022

Test suite needs adjustments for Expat >=2.4.5 #90967

Open

python / cpython Public

bpo-46811: Make test suite support Expat >=2.4.5 #31453

bpo-46811: Make test suite support Expat >=2.4.5 #31453

hartwork commented Feb 20, 2022 •

edited

mgorny commented Feb 20, 2022

ambv Feb 21, 2022

hartwork Feb 21, 2022

ambv Feb 21, 2022

hartwork Feb 21, 2022

ambv Feb 21, 2022

hartwork Feb 21, 2022

hartwork Feb 21, 2022

hartwork Feb 21, 2022

miss-islington commented Feb 21, 2022

miss-islington commented Feb 21, 2022

bedevere-bot commented Feb 21, 2022

bedevere-bot commented Feb 21, 2022

bedevere-bot commented Feb 21, 2022

miss-islington commented Feb 21, 2022

bedevere-bot commented Feb 21, 2022

python / cpython Public

bpo-46811: Make test suite support Expat >=2.4.5 #31453

bpo-46811: Make test suite support Expat >=2.4.5 #31453

Conversation

hartwork commented Feb 20, 2022 • edited

mgorny commented Feb 20, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

miss-islington commented Feb 21, 2022

miss-islington commented Feb 21, 2022

bedevere-bot commented Feb 21, 2022

bedevere-bot commented Feb 21, 2022

bedevere-bot commented Feb 21, 2022

miss-islington commented Feb 21, 2022

bedevere-bot commented Feb 21, 2022

hartwork commented Feb 20, 2022 •

edited