Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[3.7] gh-95778: CVE-2020-10735: Prevent DoS by very large int() #96504

Merged
merged 23 commits into from Sep 6, 2022

Conversation

gpshead
Copy link
Member

@gpshead gpshead commented Sep 2, 2022

Integer to and from text conversions via CPython's bignum int type is not safe against denial of service attacks due to malicious input. Very large input strings with hundred thousands of digits can consume several CPU seconds.

This PR comes fresh from a pile of work done in our private PSRT security response team repo.

This backports #96499 aka 511ca94

Signed-off-by: Christian Heimes [Red Hat] christian@python.org
Tons-of-polishing-up-by: Gregory P. Smith [Google] greg@krypto.org
Reviews via the private PSRT repo via many others (see the NEWS entry in the PR).

I wrote up a one pager for the release managers.

Additional review is wise on this 3.7 PR as the backport was more complicated due to internal Python config and startup code cleanup in 3.8 and onwards.

gpshead and others added 13 commits September 1, 2022 17:42
This is based off of psrt/CVE-2020-10735-3.8backport branch at cd54fc3.
This is required for the 3.7 tree to pass on modern macOS.
I don't know why, but macOS in 3.7 CI is failing to build the zlib
module these days so it's exposing this test that didn't have the
proper `@requires_zlib` annotation.

Getting it to build with zlib and other things that are now wrongly
"missing" in the 3.7 CI setup would be nice, but probably involves
invasive backporting of parts of
python@b29d0a5
by a macOS domain expert.

Not worth it.
This test also appears to require changes to
Lib/ctypes/macholib/dyld.py to work in the existing macOS CI config.
I'm just skipping it, backporting that would be a feature.
Not going to happen in 3.7.

There may be a way to configure macOS CI to use an older macOS and
toolchain instead as an alternate option.  Someone else can figure
that out if so.  This branch only lives for another 9 months per
https://peps.python.org/pep-0537/
the 3.8 branch got rid of this line already.  it blocks seeing the
actual error while testing a doc build!
@gpshead gpshead added DO-NOT-MERGE release-blocker type-security A security issue and removed type-security A security issue labels Sep 2, 2022
@gpshead gpshead requested a review from zooba September 2, 2022 05:38
@zooba
Copy link
Member

zooba commented Sep 2, 2022

Kinda looks like the Windows issue is the Yield macro conflicting with something from the OS:

 ast.c
C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\um\winbase.h(103): warning C4005: 'Yield': macro redefinition [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]
  D:\a\cpython\cpython\Include\Python-ast.h(558): note: see previous definition of 'Yield'
..\Python\ast.c(2706): warning C4002: too many actual parameters for macro 'Yield' [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]
..\Python\ast.c(2706): warning C4033: 'ast_for_expr' must return a value [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]

Then we see further down that python.exe crashes when run:

  python.vcxproj -> D:\a\cpython\cpython\PCbuild\win32\python.exe
  python.vcxproj -> D:\a\cpython\cpython\PCbuild\win32\python.pdb (Full PDB)
D:\a\cpython\cpython\PCbuild\python.vcxproj(90,5): warning MSB3073: The command "setlocal
D:\a\cpython\cpython\PCbuild\python.vcxproj(90,5): warning MSB3073: set PYTHONPATH=D:\a\cpython\cpython\Lib
D:\a\cpython\cpython\PCbuild\python.vcxproj(90,5): warning MSB3073: "D:\a\cpython\cpython\PCbuild\win32\python.exe" "D:\a\cpython\cpython\PC\validate_ucrtbase.py" ucrtbase" exited with code -1073741819.
D:\a\cpython\cpython\PCbuild\python.vcxproj(90,5): warning MSB4181: The "Exec" task returned false but did not log an error.

Not sure why pythoninfo doesn't display the exit code, but that would likely also show -1073741819.

Python/ast.c Show resolved Hide resolved
@gpshead gpshead removed their assignment Sep 2, 2022
@gpshead gpshead marked this pull request as ready for review September 2, 2022 17:15
@gpshead gpshead added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Sep 2, 2022
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @gpshead for commit 38ec6a9 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Sep 2, 2022
@gpshead
Copy link
Member Author

gpshead commented Sep 2, 2022

woo, it worked! On the PRs I merged I just pasted the markdown from the PR opening comment in as the commit message.

@gpshead
Copy link
Member Author

gpshead commented Sep 4, 2022

wait for #96537 to be integrated into this PR before merging, i'll remove the do-not-merge label then.

gpshead and others added 5 commits September 3, 2022 23:19
Per mdickinson@'s comment on the main branch PR.
…#96537)

Converting a large enough `int` to a decimal string raises `ValueError` as expected. However, the raise comes _after_ the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =)

The quick fix: essentially we catch _most_ values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact.

The justification for the current check. The C code check is:
```c
max_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10
```

In GitHub markdown math-speak, writing $M$ for `max_str_digits`, $L$ for `PyLong_SHIFT` and $s$ for `size_a`, that check is:
$$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$

From this it follows that
$$\frac{M}{3L} < \frac{s-1}{10}$$
hence that
$$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$
So
$$2^{L(s-1)} > 10^M.$$
But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything _below_ the intended limit in the check.

<!-- gh-issue-number: pythongh-95778 -->
* Issue: pythongh-95778
<!-- /gh-issue-number -->

Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
@ambv ambv added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Sep 5, 2022
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @ambv for commit 7f911c1 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Sep 5, 2022
@ambv
Copy link
Contributor

ambv commented Sep 5, 2022

"12 successful checks" is a little thin, no? test-with-buildbots on 3.8 runs 80 checks. Let's run those here, too, as the 3.8 backport's got some legitimate failures.

@ambv
Copy link
Contributor

ambv commented Sep 5, 2022

In fact, it looks like the latest test-with-buildbots run on this PR from 3 days ago on 38ec6a9 did indeed have the same ASAN and USAN failures.

@gpshead
Copy link
Member Author

gpshead commented Sep 5, 2022

"as the 3.8 backport's got some legitimate failures."

I don't believe it does. There is no history of success for those ASAN and USAN builds on the 3.8 and 3.7 branches.

@ambv
Copy link
Contributor

ambv commented Sep 5, 2022

We looked into this. 3.7 and 3.8 are typically not running address and undefined behavior sanitizers on buildbots. The label-driven trigger on PRs is overly broad and runs configurations that aren't used by us for the oldest branches in regular testing. At least 3.8 fails ASAN and UBSAN without GH-96503 applied as well. I'll leave it to @ned-deily to land this to his branch but I'm landing 3.8.

@ambv
Copy link
Contributor

ambv commented Sep 5, 2022

@ypankovych, don't leave unsolicited approving reviews without comments, those are not actionable for us.

@ypankovych
Copy link
Contributor

@ambv sorry, that was more of a stamp

@ambv
Copy link
Contributor

ambv commented Sep 5, 2022

I get it, @ypankovych, your intentions were good. But we get many random stamps on open PRs and when they don't even have a comment saying why that person is confident the PR is good, we can't really do anything with that.

@ned-deily ned-deily merged commit 15ec1af into python:3.7 Sep 6, 2022
59 of 79 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants