bpo-39068 guard _b85chars2 initialization #17627

drmonkeysee · 2019-12-16T17:49:14Z

Under multi-threading scenarios a race condition may occur where a thread sees an initialized _b85chars table but an uninitialized _b85chars2 table due to the guard only checking the first table.

This causes an exception like:

  File "/usr/lib/python3.6/base64.py", line 434, in b85encode
    return _85encode(b, _b85chars, _b85chars2, pad),
  File "/usr/lib/python3.6/base64.py", line 294, in _85encode
    for word in words],
  File "/usr/lib/python3.6/base64.py", line 294, in <listcomp>
    for word in words],
 "TypeError: 'NoneType' object is not subscriptable

https://bugs.python.org/issue39068

the-knights-who-say-ni · 2019-12-16T17:49:18Z

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

Recognized GitHub username

We couldn't find a bugs.python.org (b.p.o) account corresponding to the following GitHub usernames:

@drmonkeysee

This might be simply due to a missing "GitHub Name" entry in one's b.p.o account settings. This is necessary for legal reasons before we can look at this contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

serhiy-storchaka · 2019-12-16T18:32:56Z

Lib/base64.py

@@ -430,6 +430,7 @@ def b85encode(b, pad=False):
    # if the function is never called
    if _b85chars is None:


It is enough to check if _b85chars2 is None here.

These checks are cheap, but they are performed at every b85encode() call. Lesser checks is better.

a85encode() should be updated too. And may be other functions if they initialize two or more globals.

done. visually scanned the file for other cases but only found the additional one in a85encode().

serhiy-storchaka · 2020-01-05T12:13:27Z

Please add a NEWS entry. Since it is your first contribution, add also your name in Misc/ACKS.

serhiy-storchaka · 2020-12-31T09:46:44Z

Thank you for your contribution @drmonkeysee. Sorry for the delay, this ЗК fell out of my window of attention.

miss-islington · 2020-12-31T09:47:07Z

Thanks @drmonkeysee for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

miss-islington · 2020-12-31T09:47:07Z

Thanks @drmonkeysee for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

There was a race condition in base64 in lazy initialization of multiple globals. (cherry picked from commit 9655434) Co-authored-by: Brandon Stansbury <brandonrstansbury@gmail.com>

bedevere-bot · 2020-12-31T09:47:29Z

GH-24020 is a backport of this pull request to the 3.9 branch.

miss-islington · 2020-12-31T09:47:31Z

Sorry, @drmonkeysee and @serhiy-storchaka, I could not cleanly backport this to 3.8 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 9655434cca5dfbea97bf6d355aec028e840b289c 3.8

There was a race condition in base64 in lazy initialization of multiple globals.. (cherry picked from commit 9655434) Co-authored-by: Brandon Stansbury <brandonrstansbury@gmail.com>

bedevere-bot · 2020-12-31T12:37:09Z

GH-24022 is a backport of this pull request to the 3.8 branch.

There was a race condition in base64 in lazy initialization of multiple globals. (cherry picked from commit 9655434) Co-authored-by: Brandon Stansbury <brandonrstansbury@gmail.com>

miss-islington · 2021-01-01T17:46:19Z

Thanks @drmonkeysee for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

There was a race condition in base64 in lazy initialization of multiple globals. (cherry picked from commit 9655434) Co-authored-by: Brandon Stansbury <brandonrstansbury@gmail.com>

bedevere-bot · 2021-01-01T17:46:30Z

GH-24051 is a backport of this pull request to the 3.9 branch.

There was a race condition in base64 in lazy initialization of multiple globals. (cherry picked from commit 9655434) Co-authored-by: Brandon Stansbury <brandonrstansbury@gmail.com>

There was a race condition in base64 in lazy initialization of multiple globals.

guard _b85chars2 table

04967d1

the-knights-who-say-ni added the CLA not signed label Dec 16, 2019

bedevere-bot added the awaiting review label Dec 16, 2019

Commod0re approved these changes Dec 16, 2019

View changes

bedevere-bot added awaiting core review and removed awaiting review labels Dec 16, 2019

camhawkins approved these changes Dec 16, 2019

View changes

drmonkeysee changed the title ~~guard _b85chars2 table~~ bpo-39068 guard _b85chars2 table Dec 16, 2019

drmonkeysee changed the title ~~bpo-39068 guard _b85chars2 table~~ bpo-39068 guard _b85chars2 initialization Dec 16, 2019

IEarle approved these changes Dec 16, 2019

View changes

serhiy-storchaka reviewed Dec 16, 2019

View changes

only check latter var and fix a85 as well

bf5a325

the-knights-who-say-ni added CLA signed and removed CLA not signed labels Dec 17, 2019

drmonkeysee added 2 commits Jan 6, 2020

add news and ack entries

9e86e22

rename file to match time the bug was filed

6562ee4

serhiy-storchaka approved these changes Dec 31, 2020

View changes

bedevere-bot added awaiting merge and removed awaiting core review labels Dec 31, 2020

serhiy-storchaka merged commit 9655434 into python:master Dec 31, 2020
9 checks passed

bedevere-bot removed the awaiting merge label Dec 31, 2020

serhiy-storchaka added needs backport to 3.8 needs backport to 3.9 type-bug labels Dec 31, 2020

bedevere-bot removed the needs backport to 3.9 label Dec 31, 2020

miss-islington assigned serhiy-storchaka Dec 31, 2020

bedevere-bot removed the needs backport to 3.8 label Dec 31, 2020

serhiy-storchaka removed their assignment Dec 31, 2020

serhiy-storchaka added the needs backport to 3.9 label Jan 1, 2021

bedevere-bot removed the needs backport to 3.9 label Jan 1, 2021

drmonkeysee deleted the base85-race-condition-fix branch Jan 4, 2021

adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 13, 2021

bpo-39068: Fix race condition in base64 (pythonGH-17627)

287d230

There was a race condition in base64 in lazy initialization of multiple globals.

python / cpython Public

bpo-39068 guard _b85chars2 initialization #17627

bpo-39068 guard _b85chars2 initialization #17627

drmonkeysee commented Dec 16, 2019 •

edited by bedevere-bot

the-knights-who-say-ni commented Dec 16, 2019

serhiy-storchaka Dec 16, 2019

drmonkeysee Dec 16, 2019 •

edited

serhiy-storchaka commented Jan 5, 2020

serhiy-storchaka commented Dec 31, 2020

miss-islington commented Dec 31, 2020

miss-islington commented Dec 31, 2020

bedevere-bot commented Dec 31, 2020

miss-islington commented Dec 31, 2020

bedevere-bot commented Dec 31, 2020

miss-islington commented Jan 1, 2021

bedevere-bot commented Jan 1, 2021

		@@ -430,6 +430,7 @@ def b85encode(b, pad=False):
		# if the function is never called
		if _b85chars is None:

python / cpython Public

bpo-39068 guard _b85chars2 initialization #17627

bpo-39068 guard _b85chars2 initialization #17627

Conversation

drmonkeysee commented Dec 16, 2019 • edited by bedevere-bot

the-knights-who-say-ni commented Dec 16, 2019

Recognized GitHub username

serhiy-storchaka Dec 16, 2019

Choose a reason for hiding this comment

drmonkeysee Dec 16, 2019 • edited

Choose a reason for hiding this comment

serhiy-storchaka commented Jan 5, 2020

serhiy-storchaka commented Dec 31, 2020

miss-islington commented Dec 31, 2020

miss-islington commented Dec 31, 2020

bedevere-bot commented Dec 31, 2020

miss-islington commented Dec 31, 2020

bedevere-bot commented Dec 31, 2020

miss-islington commented Jan 1, 2021

bedevere-bot commented Jan 1, 2021

drmonkeysee commented Dec 16, 2019 •

edited by bedevere-bot

drmonkeysee Dec 16, 2019 •

edited