bpo-43950: optimize column table assembling with pre-sizing the column table #26997

isidentical · 2021-07-02T17:34:37Z

The new resizing system works like this;

$ cat t.py
a + a + a + b + c + a + a + a + b + c + a + a + a + b + c + a + a + a + b + c
[repeated 99 more times]
$ ./python t.py
RESIZE: prev len = 32, new len = 66
FINAL SIZE: 56
-----------------------------------------------------
RESIZE: prev len = 32, new len = 66
RESIZE: prev len = 66, new len = 134
RESIZE: prev len = 134, new len = 270
RESIZE: prev len = 270, new len = 542
RESIZE: prev len = 542, new len = 1086
RESIZE: prev len = 1086, new len = 2174
RESIZE: prev len = 2174, new len = 4350
RESIZE: prev len = 4350, new len = 8702
FINAL SIZE: 8004

So now we do considerably lower number of _PyBytes_Resize calls.

https://bugs.python.org/issue43950

Automerge-Triggered-By: GH:isidentical

pablogsal

I can confirm that this speeds up the allocation quite a lot as expected 👍

❯ time ./python -m test test_compile -m test_stack_overflow -v
== CPython 3.11.0a0 (heads/optimize-co-positions:760daf9858, Jul 2 2021, 19:08:05) [GCC 11.1.0]
== Linux-5.12.13-arch1-2-x86_64-with-glibc2.33 little-endian
== cwd: /home/pablogsal/github/python/main/build/test_python_15988æ
== CPU count: 36
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 2.15 Run tests sequentially
0:00:00 load avg: 2.15 [1/1] test_compile
test_stack_overflow (test.test_compile.TestSpecifics) ... ok

----------------------------------------------------------------------

Ran 1 test in 3.945s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 4.0 sec
Tests result: SUCCESS
./python -m test test_compile -m test_stack_overflow -v  3.77s user 0.47s system 99% cpu 4.241 total

miss-islington · 2021-07-02T18:30:37Z

@isidentical: Status check is done, and it's a success ❌ .

isidentical · 2021-07-02T18:42:56Z

concurrent_futures error, irrelevant, restarting the CI

miss-islington · 2021-07-02T18:43:10Z

@isidentical: Status check is done, and it's a success ✅ .

miss-islington · 2021-07-02T18:43:11Z

Sorry, I can't merge this PR. Reason: 3 of 5 required status checks have not succeeded: 2 expected..

miss-islington · 2021-07-02T19:03:57Z

@isidentical: Status check is done, and it's a success ✅ .

bpo-43950: optimize column table assembling with pre-sizing object

760daf9

isidentical requested a review from markshannon as a code owner Jul 2, 2021

the-knights-who-say-ni added the CLA signed label Jul 2, 2021

bedevere-bot added the awaiting core review label Jul 2, 2021

isidentical added the skip news label Jul 2, 2021

pablogsal approved these changes Jul 2, 2021

View changes

bedevere-bot added awaiting merge and removed awaiting core review labels Jul 2, 2021

isidentical added the 🤖 automerge label Jul 2, 2021

bedevere-bot removed the awaiting merge label Jul 2, 2021

isidentical changed the title ~~bpo-43950: optimize column table assembling with pre-sizing object~~ bpo-43950: optimize column table assembling with pre-sizing the column table Jul 3, 2021

python / cpython

bpo-43950: optimize column table assembling with pre-sizing the column table #26997

bpo-43950: optimize column table assembling with pre-sizing the column table #26997

isidentical commented Jul 2, 2021 •

edited by miss-islington

pablogsal left a comment

miss-islington commented Jul 2, 2021

isidentical commented Jul 2, 2021

miss-islington commented Jul 2, 2021

miss-islington commented Jul 2, 2021

miss-islington commented Jul 2, 2021

python / cpython

Sponsor python/cpython

bpo-43950: optimize column table assembling with pre-sizing the column table #26997

bpo-43950: optimize column table assembling with pre-sizing the column table #26997

Conversation

isidentical commented Jul 2, 2021 • edited by miss-islington

pablogsal left a comment

miss-islington commented Jul 2, 2021

isidentical commented Jul 2, 2021

miss-islington commented Jul 2, 2021

miss-islington commented Jul 2, 2021

miss-islington commented Jul 2, 2021

isidentical commented Jul 2, 2021 •

edited by miss-islington