GH-99554: Pack location tables more effectively #99556

brandtbucher · 2022-11-17T00:45:21Z

This shrinks the size of .pyc files by about 10%. It also shrinks the memory footprint of all code objects in the process.

Issue: .pyc files are larger than they need to be #99554

iritkatriel · 2022-11-17T08:47:25Z

Python/compile.c

+static inline bool
+same_location(location a, location b)
+{
+    if (a.lineno < 0 && b.lineno < 0) {


I’d check also that they are the same value in case we will want to distinguish between ‘no location’ and ‘unknown location’ or something like that.

(I’m assuming you don’t rely on the full equality check because we’re inconsistent about what the other 3 values are).

Should an "unknown location" ever exist? That sounds like a compiler bug.

I'm thinking of cases where we just want to take the location of the previous instruction. Currently we just say "NO_LOCATION", but would we ever want to distinguish between this case and the case where there truly is no location (because the instructions are fake)? I don't know, but we might not want to rule it out.

(I’m assuming you don’t rely on the full equality check because we’re inconsistent about what the other 3 values are).

Actually, it appears that we can remove this special case entirely. Just using the full equality check results in the exact same PYC sizes for the entire stdlib.

netlify · 2022-12-15T18:00:55Z

✅ Deploy Preview for python-cpython-preview ready!

Name	Link
🔨 Latest commit	`be7eb0a`
🔍 Latest deploy log	https://app.netlify.com/sites/python-cpython-preview/deploys/639b60d4868560000855722c
😎 Deploy Preview	https://deploy-preview-99556--python-cpython-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

bedevere-bot · 2022-12-22T00:50:29Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x SLES 3.x has failed when building commit 3c033a2.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/540/builds/4270) and take a look at the build logs.
Check if the failure is related to this commit (3c033a2) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/540/builds/4270

Summary of the results of the build (if available):

== Tests result: ENV CHANGED ==

414 tests OK.

10 slowest tests:

test_concurrent_futures: 2 min 42 sec
test_tools: 2 min 41 sec
test_multiprocessing_spawn: 2 min 29 sec
test_asyncio: 1 min 49 sec
test_multiprocessing_forkserver: 1 min 20 sec
test_multiprocessing_fork: 1 min 19 sec
test_subprocess: 1 min 15 sec
test_signal: 1 min 9 sec
test_capi: 1 min 2 sec
test_venv: 58.4 sec

1 test altered the execution environment:
test_asyncio

18 tests skipped:
test_check_c_globals test_devpoll test_ioctl test_kqueue
test_launcher test_msilib test_nis test_peg_generator
test_perf_profiler test_startfile test_tix test_tkinter test_ttk
test_winconsoleio test_winreg test_winsound test_wmi
test_zipfile64

Total duration: 6 min 4 sec

Click to see traceback logs

remote: Enumerating objects: 20, done.        
remote: Counting objects:   5% (1/20)        
remote: Counting objects:  10% (2/20)        
remote: Counting objects:  15% (3/20)        
remote: Counting objects:  20% (4/20)        
remote: Counting objects:  25% (5/20)        
remote: Counting objects:  30% (6/20)        
remote: Counting objects:  35% (7/20)        
remote: Counting objects:  40% (8/20)        
remote: Counting objects:  45% (9/20)        
remote: Counting objects:  50% (10/20)        
remote: Counting objects:  55% (11/20)        
remote: Counting objects:  60% (12/20)        
remote: Counting objects:  65% (13/20)        
remote: Counting objects:  70% (14/20)        
remote: Counting objects:  75% (15/20)        
remote: Counting objects:  80% (16/20)        
remote: Counting objects:  85% (17/20)        
remote: Counting objects:  90% (18/20)        
remote: Counting objects:  95% (19/20)        
remote: Counting objects: 100% (20/20)        
remote: Counting objects: 100% (20/20), done.        
remote: Compressing objects:  12% (1/8)        
remote: Compressing objects:  25% (2/8)        
remote: Compressing objects:  37% (3/8)        
remote: Compressing objects:  50% (4/8)        
remote: Compressing objects:  62% (5/8)        
remote: Compressing objects:  75% (6/8)        
remote: Compressing objects:  87% (7/8)        
remote: Compressing objects: 100% (8/8)        
remote: Compressing objects: 100% (8/8), done.        
remote: Total 11 (delta 9), reused 4 (delta 3), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to '3c033a2e6fbde56f904aeca138141be6564341cf'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 3c033a2e6f GH-99554: Pack location tables more effectively (GH-99556)
Switched to and reset branch 'main'

make: *** [Makefile:1892: buildbottest] Error 3

gvanrossum · 2022-12-22T01:01:56Z

Python/compile.c

+static inline bool
+same_location(location a, location b)
+{
+    return a.lineno == b.lineno && 


According to git diff you have a trailing space here. Surprised CI didn't catch it?

Wow, nice catch! I'll fix it.

brandtbucher added 4 commits Nov 15, 2022

Refactor location emission to not use instructions

2b90c42

Compress location entries more effectively

6d96e8c

blurb add

f191f95

fixup

24ce84f

brandtbucher added performance Performance or resource usage interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Nov 17, 2022

brandtbucher requested a review from markshannon as a code owner Nov 17, 2022

brandtbucher self-assigned this Nov 17, 2022

brandtbucher requested a review from iritkatriel as a code owner Nov 17, 2022

bedevere-bot mentioned this pull request Nov 17, 2022

.pyc files are larger than they need to be #99554

Open

bedevere-bot added the awaiting core review label Nov 17, 2022

iritkatriel reviewed Nov 17, 2022

View changes

Catch up with main

be7eb0a

brandtbucher added 2 commits Dec 17, 2022

Catch up with main

37909a4

Remove special case

d616c2d

brandtbucher merged commit 3c033a2 into python:main Dec 22, 2022
16 checks passed

bedevere-bot removed the awaiting core review label Dec 22, 2022

gvanrossum reviewed Dec 22, 2022

View changes

markshannon mentioned this pull request Dec 22, 2022

code.co_positions behaviour does not match documentation #100117

Open

GH-99554: Pack location tables more effectively #99556

GH-99554: Pack location tables more effectively #99556

brandtbucher commented Nov 17, 2022 •

edited by bedevere-bot

iritkatriel Nov 17, 2022

markshannon Nov 28, 2022

iritkatriel Nov 28, 2022

brandtbucher Dec 22, 2022

netlify bot commented Dec 15, 2022 •

edited

bedevere-bot commented Dec 22, 2022

gvanrossum Dec 22, 2022

brandtbucher Dec 22, 2022

GH-99554: Pack location tables more effectively #99556

GH-99554: Pack location tables more effectively #99556

Conversation

brandtbucher commented Nov 17, 2022 • edited by bedevere-bot

iritkatriel Nov 17, 2022

Choose a reason for hiding this comment

markshannon Nov 28, 2022

Choose a reason for hiding this comment

iritkatriel Nov 28, 2022

Choose a reason for hiding this comment

brandtbucher Dec 22, 2022

Choose a reason for hiding this comment

netlify bot commented Dec 15, 2022 • edited

✅ Deploy Preview for python-cpython-preview ready!

bedevere-bot commented Dec 22, 2022

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

gvanrossum Dec 22, 2022

Choose a reason for hiding this comment

brandtbucher Dec 22, 2022

Choose a reason for hiding this comment

brandtbucher commented Nov 17, 2022 •

edited by bedevere-bot

netlify bot commented Dec 15, 2022 •

edited