Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.pyc files are larger than they need to be #99554

Open
brandtbucher opened this issue Nov 17, 2022 · 0 comments
Open

.pyc files are larger than they need to be #99554

brandtbucher opened this issue Nov 17, 2022 · 0 comments
Assignees
Labels
3.12 interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage stdlib Python modules in the Lib dir

Comments

@brandtbucher
Copy link
Member

brandtbucher commented Nov 17, 2022

Python 3.11 made .pyc files almost twice as large. There are two main reasons for this:

  • PEP 659 made the bytecode stream ~3x as large as 3.10.
  • PEP 657 made the location tables ~9x as large as 3.10.

(Note that these effects compound each other, since longer bytecode means more location entries.)

However, there is low-hanging fruit for improving this situation in 3.12:

  • Bytecode can be compressed using a fairly simple scheme (one byte for instructions without an oparg, two bytes for instructions with an oparg, and zero bytes for CACHE entries). This results in serialized bytecode that is ~66% smaller than 3.11.
  • The location table format already has a mechanism for compressing multiple code units into a single entry. Currently it's only used for EXTENDED_ARGs and CACHEs corresponding to a single instruction, but with slight changes the compiler can use the same mechanism to share location table entries between adjacent instructions. This is a double-win, since it not only makes .pyc files smaller, but also shrinks the memory footprint of all code objects in the process. Experiments show that this makes location tables ~33% smaller than 3.11.

When both of these optimizations are applied, .pyc files become ~33% smaller than 3.11. This is only ~33% larger than 3.10, despite all of the rich new debugging information present.

Linked PRs

@brandtbucher brandtbucher added performance Performance or resource usage interpreter-core (Objects, Python, Grammar, and Parser dirs) stdlib Python modules in the Lib dir 3.12 labels Nov 17, 2022
@brandtbucher brandtbucher self-assigned this Nov 17, 2022
miss-islington pushed a commit that referenced this issue Dec 22, 2022
Automerge-Triggered-By: GH:brandtbucher
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.12 interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage stdlib Python modules in the Lib dir
Projects
None yet
Development

No branches or pull requests

1 participant