Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-90928: Statically Initialize the Keywords Tuple in Clinic-Generated Code #95860

Conversation

ericsnowcurrently
Copy link
Member

@ericsnowcurrently ericsnowcurrently commented Aug 10, 2022

We only statically initialize for core code and builtin modules. Extension modules still create the tuple at runtime. We'll solve that part of interpreter isolation separately.

This change includes generated code. The significant changes are in:

  • Tools/clinic/clinic.py
  • Python/getargs.c
  • Include/cpython/modsupport.h
  • Makefile.pre.in (re-generate global strings after running clinic)
  • very minor tweaks to Modules/_codecsmodule.c and Python/Python-tokenize.c

All other changes are generated code (clinic, global strings).

@ericsnowcurrently ericsnowcurrently marked this pull request as ready for review Aug 10, 2022
#define NUM_KEYWORDS {num_keywords}
#if NUM_KEYWORDS == 0
Copy link
Contributor

@erlend-aasland erlend-aasland Aug 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, what is the reason for leaving this condition to the preprocessor iso. just outputting the two blocks (keywords == 0, keywords != 0) conditionally from AC? I've probably overlooked some scenario here :)

Copy link
Member Author

@ericsnowcurrently ericsnowcurrently Aug 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found it made it easier to identify what was going on. However, that might not be as valuable now that I've ironed out the kinks and the change is working well. I'll see what it looks like with the unnecessary condition removed.

Copy link
Member Author

@ericsnowcurrently ericsnowcurrently Aug 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now I remember... clinic.py works by building a template for generating code and then applying values to that template in a later step. The code to which you referred is part of that generation step. We don't have the actual num_keywords value at that point, so we generate all the cases. It isn't great but we'd have to (almost) completely re-write clinic.py if we wanted to do this in a simpler way. 🙁

Copy link
Contributor

@erlend-aasland erlend-aasland Aug 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes, I remember this from an earlier issue or PR. Perhaps we should leave a comment in there in case such a restructuring/rewrite happens in the future?

Copy link
Contributor

@erlend-aasland erlend-aasland Aug 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at this again, it should be possible to deduce num_keywords during the template generation. We've got the Function object f, which contains the parameters dict. Each value in the dict is a Parameter object, which contains the information needed to tell if a parameter can be a keyword parameter or not.

I'll submit a PR against your fork with a proof-of-concept.

Copy link
Member Author

@ericsnowcurrently ericsnowcurrently Aug 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The situation is inherent to the clinic.py implementation, so a comment here feels out of place. If anything, it would make sense to add some in-tree documentation about the implementation: either a long docstring at the top of clinic.py or an adjacent README. However, that would happen in a separate PR.

Copy link
Contributor

@erlend-aasland erlend-aasland Aug 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding adding a comment: agreed.

Regarding my suggestion of calculating num_keywords during template generation, in order to generate nicer AC output: see gh-95907

@kumaraditya303
Copy link
Contributor

kumaraditya303 commented Aug 11, 2022

LGTM.

We only statically initialize for core code and builtin modules. Extension modules still create the tuple at runtime.

Yeah, its better to solve it for builtins now, extensions modules can be solved later on.
FTR, I tried solving both at once 1 and it worked on Linux but does not work on Windows because it does not allows taking addresses of DLL loaded symbols.

Footnotes

  1. https://github.com/kumaraditya303/cpython/pull/4

@ericsnowcurrently
Copy link
Member Author

ericsnowcurrently commented Aug 11, 2022

Yeah, its better to solve it for builtins now, extensions modules can be solved later on.
FTR, I tried solving both at once 2 and it worked on Linux but does not work on Windows because it does not allows taking addresses of DLL loaded symbols.

I'm glad you agree. 😄 Splitting it up this way made the most sense.

@ericsnowcurrently ericsnowcurrently added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 11, 2022
@bedevere-bot
Copy link

bedevere-bot commented Aug 11, 2022

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit 3b308c4 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 11, 2022
@erlend-aasland
Copy link
Contributor

erlend-aasland commented Aug 11, 2022

CC. @colorfulappl, regarding your PR #32092. This PR touches some of the same parts of AC and getarg.c.

@ericsnowcurrently ericsnowcurrently merged commit 6f6a4e6 into python:main Aug 11, 2022
13 checks passed
@ericsnowcurrently ericsnowcurrently deleted the global-objects-pyargs-parser-keywords-init branch Aug 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants