Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gh-94673: Add Per-Interpreter tp_subclasses for Static Builtin Types #95301

Merged
merged 23 commits into from Aug 5, 2022
Merged

gh-94673: Add Per-Interpreter tp_subclasses for Static Builtin Types #95301

merged 23 commits into from Aug 5, 2022
+120 −31

Conversation

ericsnowcurrently
Copy link
Member

@ericsnowcurrently ericsnowcurrently commented Jul 26, 2022 •
edited by bedevere-bot

@ericsnowcurrently
Add a note about tp_cache.
7b299d1
@ericsnowcurrently
Hide tp_subclasses behind functions.
4fefd9b
@ericsnowcurrently
Distinquish between static and heap types.
129ca1f
@ericsnowcurrently
Move tp_subclasses for static types to PyInterpreterState.
183b6b5
@ericsnowcurrently
Add a note about heap types and tp_subclasses.
a707c38
@ericsnowcurrently
Use PyDict_GetItemWithError().
5dd2474
@ericsnowcurrently
Add lookup___hash__().
80b1d26
@ericsnowcurrently
Handle the case of unhashable classes.
8bced84
@ericsnowcurrently
Really handle the case of unhashable classes.
533951d
@ericsnowcurrently
Stop special-casing heap types for tp_subclasses.
f9ab261
@ericsnowcurrently
Revert "Stop special-casing heap types for tp_subclasses."
ccabeb5 
This reverts commit 0046b57446a26114fdf33d11a31a0fc84e7a764a.
@ericsnowcurrently
Point to the relevant GH issue.
df204e8
@ericsnowcurrently
Only store builtins state on the interpreter state.
3529f0f
@ericsnowcurrently
state.subclasses -> state.tp_subclasses
47c6ded
@ericsnowcurrently
Do not set type.tp_subclasses for static builtin types.
eade37b
@ericsnowcurrently
Store the index in tp_subclasses.
5687c8e
@ericsnowcurrently ericsnowcurrently removed the request for review from markshannon Jul 26, 2022
@ericsnowcurrently ericsnowcurrently mentioned this pull request Jul 26, 2022
Closed
@ericsnowcurrently ericsnowcurrently added the 🔨 test-with-buildbots label Jul 27, 2022
@bedevere-bot
Copy link

@bedevere-bot bedevere-bot commented Jul 27, 2022

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit 2fa815e 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots label Jul 27, 2022
kumaraditya303
kumaraditya303 reviewed Jul 27, 2022
View changes
Objects/structseq.c Outdated Show resolved Hide resolved
kumaraditya303
kumaraditya303 reviewed Jul 27, 2022
View changes
Objects/typeobject.c Outdated Show resolved Hide resolved
kumaraditya303
kumaraditya303 approved these changes Jul 27, 2022
View changes
Copy link
Contributor

@kumaraditya303 kumaraditya303 left a comment

LGTM, I have left some questions.

FWIW, we could also get rid of the new type flag and use tp_subclasses as a tagged pointer and steal one bit to check if the tp_subclasses is per interpreter or local, I'll probably do that but for now this is fine.

kumaraditya303
kumaraditya303 reviewed Jul 27, 2022
View changes
Objects/typeobject.c Outdated Show resolved Hide resolved
@encukou
Copy link
Member

@encukou encukou commented Jul 27, 2022 •
edited

I'm afraid this isn't backwards compatible. The tp_subclasses docs do say it's internal only, but do currently guarantee that it's a list, so IMO it's valid for API users to treat it as a PyObject*, and expect exceptions (rather than crashes).

Consider:

  • Seeking a SC exception from Python's backcompat policy
  • Renaming the field to _tp_subclasses, so anyone currently using it gets a compiler error
  • Adding the change to What's New so people know what's up
  • Removing the field from docs
  • Getting a PEP accepted for changes like this, to find unintended consequences
  • (edit) Changing the type to void* since it's not PyObject* any more

@ericsnowcurrently
Copy link
Member Author

@ericsnowcurrently ericsnowcurrently commented Jul 27, 2022

FWIW, tp_subclasses is already a dict and probably has been for quite a while.

@kumaraditya303
Copy link
Contributor

@kumaraditya303 kumaraditya303 commented Jul 27, 2022 •
edited

but do currently guarantee that it's a list,

The current implementation uses a dictionary to store subclasses as a mapping of PyLongObject * to subclass. On the Python level it is a list but not at C level.

so IMO it's valid for API users to treat it as a PyObject*, and expect exceptions (rather than crashes).

Why do you expect it to raise an exception when you dereference a pointer which is clearly documented to be a internal implementation detail?

@encukou
Copy link
Member

@encukou encukou commented Jul 28, 2022

I'm not saying this is a bad change. I'm asking to be mindful of the possible effects on code other than CPython that might look at this field.

FWIW, tp_subclasses is already a dict and probably has been for quite a while.

Yes, the docs are wrong. I don't think that's a free pass make this an invalid PyObject* pointer, though.
Bad docs just mean that people will experiment, or read the source, to see what things do.

Why do you expect it to raise an exception when you dereference a pointer which is clearly documented to be a internal implementation detail?

Well, I do expect to be able to inspect internal details, e.g. for debugging.

@ericsnowcurrently
Copy link
Member Author

@ericsnowcurrently ericsnowcurrently commented Jul 28, 2022

Hey, the feedback is super helpful! Thanks!

I'll go ahead and add a whatsnew entry about the special-casing of tp_subclasses for static builtin types.

@ericsnowcurrently
Copy link
Member Author

@ericsnowcurrently ericsnowcurrently commented Jul 29, 2022

Making tp_sublcasses void * is a good idea.

@ericsnowcurrently ericsnowcurrently added the 🔨 test-with-buildbots label Jul 29, 2022
@bedevere-bot
Copy link

@bedevere-bot bedevere-bot commented Jul 29, 2022

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit dace9c4 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots label Jul 29, 2022
@encukou encukou mentioned this pull request Jul 29, 2022
Merged
@encukou
Copy link
Member

@encukou encukou commented Jul 29, 2022

Thanks!
I left some more suggestions on the docs in ericsnowcurrently#31

encukou and others added 2 commits Aug 1, 2022
@encukou @ericsnowcurrently
Adjust docs for tp_subclasses
e7f580d 
- Member documentation describes the current state, for users.
- The ``versionchanged`` note describes the change.
- The What's New entry is, ideally, usable for someone stuck with
  maintaining a mysterious legacy codebase that stopped working.

Details of internals are left out -- if it isn't enough to look
at the source, they should be documented in the devguide rather
than user-facing docs.
@ericsnowcurrently
lookup_subclasses() never fails.
6b19763
@ericsnowcurrently ericsnowcurrently merged commit 87154d8 into python:main Aug 5, 2022
14 checks passed
@ericsnowcurrently ericsnowcurrently deleted the shareable-static-types-tp_subclasses branch Aug 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kumaraditya303 kumaraditya303

Assignees
No one assigned
Labels
skip news
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@ericsnowcurrently @bedevere-bot @encukou @kumaraditya303