Hi @vstinner 👋 -- here my attempt at a minimal CPython patch that is needed to make nanobind work with Py_LIMITED_API.

Also pinging @abalkin @loewis @seberg @encukou who commented on the previous PR #60074.

If these changes are acceptable, it would be great if this feature could still be included in Python 3.11.

If these changes are acceptable, it would be great if this feature could still be included in Python 3.11.

That is not possible; we passed the feature freeze with several weeks. You will need to target 3.12.

Isn't that naming a bit off? PyMetaType_FromModuleSpec sounds to me like you're creating a new meta type, but you're just creating a type with a custom meta-type. iirc the patch on bpo just called it PyType_FromModuleAndSpec which I think is the better name

@erlend-aasland This means that usable stable API interface for libraries like nanobind/pybind11 would be available starting in 2023-10, i.e., in more than a year! 😢

@timfel  1�7 PyType_FromModuleAndSpec is already taken. I was considering PyType_FromModuleAndSpecAndMeta but this seemed getting silly/long. Thoughts?

(Edit: ultimately converged towards the name PyType_FromMetaclass)

I was considering PyType_FromModuleAndSpecAndMeta

Ah, right, that was the naming proposal in the old patch. Yes, this does seem pretty long 🤔

@erlend-aasland This means that usable stable API interface for nanobind would be available in 2023-10, i.e. in more than a year! 😢

You can of course try your case with the 3.11 release manager (Pablo IIRC) :)

Quoting PEP-602:

The following three months are spent on four versioned beta releases where no new features can be added but bug fixes are still included.

We are now in the 3.11 beta phase. Note that "no new features" are emphasised in bold type.

I would hope @encukou can make a quick call (or give an opinion). The new API probably comes with no compatibility concerns the only thing would be getting the API itself wrong (and even that can be deprecated).

For NumPy, it would also be nice to have something like this, but it did not seem particularly pressing (static types work and our DTypes are de-facto immortal anyway).

I would still be 👍 with pushing this forward quickly. As per API, I can think of two things that may need quick input:

1. Petr had brought up the idea of making this one InitFromSpec (and do not include the alloc part). This was in gh-89546.
2. In principle bases should be scanned for metatypes as per my old PR. That is a bug-fix, but coupled with the tp_new check might have a tiny compat concern: It could probably be just ignored for now.

But, things don't need to be perfect and besides those two calls to make, I don't think there is any reason to not just do this. We clearly need this problem to be fixed.

I have written a separate email to appeal to @pablogsal.

Could you please add a bunch of tests for this API in Modules/_testcapimodule.c?

Could you please add a bunch of tests for this API in Modules/_testcapimodule.c?

Yes, I will do this ASAP.

Given the valid concern by @timfel that this function sounds like it creates a metatype, I will also change the name of the function to the somewhat long but more descriptive PyType_FromMetaModuleAndSpec.

(I will answer your email as soon as I can)

@pablogsal - Finished renaming, and test added.

(Could somebody push the button to approve the workflows?)

Please, add a test that checks that we raise if you provide a tp_new. We likely also need a bunch of tests around this case if the downstream class has a custom allocation schema but I need to think about this in more detail.

@pablogsal done.

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@pablogsal I have made the requested changes; please review again.

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

I have made the requested changes; please review again.

Thanks for making the requested changes!

@erlend-aasland: please review the changes made to this pull request.

For posterity, here is cursed code that emulates PyType_FromMetaclass on Python versions without this functionality (naturally, this then cannot target the stable ABI).

It calls PyType_FromSpec to initializes a type from an existing PyType_Spec, allocates another type from the metaclass via PyType_GenericAlloc, then runs a memcpy to copy the filled-out slots and then does some manual fixups. Finally, the initial type is garbage-collected. 🙈

It might be worth it to put a copy of PyType_FromModuleAndSpec in pythoncapi-compat, for projects that need it sooner that 3.12. (Patch releases of CPython won't change much, so for the older versions pythoncapi-compat can get away with relying on more implementation details than normal.)
Probably best to get #28748 in first, though.

Coincidentally, JPype's @Thrameos just chimed in, and they're struggling with the same issue (among others). The JPype workaround looks less cursed, but more repetitive.

@Thrameos, PyType_FromMetaclass is being added to 3.12, and it looks like it'll be worth it to share maintenance of a backport to earlier versions.
See proposed new docs for an overview. (That's from #28748, which follows up on this PR.)

Heads up for @vstinner: We'll probably want to put PyType_FromMetaclass in pythoncapi-compat. It would be much larger than the tiny helpers that are already there. Hopefully that won't be an issue.

@encukou If any of my implementations look useable I would happily gift them to the Python community. Unfortunately my employer (Department of Energy) has declined to sign the Python contribution agreement and barred me from signing it personally. I am permitted to contribute to open source projects (Ie I can submit code to public domain, GPL, or Apache such as I have done with JPype or any project that does not require a signed agreement) , and someone else can incorporate it, but our legal team indicates that number of hours required to review a signed license requires a funded billable account which I lack. Hence, leaving me unable to contribute directly.

IMO it's best to start with a copy of CPython's own code, anyway.