@mdickinson You recently fixed a few things in ctypes, perhaps you could kindly help me find someone to review these changes here, please? Thanks!

Gentle ping to @abalkin or @pitrou , this might interest y'all.

cc @serhiy-storchaka

The git history of this PR is a bit dirty at the moment. I can clean it up, if there's interest.

I can also polish the code more, if you give me some hints about what you want to see. I also have a repository with lots of extra tests via Python Hypothesis. (Alas, we can't use Python Hypothesis to test the standard library? Or can we?)

In general, this PR breaks bug-for-bug compatibility with the past, but does not change documented behaviour.

This is the largest challenge I in getting this change out safely. Our API behavior is wrong, but it has been shipped in a couple of CPython releases already, so we probably need to assume that some users depend on it. That means documenting properly what the actual behavior was and adding an API flag to pick which behavior people want.

If it appears this API has wide usage expecting it, defaulting to the existing behavior for a couple of releases before we can flip the default. If usage of the wrong behavior is rare, flipping the default while providing a flag for the old behavior could be done more rapidly than our usual deprecation period there.

Searching a large corpus of code internally at work, usage of the three item tuple in ctypes Structure context (meaning bit field width) is not common. But it does come up.

• https://github.com/dmlc/dlpack/blob/main/apps/numpy_dlpack/dlpack/dlpack.py#L70
• https://github.com/pyserial/pyserial/blob/master/serial/win32.py
• https://github.com/vivisect/vivisect/blob/master/vtrace/platforms/linux.py#L123
• https://github.com/nwhitehead/pyfluidsynth/blob/master/fluidsynth.py#L251
• https://github.com/ROCmSoftwarePlatform/Tensile/blob/develop/Tensile/Code.py#L710
• (plus in some hardware vendor code that doesn't appear OSS so I can't link to it).

@gpshead Thanks for having a look. I agree.

Thanks for digging up those uses of ctypes. I'll try to run the test suites of those projects, perhaps.

Some complications: the existing code often leads to undefined behaviour. Thus in a strict sense, we can not guarantee bug-for-bug compatibility, even if we don't change anything. Of course, in practice it's probably fine as long as we keep the old code in place (with a feature toggle as you suggest) and don't change compilers versions nor compiler flags too much.

The undefined behaviour in the old version was what lead me to investigate ctypes in the first place. See #96823 for context.

As an ambitious goal, in addition to the providing a proper fix behind a feature toggle, perhaps we can also fix (some of) the undefined behaviour in the old version, without breaking anything? I'm not sure. Do you have an opinion?

ideally - if tests on those examples of use still happen to work as written, we can probably just go ahead and make the change without a breaking change cycle. Ultimately I want us to be rid of the existing bad/old behavior entirely. It's mostly a matter of seeing if it is feasible to give people depending on that (possibly undefined) behavior a heads up and time to fix things to work with the correct implementation.

I am working my way through the list of users that you found.

dlpack doesn't seem to actually use bitfields, or at least doesn't let ctypes handle bitfields, as far as I can tell. (But let me double check that it still works again.)

vivisect is more interesting. vivisect/vivisect#593 / vivisect/vivisect#594 has my attempts at making it work with Python 3.12 at all.

I'm waiting for local vivisect test runs now, and will look at pyfluidsynth in the meantime.

@gpshead The problematic code is both gnarly and rarely executed: only once when we are reading in a struct / union definition. That shouldn't be on anyone's critical path: creation, reading or writing of those structs is done by other code.

That should make it a prime candidate for implementing in Python instead of C? Do you have an opinion?

That would make for a larger diff, maybe. But it's definitely easier to write readable and maintainable code in Python than in C.

Hmm, it looks like if I want to test the changes here with client code, I might be better off back-porting to an older Python (like 3.11 or even older), so that eg numpy works.

(I'm going to guess that ctypes changes so rarely the .diff of this branch applies pretty cleanly to 3.11 for testing purposes)

regarding writing the critical path for this in Python instead of C... that sounds like a great idea. I agree that Structure classes should primarily be one time definition costs.

(I'm going to guess that ctypes changes so rarely the .diff of this branch applies pretty cleanly to 3.11 for testing purposes)

See #102774

The main change necessary was _Py_IDENTIFIER vs _Py_ID.