PyType_FromSpec refuses to create classes with tp_new #103968

encukou · 2023-04-28T14:17:32Z

As reported in #60074, since PyType_FromMetaclass was added, other functions from the PyType_FromSpec family refuse to create classes whose metaclass has a non-default tp_new. IMO this is the correct default behaviour -- we don't have the arguments to call tp_new with, and skipping it may be dangerous.
Nevertheless, it's a backwards-incompatible change. It should only be made after a deprecation period.

I'm working on a fix.

Linked PRs

gh-103968: Deprecate creating heap types whose metaclass has custom tp_new. #103972
gh-103968: PyType_FromMetaclass: Allow metaclasses with tp_new=NULL #105386
[3.12] gh-103968: PyType_FromMetaclass: Allow metaclasses with tp_new=NULL (GH-105386) #105697
gh-103968: What's New: Add porting hints for PyType_From with metaclasses #105698
[3.12] gh-103968: What's New: Add porting hints for PyType_From with metaclasses (GH-105698) #106619

The text was updated successfully, but these errors were encountered:

…stom tp_new. (That's a mouthful of an edge case!)

…p_new. (GH-103972) (That's a mouthful of an edge case!) Co-authored-by: Barney Gale <barney.gale@gmail.com>

encukou · 2023-05-03T13:17:48Z

Deprecation period started.

gpshead · 2023-06-02T22:15:43Z

The 3.12 docs now say "The tp_new of the metaclass is ignored. which may result in incomplete initialization." statements, but it isn't clear if this is a part of the versionchanged in 3.12 behavior, or if this was always the case.

Basically, we need explicit instructions on what code finding itself in the situation that protobuf protocolbuffers/protobuf#12186 finds itself in is supposed to do to reconcile the situation.

3.12's "What's New" does not directly address this today. It just talks about tp_new being ignored as if that is a behavior change rather than if it was always ignored without being documented as such or what should be done to get the previous behavior.

gpshead · 2023-06-02T22:23:42Z

Late discussion on #60074 which created the TypeError that this issue properly made into a DeprecationWarning has suggestions for a workaround but they definitely qualify as hacks. Deeper understanding of why that tp_new was set and what it's goal was and when it was ever called or not in the past is likely required.

gpshead · 2023-06-02T22:24:35Z

(marking deferred-blocker as we should at least address the documentation on this during the 3.12 beta phase before release)

encukou · 2023-06-05T09:00:18Z

Previously, the metaclass was ignored entirely, so its tp_new wasn't considered.

The old behaviour was dangerous. Consider:

You should be able to use PyType_FromMetaclass to extend a type you know nothing about (e.g. add a mixin).
A metaclass can override tp_new to override what PyType_FromMetaclass normally does.

PyType_FromMetaclass cannot do the right thing if the metaclass overrides tp_new.
The old behaviour -- skipping tp_new because the metaclass wasn't considered at all -- works in simple cases, but as soon as tp_new does something important, it'll blow up. Hence the deprecation.

If you control the metaclass and know that tp_new is safe to skip, IMO the proper workaround is to stop using either tp_new or PyType_FromMetaclass. They're incompatible with each other.
In the future, we probably need a tp_cinit, so you can write an initializer/validator without overriding type creation entirely.

The hack -- unsetting metaclass tp_new while creating a type -- is explicitly writing out what's going on. It's ugly, but that's because the previous behaviour is ugly.

I'll add explicit porting instructions to What's New, including the hack.

encukou · 2023-06-05T14:01:28Z

I looked at protobuf.

protobuf uses PyType_FromSpecWithBases to create ScalarMapContainer and MessageMapContainer, which derive from collections.abc.MutableMapping, so their metaclass is abc.ABCMeta.

abc.ABCMeta defines __new__, meaning it overrides how ABC types should be constructed.
This __new__ is not called on protobuf's container types, which is, IMO, a bug.

I didn't find out how to get a hold of protobuf's containers, but here's a small reproducer:

C extension reproducer

mymod.c:

#include <assert.h>
#include <Python.h>

static PyType_Slot MyType_slots[] = {
    {0, NULL},
};

PyType_Spec MyType_spec = {
    "mymod.MyType", sizeof(PyObject), 0,
    Py_TPFLAGS_DEFAULT, MyType_slots};

PyObject *
get_mytype(PyObject *self, PyObject *unused)
{
  PyObject *abc = PyImport_ImportModule("collections.abc");
  assert(abc); // XXX error handling

  PyObject *mutable_mapping = PyObject_GetAttrString(abc, "MutableMapping");
  assert(mutable_mapping); // XXX error handling

  PyObject *MyType = PyType_FromSpecWithBases(&MyType_spec, mutable_mapping);
  assert(MyType); // XXX error handling

  Py_DECREF(abc);
  Py_DECREF(mutable_mapping);
  return MyType;
}

PyObject *
reproducer(PyObject *self, PyObject *unused)
{
  return PyRun_String("\
t = get_mytype()                                  \n\
for base in t.mro():                              \n\
    print(getattr(base, '_abc_impl', None), base) \n\
", Py_file_input, PyModule_GetDict(self), NULL);

}

PyMethodDef mod_methods[] = {
  {"get_mytype", get_mytype, METH_NOARGS, NULL},
  {"reproducer", reproducer, METH_NOARGS, NULL},
  {NULL},
};

PyModuleDef mod = {
  .m_base = PyModuleDef_HEAD_INIT,
  .m_name = "mymod",
  .m_methods = mod_methods,
};

PyObject *
PyInit_mymod(void) {
  return PyModuleDef_Init(&mod);
}

Hacky non-distutils compilation/run (Linux, installed Python):

V=3.11
gcc $(python$V-config --cflags --ldflags) -lpython$V$(python$V-config --abiflags) -shared mymod.c -o mymod.so
python$V -c 'import mymod; mymod.reproducer()'

Result:

<_abc._abc_data object at 0x7f8ba8aa2700> <class 'mymod.MyType'>
<_abc._abc_data object at 0x7f8ba8aa2700> <class 'collections.abc.MutableMapping'>
<_abc._abc_data object at 0x7f8ba8aa2040> <class 'collections.abc.Mapping'>
<_abc._abc_data object at 0x7f8ba8aa1640> <class 'collections.abc.Collection'>
<_abc._abc_data object at 0x7f8ba8aa12c0> <class 'collections.abc.Sized'>
<_abc._abc_data object at 0x7f8ba8aa0c40> <class 'collections.abc.Iterable'>
<_abc._abc_data object at 0x7f8ba8aa14c0> <class 'collections.abc.Container'>
None <class 'object'>

Note that mymod.MyType shares its _abc_data with its superclass. This means __new__ of abc.ABCMeta, which assigns a class-specific instance here, was skipped.

In this particular case, it appears the bug is harmless -- the type for which __new__ wasn't called isn't instantiated directly, so the setup __new__ does is not used?
But it's a fragile kind of harmlessness. The metaclass should be allowed to assume its __new__ is called.
And PyType_FromSpec can't call tp_new. It wants to call tp_alloc, set slots and call PyType_Ready itself, but tp_new normally does all of that.

I don't know what the correct general solution is, but IMO, the current behaviour is worth deprecating.

encukou · 2023-06-06T14:09:43Z

It would be safe, and useful in some cases, to allow tp_new=NULL (i.e. the metaclass can't be instantiated from Python). I'll try to get that in before the What's New PR.

For What's New, here's the text I have now:

Since tp_new overrides almost everything PyType_From* functions do,
the two are incompatible with each other.
The existing behavior -- ignoring the metaclass for several steps
of type creation -- is unsafe in general, since (meta)classes assume that
tp_new was called.
There is no simple general workaround. One of the following may work for you:

If you control the metaclass, avoid using tp_new in it:
- If initialization can be skipped, it can be done in
  :c:member:~PyTypeObject.tp_init instead.
- If the metaclass doesn't need to be instantiated from Python,
  set its tp_new to NULL using
  the :const:Py_TPFLAGS_DISALLOW_INSTANTIATION flag.
  This makes it acceptable for PyType_From* functions.
Avoid PyType_From* functions: if you don't need C-specific features
(slots or setting the instance size), create types by :ref:calling <call>
the metaclass.
If you know the tp_new can be skipped safely, filter the deprecation
warning out using :func:warnings.catch_warnings from Python.

…H-105386)

…NULL (pythonGH-105386) (cherry picked from commit 2b90796) Co-authored-by: Petr Viktorin <encukou@gmail.com>

…=NULL (GH-105386) (GH-105697) gh-103968: PyType_FromMetaclass: Allow metaclasses with tp_new=NULL (GH-105386) (cherry picked from commit 2b90796) Co-authored-by: Petr Viktorin <encukou@gmail.com>

haberman · 2023-06-13T19:53:37Z

protobuf uses PyType_FromSpecWithBases to create ScalarMapContainer and MessageMapContainer, which derive from collections.abc.MutableMapping, so their metaclass is abc.ABCMeta.

Protobuf is using MutableMapping as a mixin. The goal is that we can define only a core set of methods (__getitem__, __setitem__, etc.) and get correct implementations of all other MutableMapping methods "for free" (__contains__, keys, etc.).

abc.ABCMeta defines __new__, meaning it overrides how ABC types should be constructed.
This __new__ is not called on protobuf's container types, which is, IMO, a bug.

It looks like the primary purpose of abc.ABCMeta is to add a register() method to the class, so that you can call MutableMapping.register(Foo), which will cause isinstance(Foo(), MutableMapping) to return true.

We do not want that functionality in this case. We do not want people to be able to call ScalarMapContainer.register(Foo), because ScalarMapContainer is a concrete class, not an abstract class.

I suspect that anyone who uses MutableMapping as a mixin is in the same boat as us. I would consider it a bug if a class becomes an ABC merely because it mixed in an ABC.

But this seems to be what happens if you mix in an ABC in pure Python:

from collections.abc import MutableMapping

class MyMapping(MutableMapping):
    pass

class OtherMapping:
    pass

# This works -- MyMapping has become an ABC!
MyMapping.register(OtherMapping)
assert isinstance(OtherMapping(), MyMapping)

The simplest fix would be to stop inheriting from MutableMapping. Maybe we could "manually" mix in the other methods without using inheritance:

from collections.abc import MutableMapping

class ScalarMapContainer:
    # ...

MutableMapping.register(ScalarMapContainer)

# Mixin the methods manually.
ScalarMapContainer.__contains__ = MutableMapping.__contains__
ScalarMapContainer.keys = MutableMapping.keys
ScalarMapContainer.items = MutableMapping.items
ScalarMapContainer.values = MutableMapping.values
# ...

commit 74c2422 Author: Guido van Rossum <guido@python.org> Date: Wed Jun 14 08:19:24 2023 -0700 Update DSL docs for cases generator (python#105753) * Clarify things around goto error/ERROR_IF a bit * Remove docs for super-instructions * Add pseudo; fix heading markup commit 1d857da Author: Mark Shannon <mark@hotpy.org> Date: Wed Jun 14 16:15:08 2023 +0100 pythonGH-77273: Better bytecodes for f-strings (pythonGH-6132) commit 307bcea Author: Victor Stinner <vstinner@python.org> Date: Wed Jun 14 17:00:40 2023 +0200 xmlrpc.client uses datetime.datetime.isoformat() (python#105741) Reimplement _iso8601_format() using the datetime isoformat() method. Ignore the timezone. Co-Authored-by: Paul Ganssle <1377457+pganssle@users.noreply.github.com> commit 7b1f0f2 Author: Alex Waygood <Alex.Waygood@Gmail.com> Date: Wed Jun 14 15:58:41 2023 +0100 pythongh-105570: Deprecate unusual ways of creating empty TypedDicts (python#105780) Deprecate two methods of creating typing.TypedDict classes with 0 fields using the functional syntax: `TD = TypedDict("TD")` and `TD = TypedDict("TD", None)`. Both will be disallowed in Python 3.15. To create a TypedDict class with 0 fields, either use `class TD(TypedDict): pass` or `TD = TypedDict("TD", {})`. commit d32e8d6 Author: TATHAGATA ROY <royzen9495@gmail.com> Date: Wed Jun 14 19:51:30 2023 +0530 pythongh-105196: Fix indentations of section headings in C API docs (python#105672) commit 4a113e2 Author: Alex Waygood <Alex.Waygood@Gmail.com> Date: Wed Jun 14 15:19:27 2023 +0100 Typing docs: move the deprecated stuff below the non-deprecated stuff (python#105781) commit 7199584 Author: Mark Shannon <mark@hotpy.org> Date: Wed Jun 14 13:46:37 2023 +0100 pythonGH-100987: Allow objects other than code objects as the "executable" of an internal frame. (pythonGH-105727) * Add table describing possible executable classes for out-of-process debuggers. * Remove shim code object creation code as it is no longer needed. * Make lltrace a bit more robust w.r.t. non-standard frames. commit ad56340 Author: Alex Waygood <Alex.Waygood@Gmail.com> Date: Wed Jun 14 13:38:49 2023 +0100 pythongh-105566: Deprecate unusual ways of creating `typing.NamedTuple` classes (python#105609) Deprecate creating a typing.NamedTuple class using keyword arguments to denote the fields (`NT = NamedTuple("NT", x=int, y=str)`). This will be disallowed in Python 3.15. Use the class-based syntax or the functional syntax instead. Two methods of creating `NamedTuple` classes with 0 fields using the functional syntax are also deprecated, and will be disallowed in Python 3.15: `NT = NamedTuple("NT")` and `NT = NamedTuple("NT", None)`. To create a `NamedTuple` class with 0 fields, either use `class NT(NamedTuple): pass` or `NT = NamedTuple("NT", [])`. commit fc8037d Author: Jelle Zijlstra <jelle.zijlstra@gmail.com> Date: Wed Jun 14 05:35:06 2023 -0700 pythongh-104873: Add typing.get_protocol_members and typing.is_protocol (python#104878) Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> commit ba516e7 Author: Kirill Podoprigora <kirill.bast9@mail.ru> Date: Wed Jun 14 15:17:12 2023 +0300 pythongh-102541: Hide traceback in help prompt (pythongh-102614) commit 0316063 Author: Victor Stinner <vstinner@python.org> Date: Wed Jun 14 12:30:22 2023 +0200 gdb libpython.py: Remove compatibility code (python#105739) Remove compatibility code for Python 2 and early Python 3 versions. * Remove os_fsencode() reimplementation: use os.fsencode() directly. os.fsencode() was added to Python 3.2. * Remove references to Python 2 and "Python 3": just say "Python". * Remove outdated u'' string format: use '' instead. commit e5d45b7 Author: Nikita Sobolev <mail@sobolevn.me> Date: Wed Jun 14 13:29:16 2023 +0300 pythongh-105745: Fix open method of webbrowser.Konqueror (python#105746) commit 67f69db Author: Nikita Sobolev <mail@sobolevn.me> Date: Wed Jun 14 13:26:20 2023 +0300 pythongh-105687: Remove deprecated objects from `re` module (python#105688) commit fb655e0 Author: Victor Stinner <vstinner@python.org> Date: Wed Jun 14 12:12:25 2023 +0200 _ctypes callbacks.c uses _Py_COMP_DIAG_IGNORE_DEPR_DECLS (python#105732) Replace #pragma with _Py_COMP_DIAG_PUSH, _Py_COMP_DIAG_IGNORE_DEPR_DECLS and _Py_COMP_DIAG_POP to ease Python maintenance. Also add a comment explaining why callbacks.c ignores a deprecation warning. commit 5cdd5ba Author: Jelle Zijlstra <jelle.zijlstra@gmail.com> Date: Wed Jun 14 00:26:48 2023 -0700 tarfile: Fix positional-only syntax in docs (pythonGH-105770) The syntax used in the current docs (a / before any args) is invalid. I think the right approach is for the arguments to arbitrary filter functions to be treated as positional-only, meaning that users can supply filter functions with any names for the argument. tarfile.py only calls the filter function with positional arguments. commit 6199fe3 Author: Eddie Elizondo <eduardo.elizondorueda@gmail.com> Date: Wed Jun 14 00:33:32 2023 -0400 pythongh-105587: Remove assertion from `_PyStaticObject_CheckRefcnt` (python#105638) commit dab5a3e Author: Barney Gale <barney.gale@gmail.com> Date: Wed Jun 14 05:06:58 2023 +0100 pythonGH-89812: Clean up pathlib tests. (python#104829) Clean up pathlib tests. Merge `PurePathTest` into `_BasePurePathTest`, and `PathTest` into `_BasePathTest`. commit b95de96 Author: Victor Stinner <vstinner@python.org> Date: Wed Jun 14 05:34:11 2023 +0200 pythongh-105751: test_ctypes avoids "from ctypes import *" (python#105768) Using "import *" prevents linters like pyflakes to detect undefined names (usually missing imports). Replace c_voidp with c_void_p. commit 381a1dc Author: Victor Stinner <vstinner@python.org> Date: Wed Jun 14 04:47:01 2023 +0200 pythongh-105751: test_ctypes.test_numbers uses top level imports (python#105762) Moroever, c_ulonglong and c_bool are always available. commit ac7b551 Author: Victor Stinner <vstinner@python.org> Date: Wed Jun 14 04:46:47 2023 +0200 pythongh-105751: test_ctypes gets Windows attrs from ctypes (python#105758) test_ctypes now gets attributes specific to Windows from the ctypes module, rather than relying on "from ctypes import *". Attributes: * ctypes.FormatError * ctypes.WINFUNCTYPE * ctypes.WinError * ctypes.WinDLL * ctypes.windll * ctypes.oledll * ctypes.get_last_error() * ctypes.set_last_error() commit b87d288 Author: Eric Snow <ericsnowcurrently@gmail.com> Date: Tue Jun 13 18:58:23 2023 -0600 pythongh-105699: Use a Thread-Local Variable for PKGCONTEXT (pythongh-105740) This fixes a race during import. The existing _PyRuntimeState.imports.pkgcontext is shared between interpreters, and occasionally this would cause a crash when multiple interpreters were importing extensions modules at the same time. To solve this we add a thread-local variable for the value. We also leave the existing state (and infrequent race) in place for platforms that do not support thread-local variables. commit fcf0647 Author: Eric Snow <ericsnowcurrently@gmail.com> Date: Tue Jun 13 18:39:18 2023 -0600 pythongh-104812: Skip Pending Calls Tests if No Threading (pythongh-105761) This fixes the WASM buildbots. commit b542972 Author: Victor Stinner <vstinner@python.org> Date: Wed Jun 14 02:33:00 2023 +0200 pythongh-105387: Limited C API implements Py_INCREF() as func (python#105388) In the limited C API version 3.12, Py_INCREF() and Py_DECREF() functions are now implemented as opaque function calls to hide implementation details. commit f3266c0 Author: zentarim <33746047+zentarim@users.noreply.github.com> Date: Wed Jun 14 02:45:47 2023 +0300 pythonGH-104554: Add RTSPS support to `urllib/parse.py` (python#104605) * pythonGH-104554: Add RTSPS support to `urllib/parse.py` RTSPS is the permanent scheme defined in https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml alongside RTSP and RTSPU schemes. * 📜🤖 Added by blurb_it. --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> commit 4cefe3c Author: Steve Dower <steve.dower@python.org> Date: Wed Jun 14 00:00:16 2023 +0100 pythongh-105436: Ignore unrelated errors when checking empty env (pythonGH-105742) commit 457a459 Author: Victor Stinner <vstinner@python.org> Date: Wed Jun 14 00:32:12 2023 +0200 pythongh-98040: Fix importbench: use types.ModuleType() (python#105743) Replace removed imp.new_module(name) with types.ModuleType(name). commit 757b402 Author: Eric Snow <ericsnowcurrently@gmail.com> Date: Tue Jun 13 15:02:19 2023 -0600 pythongh-104812: Run Pending Calls in any Thread (pythongh-104813) For a while now, pending calls only run in the main thread (in the main interpreter). This PR changes things to allow any thread run a pending call, unless the pending call was explicitly added for the main thread to run. commit 4e80082 Author: Łukasz Langa <lukasz@langa.pl> Date: Tue Jun 13 20:44:27 2023 +0000 pythongh-102613: Bump recursion limit to fix running test_pathlib under Coverage (python#105744) commit be2779c Author: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Tue Jun 13 21:42:03 2023 +0100 pythongh-105481: add flags to each instr in the opcode metadata table, to replace opcode.hasarg/hasname/hasconst (python#105482) commit 2211454 Author: Victor Stinner <vstinner@python.org> Date: Tue Jun 13 20:16:26 2023 +0200 pythongh-105733: Deprecate ctypes SetPointerType() and ARRAY() (python#105734) commit b97e14a Author: Eric Snow <ericsnowcurrently@gmail.com> Date: Tue Jun 13 11:08:32 2023 -0600 pythongh-105603: Change the PyInterpreterConfig.own gil Field (pythongh-105620) We are changing it to be more flexible that a strict bool can be for possible future expanded used cases. commit abfbab6 Author: Lysandros Nikolaou <lisandrosnik@gmail.com> Date: Tue Jun 13 17:18:11 2023 +0200 pythongh-105718: Fix buffer allocation in tokenizer with readline (python#105728) commit d0f1afd Author: Victor Stinner <vstinner@python.org> Date: Tue Jun 13 13:49:36 2023 +0200 pythongh-105373: Remove PyArg_Parse() deprecation (python#105394) There is no plan to deprecate PyArg_Parse(). The deprecation was added as a comment in the C API documentation in 2007 by commit 85eb8c1. commit ed8217b Author: Lysandros Nikolaou <lisandrosnik@gmail.com> Date: Tue Jun 13 12:39:29 2023 +0200 pythongh-105713: Document that tokenize raises when mixing tabs/spaces (python#105723) * pythongh-105713: Document that tokenize raises when mixing tabs/spaces * Update Doc/whatsnew/3.12.rst Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> commit c3d2d64 Author: Mark Shannon <mark@hotpy.org> Date: Tue Jun 13 10:34:27 2023 +0100 Fix magic number (pythonGH-105722) commit 09ffa69 Author: Mark Shannon <mark@hotpy.org> Date: Tue Jun 13 09:51:05 2023 +0100 pythonGH-105678: Split MAKE_FUNCTION into MAKE_FUNCTION and SET_FUNCTION_ATTRIBUTE (pythonGH-105680) commit 217589d Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Tue Jun 13 10:38:01 2023 +0200 pythongh-105375: Improve error handling in _Unpickler_SetInputStream() (python#105667) Prevent exceptions from possibly being overwritten in case of multiple failures. commit 840d02f Author: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com> Date: Tue Jun 13 11:36:40 2023 +0530 pythonGH-105684: Require `asyncio.Task` implementations to support `set_name` method (python#105685) commit 829ac13 Author: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com> Date: Tue Jun 13 11:11:34 2023 +0530 pythonGH-104787: use bitfields in `_asyncio` (python#104788) commit 8da9d1b Author: Guido van Rossum <guido@python.org> Date: Mon Jun 12 14:55:15 2023 -0700 pythongh-105540: Fix code generator tests (python#105707) This involves expanding PEEK, POKE and JUMPBY macros, and removing super and register tests (those features no longer exist). commit ca3cc4b Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Mon Jun 12 23:35:07 2023 +0200 pythongh-105375: Explicitly initialise all {Pickler,Unpickler}Object fields (python#105686) All fields must be explicitly initialised to prevent manipulation of uninitialised fields in dealloc. Align initialisation order with the layout of the object structs. commit f0fb782 Author: Jay <74105438+weijay0804@users.noreply.github.com> Date: Tue Jun 13 04:29:02 2023 +0800 pythongh-105331: Change `asyncio.sleep` to raise ``ValueError` for nan (python#105641) Co-authored-by: Guido van Rossum <gvanrossum@gmail.com> Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com> commit 9544948 Author: Guido van Rossum <guido@python.org> Date: Mon Jun 12 11:19:04 2023 -0700 Remove support for legacy bytecode instructions (python#105705) (A legacy instruction is of the form `instr(FOOBAR)`, i.e. missing the `(... -- ...)` stack/cache effect annotation.) commit b9e7dc7 Author: Guido van Rossum <guido@python.org> Date: Mon Jun 12 10:47:08 2023 -0700 pythongh-105229: Remove syntactic support for super-instructions (python#105703) It will not be used again. commit 4f7d3b6 Author: Dora203 <66343334+sku2000@users.noreply.github.com> Date: Tue Jun 13 00:14:55 2023 +0800 pythongh-105436: The environment block should end with two null wchar_t values (pythonGH-105495) commit 2b90796 Author: Petr Viktorin <encukou@gmail.com> Date: Mon Jun 12 17:45:49 2023 +0200 pythongh-103968: PyType_FromMetaclass: Allow metaclasses with tp_new=NULL (pythonGH-105386) commit 58f0bda Author: Steve Dower <steve.dower@python.org> Date: Mon Jun 12 16:09:14 2023 +0100 ARM64 clamping bug also exists in MSVC 14.35 (pythonGH-105679) commit a8d69fe Author: Nikita Sobolev <mail@sobolevn.me> Date: Mon Jun 12 11:47:56 2023 +0300 pythongh-105673: Fix uninitialized warning in sysmodule.c (python#105674) In sys_add_xoption(), 'value' may be uninitialized for some error paths. commit 58f5227 Author: Irit Katriel <1055913+iritkatriel@users.noreply.github.com> Date: Sun Jun 11 22:31:59 2023 +0100 pythongh-105481: add pseudo-instructions to the bytecodes DSL (python#105506) commit 20a56d8 Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Sun Jun 11 22:18:46 2023 +0200 pythongh-105375: Harden pyexpat initialisation (python#105606) Add proper error handling to add_errors_module() to prevent exceptions from possibly being overwritten. commit 41cddc2 Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Sun Jun 11 22:02:49 2023 +0200 pythongh-105375: Improve error handling in the sys extension module (python#105611) In _PySys_AddXOptionWithError() and sys_add_xoption(), bail on first error to prevent exceptions from possibly being overwritten. commit e8998e4 Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Sun Jun 11 21:46:19 2023 +0200 pythongh-105375: Improve error handling in _ctypes (python#105593) Prevent repeated PyLong_FromVoidPtr() from possibly overwriting the current exception. commit 555be81 Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Sun Jun 11 21:29:19 2023 +0200 pythongh-105375: Improve error handling in PyUnicode_BuildEncodingMap() (python#105491) Bail on first error to prevent exceptions from possibly being overwritten. commit 567d6ae Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Sun Jun 11 21:23:28 2023 +0200 pythongh-105375: Improve PyErr_WarnExplicit() error handling (python#105610) Bail on first error to prevent exceptions from possibly being overwritten. commit 3f7c081 Author: Samet YASLAN <sametyaslan@gmail.com> Date: Sun Jun 11 20:51:21 2023 +0200 bpo-44185: Added close() to mock_open __exit__ (python#26902) commit 18d16e9 Author: Tomas R <tomas.roun8@gmail.com> Date: Sun Jun 11 17:50:34 2023 +0200 pythongh-102676: Add more convenience properties to `dis.Instruction` (python#103969) Adds start_offset, cache_offset, end_offset, baseopcode, baseopname, jump_target and oparg to dis.Instruction. Also slightly improves the disassembly output by allowing opnames to overflow into the space reserved for opargs. commit 845e593 Author: litlighilit <97860435+litlighilit@users.noreply.github.com> Date: Sun Jun 11 22:06:42 2023 +0800 Fix typo in configparser module docstring (python#105652) "zc.buildbot" -> "zc.buildout" commit d4fa529 Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Sun Jun 11 12:20:43 2023 +0200 pythongh-105375: Improve error handling in the builtins extension module (python#105585) commit c932f72 Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Sun Jun 11 12:06:06 2023 +0200 pythongh-105375: Improve _decimal error handling (python#105605) Fix a bug where an exception could end up being overwritten. commit 16d4968 Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Sun Jun 11 12:03:09 2023 +0200 pythongh-105375: Harden _datetime initialisation (python#105604) Improve error handling so init bails on the first exception. commit 35cff54 Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Sun Jun 11 11:58:08 2023 +0200 pythongh-105375: Improve array.array exception handling (python#105594) Fix a bug where 'tp_richcompare' could end up overwriting an exception. commit 01f4230 Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Sun Jun 11 11:56:32 2023 +0200 pythongh-105375: Harden _ssl initialisation (python#105599) Add proper error handling to prevent reference leaks and overwritten exceptions. commit cc87948 Author: Hugo van Kemenade <hugovk@users.noreply.github.com> Date: Sun Jun 11 12:17:35 2023 +0300 pythongh-80480: Emit DeprecationWarning for array's 'u' type code (python#95760) commit 3a314f7 Author: Gregory P. Smith <greg@krypto.org> Date: Sat Jun 10 12:09:20 2023 -0700 pythongh-99108: Mention HACL\* in the hashlib docs. (python#105634) commit 0d1d6ab Author: Gregory P. Smith <greg@krypto.org> Date: Sat Jun 10 11:49:06 2023 -0700 Cleanup and clarify our hashlib docs. (python#105624) Clarify and improve our hashlib docs. Now with 50% less mess! commit d636d7d Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Fri Jun 9 23:53:33 2023 +0200 pythongh-105375: Harden error handling in `_testcapi/heaptype.c` (python#105608) Bail on first error in heapctypesubclasswithfinalizer_finalize() commit 33c92c4 Author: Nikita Sobolev <mail@sobolevn.me> Date: Sat Jun 10 00:48:54 2023 +0300 pythongh-105375: Improve error handling in `zoneinfo` module (python#105586) Fix bugs where exceptions could end up being overwritten because of deferred error handling. Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com> commit 91441bf Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Fri Jun 9 23:14:02 2023 +0200 Docs: fix formatting in 2023-06-09-12-59-18 NEWS item (python#105607) commit b047fa5 Author: Pablo Galindo Salgado <Pablogsal@gmail.com> Date: Fri Jun 9 21:39:01 2023 +0100 pythongh-105549: Tokenize separately NUMBER and NAME tokens and allow 0-prefixed literals (python#105555) commit 00b599a Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Fri Jun 9 22:35:03 2023 +0200 pythongh-105375: Improve error handling in _elementtree (python#105591) Fix bugs where exceptions could end up being overwritten. commit f668f73 Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Fri Jun 9 22:07:47 2023 +0200 pythongh-105375: Improve posix error handling (python#105592) Fix a bug where an IndexError could end up being overwritten. commit eede1d2 Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Fri Jun 9 21:57:25 2023 +0200 pythongh-105375: Improve errnomodule error handling (python#105590) Bail immediately if an exception is set, to prevent exceptions from being overwritten. commit 89aac6f Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Fri Jun 9 19:09:53 2023 +0200 pythongh-105375: Improve _pickle error handling (python#105475) Error handling was deferred in some cases, which could potentially lead to exceptions being overwritten. commit 6c832dd Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Fri Jun 9 18:55:53 2023 +0200 pythongh-105375: Improve error handling in compiler_enter_scope() (python#105494) commit d7f46bc Author: Pablo Galindo Salgado <Pablogsal@gmail.com> Date: Fri Jun 9 17:01:26 2023 +0100 pythongh-105564: Don't include artificial newlines in the line attribute of tokens (python#105565) commit 1dd267a Author: Pablo Galindo Salgado <Pablogsal@gmail.com> Date: Fri Jun 9 16:59:37 2023 +0100 Clarify the supported cases in the tokenize module (python#105569) commit 59f009e Author: Ethan Furman <ethan@stoneleaf.us> Date: Fri Jun 9 08:56:05 2023 -0700 pythongh-105497: [Enum] Fix Flag inversion when alias/mask members exist. (pythonGH-105542) When inverting a Flag member (or boundary STRICT), only consider other canonical flags; when inverting an IntFlag member (or boundary KEEP), also consider aliases. commit 8e75592 Author: Alex Waygood <Alex.Waygood@Gmail.com> Date: Fri Jun 9 16:08:57 2023 +0100 Miscellaneous improvements to the typing docs (python#105529) Mostly, these are changes so that we use shorter sentences and shorter paragraphs. In particular, I've tried to make the first sentence introducing each object in the typing API short and declarative. commit b8fa7bd Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Fri Jun 9 15:36:59 2023 +0200 pythongh-105557: Remove duplicate sqlite3 test method (python#105558) test_func_return_too_large_int() was defined twice. Keep only the redefined method, as that also checks the tracebacks. commit 9bf8d82 Author: Thomas Grainger <tagrain@gmail.com> Date: Fri Jun 9 14:29:09 2023 +0100 pythongh-94924: support `inspect.iscoroutinefunction` in `create_autospec(async_def)` (python#94962) * support inspect.iscoroutinefunction in create_autospec(async_def) * test create_autospec with inspect.iscoroutine and inspect.iscoroutinefunction * test when create_autospec functions check their signature commit 0f885ff Author: Victor Stinner <vstinner@python.org> Date: Fri Jun 9 14:50:31 2023 +0200 pythongh-105407: Remove unused imports (python#105554)

Yhg1s · 2023-07-11T10:15:31Z

Is there anything left to do here for 3.12?

encukou · 2023-07-11T10:58:57Z

No, short of reverting the warning.

…sses (GH-105698)

…etaclasses (pythonGH-105698) (cherry picked from commit af5cf1e) Co-authored-by: Petr Viktorin <encukou@gmail.com>

…metaclasses (GH-105698) (GH-106619) gh-103968: What's New: Add porting hints for PyType_From with metaclasses (GH-105698) (cherry picked from commit af5cf1e) Co-authored-by: Petr Viktorin <encukou@gmail.com>

cdce8p · 2023-08-06T10:21:11Z

Just saw the warning while testing 3.12.0b4. Is stack_level=1 correct for the DeprecationWarning
This is my test output

<frozen importlib._bootstrap>:400
<frozen importlib._bootstrap>:400
  <frozen importlib._bootstrap>:400: DeprecationWarning: Using PyType_Spec with metaclasses that have custom tp_new is deprecated and will no longer be allowed in Python 3.14.

After reading the discussion here, it could probably be Protobuf but no way to tell for the warning alone.

Happy to open a new issue if this isn't the right place.

gpshead · 2023-08-07T01:49:21Z

After reading the discussion here, it could probably be Protobuf but no way to tell for the warning alone.

Happy to open a new issue if this isn't the right place.

@cdce8p - If you can reproduce this on 3.12.0rc1 please open a new issue with reproducer details (and ideally a link to the protobuf code in question).

In Python 3.12, it is no longer possible for C types to inherit from abc.ABC since it has a custom tp_new. So we must find an alternate solution to implement collections.abc protocols. This attempts to do that by still using the methods from collection.abc but mixing in the methods manually and registering the type. Idea from: python/cpython#103968 (comment)

encukou added type-bug An unexpected behavior, bug, or error topic-C-API labels Apr 28, 2023

encukou self-assigned this Apr 28, 2023

encukou added a commit to encukou/cpython that referenced this issue Apr 28, 2023

pythongh-103968: Deprecate creating heap types whose metaclass has cu…

2ca04bb

…stom tp_new. (That's a mouthful of an edge case!)

bedevere-bot mentioned this issue Apr 28, 2023

gh-103968: Deprecate creating heap types whose metaclass has custom tp_new. #103972

Merged

encukou added a commit that referenced this issue May 3, 2023

gh-103968: Deprecate creating heap types whose metaclass has custom t…

524a7f7

…p_new. (GH-103972) (That's a mouthful of an edge case!) Co-authored-by: Barney Gale <barney.gale@gmail.com>

encukou closed this as completed May 3, 2023

erlend-aasland mentioned this issue May 3, 2023

gh-104146: Remove unused 'second_pass_replacements' from clinic.py #104147

Merged

gpshead reopened this Jun 2, 2023

gpshead added 3.12 bugs and security fixes docs Documentation in the Doc dir labels Jun 2, 2023

gpshead added the deferred-blocker label Jun 2, 2023

This was referenced Jun 5, 2023

PyType_FromSpec should take metaclass as an argument #60074

Closed

Compiled Python extension does not work with Python 3.12 protocolbuffers/protobuf#12186

Open

bedevere-bot mentioned this issue Jun 6, 2023

gh-103968: PyType_FromMetaclass: Allow metaclasses with tp_new=NULL #105386

Merged

encukou added a commit that referenced this issue Jun 12, 2023

gh-103968: PyType_FromMetaclass: Allow metaclasses with tp_new=NULL (G…

2b90796

…H-105386)

miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 12, 2023

pythongh-103968: PyType_FromMetaclass: Allow metaclasses with tp_new=…

4d30f7d

…NULL (pythonGH-105386) (cherry picked from commit 2b90796) Co-authored-by: Petr Viktorin <encukou@gmail.com>

This was referenced Jun 12, 2023

[3.12] gh-103968: PyType_FromMetaclass: Allow metaclasses with tp_new=NULL (GH-105386) #105697

Merged

gh-103968: What's New: Add porting hints for PyType_From with metaclasses #105698

Merged

encukou closed this as completed Jul 11, 2023

encukou added a commit that referenced this issue Jul 11, 2023

gh-103968: What's New: Add porting hints for PyType_From with metacla…

af5cf1e

…sses (GH-105698)

bedevere-bot mentioned this issue Jul 11, 2023

[3.12] gh-103968: What's New: Add porting hints for PyType_From with metaclasses (GH-105698) #106619

Merged

cdce8p mentioned this issue Aug 9, 2023

Improve message of PyType_Spec deprecation warning with location #107810

Closed

Jayaseelan-James mentioned this issue Nov 9, 2023

MeasurementLink Python tests raise DeprecationWarning with Python 3.12 ni/measurementlink-python#428

Closed

neonene mentioned this issue Jan 1, 2024

gh-103092: Port some _ctypes data types to heap types #113630

Open

PyType_FromSpec refuses to create classes with tp_new #103968

PyType_FromSpec refuses to create classes with tp_new #103968

encukou commented Apr 28, 2023 •

edited by bedevere-bot

encukou commented May 3, 2023

gpshead commented Jun 2, 2023

gpshead commented Jun 2, 2023

gpshead commented Jun 2, 2023

encukou commented Jun 5, 2023

encukou commented Jun 5, 2023 •

edited

encukou commented Jun 6, 2023

haberman commented Jun 13, 2023 •

edited by gpshead

Yhg1s commented Jul 11, 2023

encukou commented Jul 11, 2023

cdce8p commented Aug 6, 2023

gpshead commented Aug 7, 2023

PyType_FromSpec refuses to create classes with tp_new #103968

PyType_FromSpec refuses to create classes with tp_new #103968

Comments

encukou commented Apr 28, 2023 • edited by bedevere-bot

Linked PRs

encukou commented May 3, 2023

gpshead commented Jun 2, 2023

gpshead commented Jun 2, 2023

gpshead commented Jun 2, 2023

encukou commented Jun 5, 2023

encukou commented Jun 5, 2023 • edited

encukou commented Jun 6, 2023

haberman commented Jun 13, 2023 • edited by gpshead

Yhg1s commented Jul 11, 2023

encukou commented Jul 11, 2023

cdce8p commented Aug 6, 2023

gpshead commented Aug 7, 2023

encukou commented Apr 28, 2023 •

edited by bedevere-bot

encukou commented Jun 5, 2023 •

edited

haberman commented Jun 13, 2023 •

edited by gpshead