Skip to content

gh-112903: Fix MRO resolution for GenericAlias types #112926

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

sobolevn
Copy link
Member

@sobolevn sobolevn commented Dec 10, 2023

There are multiple related issues, including #108403
So, let's be very simple here: we had a specific problem, which is now fixed.

Comment on lines +1141 to +1142
isinstance(b, (_BaseGenericAlias, GenericAlias))
or issubclass(b, Generic)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would happen if we did this instead?

Suggested change
isinstance(b, (_BaseGenericAlias, GenericAlias))
or issubclass(b, Generic)
isinstance(b, _BaseGenericAlias
or (isinstance(b, type) and issubclass(b, Generic))

Previously we've never needed to insert Generic into the __bases__ tuple if a class had a types.GenericAlias instance in its __orig_bases__.

I feel like determining what happens when types.GenericAlias instances are inherited from should really be the job of types.GenericAlias.__mro_entries__; typing._BaseGenericAlias.__mro_entries__ shouldn't be modifying the __bases__ tuple except to remove (and appropriately replace) typing._BaseGenericAlias instances.

Copy link
Member

@AlexWaygood AlexWaygood Dec 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I had a play around, and it looks like I was wrong on several counts:

  1. With my alternative idea, one of your test cases fails (the class-creation machinery complains that it can't find a consistent resolution order for MyMap1), whereas they both pass on Python 3.12. So my idea won't work as-is.
  2. We have actually been inserting Generic into the __bases__ tuple from this method in earlier Python versions if a class had a types.GenericAlias instance in its __orig_bases__, if the __origin__ of that types.GenericAlias instance had Generic in its mro. So this PR would be restoring the behaviour we already had in 3.12 (even if that behaviour looks to me like it was probably accidental...)

Copy link
Member

@carljm carljm Feb 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think @AlexWaygood's suggestion here is on the right track. The contract of the __mro_entries__ method is that it receives the original bases tuple. That means that the bases tuple it receives can legitimately include non-types (which themselves implement __mro_entries__). So it is just buggy for an __mro_entries__ method to assume that it can call issubclass on all bases it sees. Thus I think Alex's fix must be the correct starting point here; if we don't make that fix, it is only a matter of time before someone comes along with some other third-party non-type object in their bases along with a _BaseGenericAlias and hits this same bug.

To clarify this point, here's a valid case not involving types.GenericAlias at all, that also hits this same bug in _BaseGenericAlias.__mro_entries__():

>>> class O:
...     def __mro_entries__(self, bases):
...             return (object,)
...
>>> import typing
>>> class A(typing.List[int], O()): ...
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    class A(typing.List[int], O()): ...
  File "/home/carljm/cpython/Lib/typing.py", line 1388, in __mro_entries__
    return super().__mro_entries__(bases)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/home/carljm/cpython/Lib/typing.py", line 1140, in __mro_entries__
    if isinstance(b, _BaseGenericAlias) or issubclass(b, Generic):
                                           ~~~~~~~~~~^^^^^^^^^^^^
TypeError: issubclass() arg 1 must be a class

I think the fix here should handle this case also. The correct result in this example should be the same result as for class A(typing.List[int], object): ....

Alex's suggestion fixes this case. But clearly there are some follow-on implications we need to resolve as well, to deal with the "can't find consistent resolution order" error in the mapping example; I'm looking into that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AlexWaygood I don't understand your point (2), though. Your suggestion here would also still "insert Generic into the __bases__ tuple from this method if a class had a types.GenericAlias instance in its __orig_bases__." The behavior implemented by your suggestion ends up being "any unrecognized non-type in the MRO does not stop us from adding Generic." That seems like the right behavior to me.

Copy link
Member

@AlexWaygood AlexWaygood Feb 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AlexWaygood I don't understand your point (2), though. Your suggestion here would also still "insert Generic into the __bases__ tuple from this method if a class had a types.GenericAlias instance in its __orig_bases__." The behavior implemented by your suggestion ends up being "any unrecognized non-type in the MRO does not stop us from adding Generic." That seems like the right behavior to me.

Correct; I'm not entirely sure what I was talking about in point (2) on Dec 10 2023 :) I agree with all your points!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be more generic, we can call __mro_entries__() for any non-type in bases and check whether it returns a Generic subclass.

We can also move this logic in the type code that calls __mro_entries__(), to allow returning duplicated bases, but... the code became complicated. And we do not have real use case for this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think this (calling __mro_entries__() on non-type bases and seeing if they return Generic or a subclass) is what we have to do. It is the only way I can see to implement the desired behavior here (add Generic as an MRO entry only if nothing else will add it). The "can't resolve MRO" error Alex observed is indeed because if we don't actually check __mro_entries__() of non-type bases, we will end up adding Generic even when another non-type base is already actually a Generic after resolution of its __mro_entries__().

I agree that in principle the other option could be to modify the contract of __mro_entries__() such that type itself will remove duplicates, but I think this behavior is hard to explain and hard to implement correctly. (Even the definition of "duplicate" is non-trivial, since it is not just equality but involves subclasses , too.)

This fix does mean that __mro_entries__() can be called twice on the same non-type base in a single class creation, but I think this is OK; there is no guarantee about this, and the method should not have side effects.

I will push this fix as a new PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New PR is #115191

Copy link
Member

@serhiy-storchaka serhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@serhiy-storchaka serhiy-storchaka added needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes labels Feb 8, 2024
Copy link
Member

@JelleZijlstra JelleZijlstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this fix is right, because it changes the resulting MRO.

3.12:

>>> import typing
>>> class A(typing.Sized, list[int]): ...
... 
>>> A.__mro__
(<class '__main__.A'>, <class 'collections.abc.Sized'>, <class 'typing.Generic'>, <class 'list'>, <class 'object'>)

This PR:

>>> class A(typing.Sized, list[int]): ...
... 
>>> A.__mro__
(<class '__main__.A'>, <class 'collections.abc.Sized'>, <class 'list'>, <class 'object'>)

As a result, A[int] throws on 3.12 (correctly) and on this PR it is allowed.

@bedevere-app
Copy link

bedevere-app bot commented Feb 8, 2024

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

@sobolevn
Copy link
Member Author

sobolevn commented Feb 9, 2024

#115191 is a better version 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants