New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bpo-43224: Implement PEP 646 changes to typing.py #31021
base: main
Are you sure you want to change the base?
Conversation
Lib/typing.py
Outdated
return self._name | ||
|
||
|
||
class UnpackedTypeVarTuple(_Final, _Immutable, _root=True): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(Copying and pasting from the previous PR; haven't thought about Jelle's suggestion yet)
@pradeep90 In the previous review, you mentioned you'd prefer to avoid this middleman class, instead for example making it so that starring a TypeVarTuple results in an instance of Unpack
.
That example solution seems suboptimal to me for two reasons:
- In a few places in the rest of typing.py, we want to check whether an object is an unpacked TypeVarTuple, and
instance(obj, UnpackedTypeVarTuple)
seems more readable to me thanisinstance(obj, Unpacked)
. - It seems potentially confusing to have
Unpack
both be an operator and an object.
Another option would be something like:
class TypeVarTuple:
def __init__(self, name, _starred=False, _id=None):
self.name = name
self._starred = starred
self._id = id
def __iter__(self):
yield TypeVarTuple(self.name, _starred=True, _id=id(self))
def __eq__(self, other):
if self._id is not None and other._id is not None:
return self._id == other._id
else:
return object.__eq__(self, other) # Not sure whether this is right, but you get the gist
def __repr__(self):
return ('*' if self._starred else '') + self.name
I lean away from this solution too because I feel like manually managing IDs could lead to subtle bugs, and isinstance(obj, UnpackedTypeVarTuple)
is still simpler than isinstance(obj, TypeVarTuple) and obj._starred
, but WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Jelle's argument that Unpack
should work with arbitrary types at runtime convinced me that your approach is the way forward :) Have re-implemented Unpacked
with _SpecialForm
, as Jelle suggested. How does it look to you now?
@@ -818,6 +869,117 @@ def __init__(self, name, *constraints, bound=None, | |||
self.__module__ = def_mod | |||
|
|||
|
|||
class TypeVarTuple(_Final, _Immutable, _root=True): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pradeep90 You mentioned in the previous review you'd prefer to have this inherit from _TypeVarLike
. I'm not sure I understood your reasoning - could you explain what you meant by "The flags are things that the typechecker has to deal with"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I meant there seems to be no reason why TypeVarTuple
can't inherit from _BoundVarianceMixin
(previously TypeVarLike
). Someone can use isinstance(x, TypeVarLike)
instead of hardcoding isinstance(x, (TypeVar, ParamSpec))
and thus missing out TypeVarTuple
.
The downside is that TypeVarTuple("Ts", bound=...)
or covariant=True
will be valid at runtime. That's fine. We want the runtime to be as lenient as possible so that we can specify the behavior of bound
in the future. The counter-argument is that we might want a different name such as item_bound
for TypeVarTuple.
I'm ok with either.
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
(Reviewed three-fourths of it and got tired after a couple of hours. Will review the rest after the changes.)
Overall, I think we'd want to try other things beyond Unpack[Ts]
: Unpack[tuple[int, ...]]
, Unpack[tuple[int, str]]
, and Unpack[tuple[int, Unpack[Ts], str]]
. I've tried to point to cases inline.
I'd assumed all these comments had been posted weeks ago - I'd missed the part where you actually have to click the 'Submit code review' button to post them! Uff.
Ah, that clears up a mystery :) I was wondering why many comments were unaddressed.
Do we need to validate *args: <foo>
somewhere? For example, *args: Ts
vs *args: Unpack[Ts]
.
What if someone tries Union[*Ts]
or Union[Ts]
? Same for other special forms that call _type_check
:
ClassVar
Optional
Final
(i.e.,x: Final[Ts]
)TypeGuard
- What about
Concatenate[Ts, P]
andConcatenate[*Ts, P]
?
n00b question: how do I run the Python tests in my local clone of this repository? Wanted to experiment.
B = A[Unpack[Ts], int] | ||
self.assertTrue(repr(B[()]).endswith('A[int]')) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you try substituting with B[Unpack[Ts]]
and B[Unpack[tuple[str, ...]]]
def test_typevar_substitution(self): | ||
T1 = TypeVar('T1') | ||
T2 = TypeVar('T2') | ||
Ts = TypeVarTuple('Ts') | ||
|
||
# Too few parameters | ||
self.assertRaises( | ||
TypeError, | ||
_determine_typevar_substitution, | ||
typevars=(T1,), params=(), | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A for-loop over a list of arguments might make this more succinct and easier to follow.
error_cases = [
((T1, T2), (int,)),
...
]
for typevars, params in error_cases:
self.assertRaises(TypeError, ...)
# Correct number of parameters, TypeVars only | ||
self.assertEqual( | ||
_determine_typevar_substitution((T1,), (int,)), | ||
{T1: int}, | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same for these tests - we could have a list of inputs and expected outputs.
Ts2 = TypeVarTuple('Ts2') | ||
self.assertNotEqual(C[Unpack[Ts1]], C[Unpack[Ts2]]) | ||
|
||
def test_typevar_substitution(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to test for the type arguments (called "params" here) having Unpack[...]
? The cases are Unpack[tuple[int, str]]
, Unpack[tuple[int, ...]]
, Unpack[Ts]
, and combinations.
Things get a bit complicated with tuple[int, ...]
. For example, when the expected types are (T1, T2)
, we should have T1=int, T2=int
. When the expected types are (T1, *Ts, T2)
and the type argument is (int, Unpack[tuple[str, ...]])
, we should have T1=int, Ts=tuple[str, ...], T2=str
.
It would be great to avoid implementing all this logic, but it looks like GenericAlias
demands this behavior at runtime so that people can do MyAlias[<...>] == <the instantiated type>
, so we may not have a choice.
b = Callable[[int, Unpack[Ts]], None] | ||
self.assertEqual(b.__args__, (int, Unpack[Ts], type(None))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks a bit dodgy.
Edit: Didn't mean it was wrong. Just looks weird to see a combined tuple of the parameter and return types because I'm not familiar with the format behind __args__
for Callable
`Dict[Unpack[Tuple[int, str]]` is equivalent to `Dict[int, str]`. | ||
In newer versions of Python, this is implemented using the `*` operator. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wait, Dict[*<...>]
isn't a valid type, right? Otherwise, someone may write Dict[*Ts]
and that may fail (during type-checking).
Even if Dict[*tuple[int, str]]
ends up being correct at runtime, I think we'd want to discourage this. Could you use an Array
instead?
"""Mixin giving __init__ bound and variance arguments.""" | ||
def __init__(self, bound, covariant, contravariant): | ||
"""Used to setup TypeVars and ParamSpec's bound, covariant and | ||
contravariant attributes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you describe the three possible uses (TypeVar, ParamSpec, and TypeVarTuple) in the class docstring and the __init__
docstring? The name BoundVarianceMixin
and the docstring feel a bit cryptic.
(This is assuming we use this mixin for TypeVarTuple
- discussed in comment below.)
@@ -177,7 +179,8 @@ def _type_check(arg, msg, is_argument=True, module=None, *, allow_special_forms= | |||
return arg | |||
if isinstance(arg, _SpecialForm) or arg in (Generic, Protocol): | |||
raise TypeError(f"Plain {arg} is not valid as type argument") | |||
if isinstance(arg, (type, TypeVar, ForwardRef, types.UnionType, ParamSpec)): | |||
if isinstance(arg, (type, TypeVar, TypeVarTuple, ForwardRef, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
_type_check
seems to be used in two kinds of places:
_type_check
is used on the type arguments forSpecialGenericAlias
.- Apart from the above,
_type_check
is used on the type arguments of a bunch of special forms likeOptional
,Final
,Union
, etc.
IIUC, we care about variadic tuples in (1) but not (2).
For (1), if we have MyAlias = Array[*Ts]
, we want to be able to instantiate the alias as MyAlias[int, *Ts2, str]
. That suggests what we want above is Unpack
.
For (2), with the current code, we'd accept x: Final[Ts]
or Optional[Ts]
, which again isn't valid. Note that even Optional[*Ts]
is wrong. Likewise, BoundVarianceMixin
uses _type_check
on the bound. This means we'd end up accepting T = TypeVar("T", bound=Ts)
.
(I haven't tested the above, so I could be way off.)
I see two ways ahead:
(a) replace TypeVarTuple
with Unpack
above and just live with the invalid uses like Optional[*Ts]
.
(b) make SpecialGenericAlias
check for Unpack
on top of _type_check
so that it can accept MyAlias[int, *Ts, str]
but things like Optional[*Ts]
would be ruled out.
What do you think? If that sounds right, could you add tests for these cases?
def _check_generic(cls, parameters, elen): | ||
"""Check correct count for parameters of a generic cls (internal helper). | ||
This gives a nice error message in case of count mismatch. | ||
def _check_type_parameter_count(cls, type_params): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we leave out the runtime arity checking for TypeVarTuple
? I'm in favor of keeping the runtime stuff minimal and letting the type checkers handle cases as needed. Otherwise, we're bound to miss some case and cause problems (or make it harder to extend this in the future).
Did you have a use case in mind for the runtime errors?
@@ -818,6 +869,117 @@ def __init__(self, name, *constraints, bound=None, | |||
self.__module__ = def_mod | |||
|
|||
|
|||
class TypeVarTuple(_Final, _Immutable, _root=True): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I meant there seems to be no reason why TypeVarTuple
can't inherit from _BoundVarianceMixin
(previously TypeVarLike
). Someone can use isinstance(x, TypeVarLike)
instead of hardcoding isinstance(x, (TypeVar, ParamSpec))
and thus missing out TypeVarTuple
.
The downside is that TypeVarTuple("Ts", bound=...)
or covariant=True
will be valid at runtime. That's fine. We want the runtime to be as lenient as possible so that we can specify the behavior of bound
in the future. The counter-argument is that we might want a different name such as item_bound
for TypeVarTuple.
I'm ok with either.
First compile Python, as explained in "Quick reference" at https://devguide.python.org/. To run just the typing tests, the easiest way is just |
Thanks, Jelle. Should have RTFM'ed but was being lazy :| @mrahtz Some tests fail locally (and on CI, apparently). |
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
@gvanrossum @JelleZijlstra @pradeep90 There's a high-level design point I've just realised we should discuss. In the PEP itself, we stated that (@gvanrossum, in case you haven't followed the chain of comments on the PR so far, there were two main reasons to try doing it this way: 1. It means we can avoid the need for a middleman classes This raises two questions:
|
Thanks for bringing this up! Here are my takes:
|
Exactly what Jelle says. |
def test_variadic_class_accepts_arbitrary_number_of_parameters(self): | ||
Ts = TypeVarTuple('Ts') | ||
class C(Generic[Unpack[Ts]]): pass | ||
C[()] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Fidget-Spinner commented on the parallel typing-extensions tests (python/typing#963) that it's not really clear what these are testing. I agree, so over there I'm planning to make tests like this also check that __args__
is correct.
|
||
class A(Generic[T1, T2, Unpack[Ts]]): pass | ||
A[int, str] | ||
A[int, str, float] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be good to also test A[int, str, float, bool]
, where the Ts maps to multiple types.
These are the parts of the old PR (#30398) relevant to
typing.py
.Note that we haven't yet merged the grammar PR (#31018), so for the time being all the tests just use
Unpack
. We can add tests using the actual star operator in a future PR.https://bugs.python.org/issue43224
The text was updated successfully, but these errors were encountered: