Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-43224: Implement PEP 646 changes to typing.py #31021

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

@mrahtz
Copy link

@mrahtz mrahtz commented Jan 30, 2022

These are the parts of the old PR (#30398) relevant to typing.py.

Note that we haven't yet merged the grammar PR (#31018), so for the time being all the tests just use Unpack. We can add tests using the actual star operator in a future PR.

https://bugs.python.org/issue43224

Lib/typing.py Outdated Show resolved Hide resolved
Lib/typing.py Outdated Show resolved Hide resolved
Lib/typing.py Outdated Show resolved Hide resolved
Lib/typing.py Outdated Show resolved Hide resolved
Lib/typing.py Outdated Show resolved Hide resolved
Lib/typing.py Outdated Show resolved Hide resolved
@mrahtz mrahtz marked this pull request as ready for review Jan 30, 2022
Lib/typing.py Outdated
return self._name


class UnpackedTypeVarTuple(_Final, _Immutable, _root=True):
Copy link
Author

@mrahtz mrahtz Jan 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Copying and pasting from the previous PR; haven't thought about Jelle's suggestion yet)

@pradeep90 In the previous review, you mentioned you'd prefer to avoid this middleman class, instead for example making it so that starring a TypeVarTuple results in an instance of Unpack.

That example solution seems suboptimal to me for two reasons:

  1. In a few places in the rest of typing.py, we want to check whether an object is an unpacked TypeVarTuple, and instance(obj, UnpackedTypeVarTuple) seems more readable to me than isinstance(obj, Unpacked).
  2. It seems potentially confusing to have Unpack both be an operator and an object.

Another option would be something like:

class TypeVarTuple:

    def __init__(self, name, _starred=False, _id=None):
        self.name = name
        self._starred = starred
        self._id = id

    def __iter__(self):
        yield TypeVarTuple(self.name, _starred=True, _id=id(self))

    def __eq__(self, other):
        if self._id is not None and other._id is not None:
            return self._id == other._id
        else:
            return object.__eq__(self, other)  # Not sure whether this is right, but you get the gist

    def __repr__(self):
         return ('*' if self._starred else '') + self.name

I lean away from this solution too because I feel like manually managing IDs could lead to subtle bugs, and isinstance(obj, UnpackedTypeVarTuple) is still simpler than isinstance(obj, TypeVarTuple) and obj._starred, but WDYT?

Copy link
Author

@mrahtz mrahtz Jan 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jelle's argument that Unpack should work with arbitrary types at runtime convinced me that your approach is the way forward :) Have re-implemented Unpacked with _SpecialForm, as Jelle suggested. How does it look to you now?

@@ -818,6 +869,117 @@ def __init__(self, name, *constraints, bound=None,
self.__module__ = def_mod


class TypeVarTuple(_Final, _Immutable, _root=True):
Copy link
Author

@mrahtz mrahtz Jan 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pradeep90 You mentioned in the previous review you'd prefer to have this inherit from _TypeVarLike. I'm not sure I understood your reasoning - could you explain what you meant by "The flags are things that the typechecker has to deal with"?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant there seems to be no reason why TypeVarTuple can't inherit from _BoundVarianceMixin (previously TypeVarLike). Someone can use isinstance(x, TypeVarLike) instead of hardcoding isinstance(x, (TypeVar, ParamSpec)) and thus missing out TypeVarTuple.

The downside is that TypeVarTuple("Ts", bound=...) or covariant=True will be valid at runtime. That's fine. We want the runtime to be as lenient as possible so that we can specify the behavior of bound in the future. The counter-argument is that we might want a different name such as item_bound for TypeVarTuple.

I'm ok with either.

mrahtz and others added 5 commits Jan 30, 2022
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Lib/typing.py Outdated Show resolved Hide resolved
Copy link

@pradeep90 pradeep90 left a comment

(Reviewed three-fourths of it and got tired after a couple of hours. Will review the rest after the changes.)

Overall, I think we'd want to try other things beyond Unpack[Ts]: Unpack[tuple[int, ...]], Unpack[tuple[int, str]], and Unpack[tuple[int, Unpack[Ts], str]]. I've tried to point to cases inline.

I'd assumed all these comments had been posted weeks ago - I'd missed the part where you actually have to click the 'Submit code review' button to post them! Uff.

Ah, that clears up a mystery :) I was wondering why many comments were unaddressed.


Do we need to validate *args: <foo> somewhere? For example, *args: Ts vs *args: Unpack[Ts].

What if someone tries Union[*Ts] or Union[Ts]? Same for other special forms that call _type_check:

  • ClassVar
  • Optional
  • Final (i.e., x: Final[Ts])
  • TypeGuard
  • What about Concatenate[Ts, P] and Concatenate[*Ts, P]?

n00b question: how do I run the Python tests in my local clone of this repository? Wanted to experiment.

B = A[Unpack[Ts], int]
self.assertTrue(repr(B[()]).endswith('A[int]'))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you try substituting with B[Unpack[Ts]] and B[Unpack[tuple[str, ...]]]

def test_typevar_substitution(self):
T1 = TypeVar('T1')
T2 = TypeVar('T2')
Ts = TypeVarTuple('Ts')

# Too few parameters
self.assertRaises(
TypeError,
_determine_typevar_substitution,
typevars=(T1,), params=(),
)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A for-loop over a list of arguments might make this more succinct and easier to follow.

error_cases = [
  ((T1, T2), (int,)), 
  ...
]
for typevars, params in error_cases:
  self.assertRaises(TypeError, ...)

# Correct number of parameters, TypeVars only
self.assertEqual(
_determine_typevar_substitution((T1,), (int,)),
{T1: int},
)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for these tests - we could have a list of inputs and expected outputs.

Ts2 = TypeVarTuple('Ts2')
self.assertNotEqual(C[Unpack[Ts1]], C[Unpack[Ts2]])

def test_typevar_substitution(self):

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to test for the type arguments (called "params" here) having Unpack[...]? The cases are Unpack[tuple[int, str]], Unpack[tuple[int, ...]], Unpack[Ts], and combinations.

Things get a bit complicated with tuple[int, ...]. For example, when the expected types are (T1, T2), we should have T1=int, T2=int. When the expected types are (T1, *Ts, T2) and the type argument is (int, Unpack[tuple[str, ...]]), we should have T1=int, Ts=tuple[str, ...], T2=str.

It would be great to avoid implementing all this logic, but it looks like GenericAlias demands this behavior at runtime so that people can do MyAlias[<...>] == <the instantiated type>, so we may not have a choice.

b = Callable[[int, Unpack[Ts]], None]
self.assertEqual(b.__args__, (int, Unpack[Ts], type(None)))
Copy link

@pradeep90 pradeep90 Feb 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks a bit dodgy.

Edit: Didn't mean it was wrong. Just looks weird to see a combined tuple of the parameter and return types because I'm not familiar with the format behind __args__ for Callable

`Dict[Unpack[Tuple[int, str]]` is equivalent to `Dict[int, str]`.
In newer versions of Python, this is implemented using the `*` operator.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, Dict[*<...>] isn't a valid type, right? Otherwise, someone may write Dict[*Ts] and that may fail (during type-checking).

Even if Dict[*tuple[int, str]] ends up being correct at runtime, I think we'd want to discourage this. Could you use an Array instead?

"""Mixin giving __init__ bound and variance arguments."""
def __init__(self, bound, covariant, contravariant):
"""Used to setup TypeVars and ParamSpec's bound, covariant and
contravariant attributes.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you describe the three possible uses (TypeVar, ParamSpec, and TypeVarTuple) in the class docstring and the __init__ docstring? The name BoundVarianceMixin and the docstring feel a bit cryptic.

(This is assuming we use this mixin for TypeVarTuple - discussed in comment below.)

@@ -177,7 +179,8 @@ def _type_check(arg, msg, is_argument=True, module=None, *, allow_special_forms=
return arg
if isinstance(arg, _SpecialForm) or arg in (Generic, Protocol):
raise TypeError(f"Plain {arg} is not valid as type argument")
if isinstance(arg, (type, TypeVar, ForwardRef, types.UnionType, ParamSpec)):
if isinstance(arg, (type, TypeVar, TypeVarTuple, ForwardRef,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_type_check seems to be used in two kinds of places:

  1. _type_check is used on the type arguments for SpecialGenericAlias.
  2. Apart from the above, _type_check is used on the type arguments of a bunch of special forms like Optional, Final, Union, etc.

IIUC, we care about variadic tuples in (1) but not (2).

For (1), if we have MyAlias = Array[*Ts], we want to be able to instantiate the alias as MyAlias[int, *Ts2, str]. That suggests what we want above is Unpack.

For (2), with the current code, we'd accept x: Final[Ts] or Optional[Ts], which again isn't valid. Note that even Optional[*Ts] is wrong. Likewise, BoundVarianceMixin uses _type_check on the bound. This means we'd end up accepting T = TypeVar("T", bound=Ts).

(I haven't tested the above, so I could be way off.)

I see two ways ahead:
(a) replace TypeVarTuple with Unpack above and just live with the invalid uses like Optional[*Ts].
(b) make SpecialGenericAlias check for Unpack on top of _type_check so that it can accept MyAlias[int, *Ts, str] but things like Optional[*Ts] would be ruled out.

What do you think? If that sounds right, could you add tests for these cases?

def _check_generic(cls, parameters, elen):
"""Check correct count for parameters of a generic cls (internal helper).
This gives a nice error message in case of count mismatch.
def _check_type_parameter_count(cls, type_params):

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we leave out the runtime arity checking for TypeVarTuple? I'm in favor of keeping the runtime stuff minimal and letting the type checkers handle cases as needed. Otherwise, we're bound to miss some case and cause problems (or make it harder to extend this in the future).

Did you have a use case in mind for the runtime errors?

@@ -818,6 +869,117 @@ def __init__(self, name, *constraints, bound=None,
self.__module__ = def_mod


class TypeVarTuple(_Final, _Immutable, _root=True):

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant there seems to be no reason why TypeVarTuple can't inherit from _BoundVarianceMixin (previously TypeVarLike). Someone can use isinstance(x, TypeVarLike) instead of hardcoding isinstance(x, (TypeVar, ParamSpec)) and thus missing out TypeVarTuple.

The downside is that TypeVarTuple("Ts", bound=...) or covariant=True will be valid at runtime. That's fine. We want the runtime to be as lenient as possible so that we can specify the behavior of bound in the future. The counter-argument is that we might want a different name such as item_bound for TypeVarTuple.

I'm ok with either.

@JelleZijlstra
Copy link
Member

@JelleZijlstra JelleZijlstra commented Feb 1, 2022

n00b question: how do I run the Python tests in my local clone of this repository? Wanted to experiment.

First compile Python, as explained in "Quick reference" at https://devguide.python.org/. To run just the typing tests, the easiest way is just ./python.exe Lib/test/test_typing.py or similar depending on your OS and CWD.

@pradeep90
Copy link

@pradeep90 pradeep90 commented Feb 1, 2022

n00b question: how do I run the Python tests in my local clone of this repository? Wanted to experiment.

First compile Python, as explained in "Quick reference" at https://devguide.python.org/. To run just the typing tests, the easiest way is just ./python.exe Lib/test/test_typing.py or similar depending on your OS and CWD.

Thanks, Jelle. Should have RTFM'ed but was being lazy :|

@mrahtz Some tests fail locally (and on CI, apparently).

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
@mrahtz
Copy link
Author

@mrahtz mrahtz commented Feb 2, 2022

@gvanrossum @JelleZijlstra @pradeep90 There's a high-level design point I've just realised we should discuss.

In the PEP itself, we stated that Unpack would primarily be for backwards compatibility. But in the current iteration of this PR, we're actually using it as the wrapper for unpacked types in general. For example, TypeVarTuple.__iter__ returns Unpack[self].

(@gvanrossum, in case you haven't followed the chain of comments on the PR so far, there were two main reasons to try doing it this way: 1. It means we can avoid the need for a middleman classes UnpackedTypeVarTuple and StarredTuple like we had before; and 2. It means we can also easily support unpacking of other types at runtime, which might be useful for people wanting to experiment with typing features - e.g. Jelle mentioned he'd like to experiment with Unpack[TypedDict type].)

This raises two questions:

  1. How much does it matter if the way that Unpack is used in typing.py doesn't match the spirit in which the PEP introduced it? (It feels a bit ugly to me, but I wonder whether I'm being too perfectionist. An alternative might be to have two separate things, Unpack as the backwards-compatible version of *, then a separate wrapper Unpacked for eliminating the middleman classes.)
  2. What should we do about unpacking a native tuple? In #31019, this essentially returns a special native 'starred tuple' type. I was about to change it so it instead returned typing.Unpack[self] - but I realise this is against the precedent set by the new union operator, where Foo | Bar also returns a special native type types.UnionType rather than typing.Union[Foo, Bar]. @gvanrossum in particular, do you know whether this was a deliberate decision? Is this a precedent we should respect?

@JelleZijlstra
Copy link
Member

@JelleZijlstra JelleZijlstra commented Feb 2, 2022

Thanks for bringing this up! Here are my takes:

  1. I agree there's some tension here, but I'd rather keep the number of classes lower. Having a separate Unpacked and Unpack doing virtually the same thing would be confusing.
  2. types.UnionType was created specifically because we didn't want built-in syntax to create an instance of a Python-defined class. We should do the same for the PEP 646 syntax.

@gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented Feb 2, 2022

Exactly what Jelle says.

def test_variadic_class_accepts_arbitrary_number_of_parameters(self):
Ts = TypeVarTuple('Ts')
class C(Generic[Unpack[Ts]]): pass
C[()]
Copy link
Member

@JelleZijlstra JelleZijlstra Feb 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Fidget-Spinner commented on the parallel typing-extensions tests (python/typing#963) that it's not really clear what these are testing. I agree, so over there I'm planning to make tests like this also check that __args__ is correct.


class A(Generic[T1, T2, Unpack[Ts]]): pass
A[int, str]
A[int, str, float]
Copy link
Member

@JelleZijlstra JelleZijlstra Feb 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be good to also test A[int, str, float, bool], where the Ts maps to multiple types.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants