Feature or enhancement

Proposal:

Problem

As of Python 3.11/PEP 655 set a precident which will mean there's no easy way to reliably extract annotations from a type at runtime. This problem looks to get worse with each new version of Python.

Background

PEP 593 introduced very useful annotations which are intended to be used at runtime. For python 3.9 and 3.10, it was trivially possible to read all annotations produced with typing.get_type_hints(..., include_extras=True). PEP 593 deliberately did not specify how these should be interpreted. However, it did take steps to make them simple to fetch collapsing Annotated[Annotated[int, "foo"], "bar"] to Annotated[int, "foo", "bar"] at runtime.

What broke?

The implementation of PEP 655 in Python 3.11 broke this simple approach by not collapsing Required and NotRequired into the annotations. Annotated[Required[Annotated[...]], "foo"], "bar"] does not collapse at all, and user code is left to go searching.

So it's now much more difficult to go from this:

from typing import TypedDict, Annotated, NamedTuple

class ExtractFrom(NamedTuple):
    name: str
    source_id: int

class UsefulInfo(TypedDict):
    a: Raquired[Annotated[int, ExtractFrom("x", 1)]]
    b: Annotated[Required[int], ExtractFrom("y", 2)]
    c: Annotated[int, ExtractFrom("z", 3)]

to this:

{
    "a": ExtractFrom("x", 1),
    "b": ExtractFrom("y", 2),
    "c": ExtractFrom("z", 3),
}

The workaround for python 3.11 isn't so terribly complex as long as you know to recurse through Required and NotRequired collecting annotations as you go:

# Workaround for python 3.11 and 3.12
from typing import Any, Annotated, Required, NotRequired, get_origin, get_args

def fetch_annotations(type_var) -> tuple[Any, ...]:
    if get_origin(type_var) in {Annotated, Required, NotRequired}:  # 👈 This set is a problem
        child_annotations = fetch_annotations(get_args(type_var)[0])
        if get_origin(type_var) is Annotated:
            return get_args(type_var)[1:] + child_annotations
        return child_annotations
    return ()

It's getting worse...

The set {Annotated, Required, NotRequired} is not a closed set. Eg: PEP 705 will add Readonly to Python 3.13.

This is problematic for two reasons:

1. There's no way to determine if an "origin" is an annotation or something more concrete. There is no is_annotation() returning True for Annotated, Required, and NotRequired but False for everything else. This is specialist knowledge that changes with every version of Python.
2. Implementations must have have specialist knowledge of how to recurse into these types. Recursing with fetch_annotations(get_args(type_var)[0]) works for now, but it's an assumption.

This makes it much harder to maintain a library which consumes annotations.

How does python deal with this internally?

typing.get_type_hints() already knows what is and isn't an annotation in order to correctly process get_type_hints(obj, include_extras=False). In some form this needs to be maintained as new annotations are added.

Presently this is the job of typing._strip_annotations(t) and its code simply contains the list of other annotation types here:

Is is obviously not intended as part of the external interface and could not be used for read annotations, only removing them.

Possible Solutions

Option 1) Make all other annotation types aliases for Annotated

Python could treat Required, NotRequired, ReadOnly, ... as aliases for Annotated with a sentinel value.

Eg Given this code:

Annotated[Required[Annotated[int, "foo"]], "bar"]

Python might process this as:

Annotated[Annotated[Annotated[int, "foo"], Required], "bar"]

... and then collapse this to:

Annotated[int, "foo", Required, "bar"]

This might be the easiest and lowest maintenance approach, however it's a breaking change.

Option 2) Make a helper function to fetch this information

Credit to Jelle Zijlstra for this suggestion.

This might be done by extending and exposing typing._strip_annotations to return the annotations it's stripped. Eg:

Example

strip_annotations returning a 3 tuple:

1. stripped type as existing _strip_annotations()
2. A list of annotations where each one is a 2 tuple:
   1. Annotation type Annotated, Required, ...
   2. A sequence of arguments to the annotation
3. a list containing the same structure for each argument annotation

assert strip_annotations(Required[int]) == (int, [(Required, ())], [])
assert strip_annotations(Annotated[Required[int], "something"] == (int, [(Annotated, ("something",)), (Required, ())], [])

# Unions make it complex. existing _strip_annotations() code already handles them
assert strip_annotations(Annotated[int, "foo"] | Annotated[str, "bar"]) == (
    int | str, 
    [],
    [(int, [(Annotated, ("foo",))], []), (str, [(Annotated, ("bar",))], [])],
)

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/what-is-the-right-way-to-extract-pep-593-annotations/42424/9