bpo-31861: Add aiter and anext to builtins #23847
Conversation
...on top of latest master. Also drop now-removed `loop` kwarg from asyncio.sleep call. Ref: https://bugs.python.org/issue42392
Hello, and thanks for your contribution! I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA). Recognized GitHub usernameWe couldn't find a bugs.python.org (b.p.o) account corresponding to the following GitHub usernames: This might be simply due to a missing "GitHub Name" entry in one's b.p.o account settings. This is necessary for legal reasons before we can look at this contribution. Please follow the steps outlined in the CPython devguide to rectify this issue. You can check yourself to see if the CLA has been received. Thanks again for the contribution, we look forward to reviewing it! |
Resolved the conflicts - hopefully we can get this reviewed before a new conflict pops up! |
* master: (129 commits) bpo-43452: Micro-optimizations to PyType_Lookup (GH-24804) bpo-43517: Fix false positive in detection of circular imports (#24895) bpo-43494: Make some minor changes to lnotab notes (GH-24861) Mention that code.co_lnotab is deprecated in what's new for 3.10. (#24902) bpo-43244: Remove symtable.h header file (GH-24910) bpo-43466: Add --with-openssl-rpath configure option (GH-24820) Fix a typo in c-analyzer (GH-24468) bpo-41561: Add workaround for Ubuntu's custom security level (GH-24915) bpo-43521: Allow ast.unparse with empty sets and NaN (GH-24897) bpo-43244: Remove the PyAST_Validate() function (GH-24911) bpo-43541: Fix PyEval_EvalCodeEx() regression (GH-24918) bpo-43244: Fix test_peg_generators on Windows (GH-24913) bpo-39342: Expose X509_V_FLAG_ALLOW_PROXY_CERTS in ssl module (GH-18011) bpo-43244: Fix test_peg_generator for PyAST_Validate() (GH-24912) bpo-42128: Add 'missing :' syntax error message to match statements (GH-24733) bpo-43244: Add pycore_ast.h header file (GH-24908) bpo-43244: Rename pycore_ast.h to pycore_ast_state.h (GH-24907) Remove unnecessary imports in the grammar parser (GH-24904) bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters (GH-24843) Add PEP 626 to what's new in 3.10. (#24892) ...
Merged in latest master and fixed some minor nits. All checks have passed against the latest revision. UPDATE (ICYMI): There is now ongoing discussion in this recent python-dev thread about whether to merge this PR (favored by @1st1, and perhaps others who've |
The basic code is solid. Some questions and suggestions about how much needs to be public and documented. |
Return an :term:`asynchronous iterator`. This is the async variant | ||
of the :func:`iter` builtin, and behaves similarly. |
gvanrossum
Mar 22, 2021
Member
Frankly, this doesn't tell me much. The description doesn't even state that aiter(x)
is equivalent to x.__aiter__()
, which to me is the key point. Certainly it shouldn't start by stating the type of what it returns; it should describe how the return value relates to the input. (Compare the entry for abs(x) above, "Return the absolute value of a number." This clearly references the input and what the function does to that value.)
Also, state explicitly that aiter(aiter(x))
is the same as aiter(x)
(IOW that aiter(x)
itself has an __aiter__()
method that returns self
.)
jab
Mar 22, 2021
Author
Contributor
Incorporated in the latest revision, but please let me know if it needs further refinement. Thanks for the great feedback!
/* Takes an AsyncIterable object and returns an AsyncIterator for it. | ||
This is typically a new iterator but if the argument is an AsyncIterator, | ||
this returns itself. */ | ||
PyAPI_FUNC(PyObject *) PyObject_GetAiter(PyObject *); |
1st1
Mar 23, 2021
Member
I don't mind adding this function -- while somewhat trivial, it's something that projects like Cython (and potentially our own modules like _asynciomodule.c
) have to reimplement.
@@ -7,6 +7,8 @@ extern "C" { | |||
|
|||
PyAPI_DATA(PyTypeObject) PySeqIter_Type; | |||
PyAPI_DATA(PyTypeObject) PyCallIter_Type; | |||
PyAPI_DATA(PyTypeObject) PyAsyncCallAwaitable_Type; |
gvanrossum
Mar 22, 2021
Member
Maybe these types and the function below should remain CPython implementation details? Just because they're returned by builtins doesn't mean all the implementation types need to be in the C-level API.
Almost there! |
Equivalent to calling ``x.__aiter__()``. | ||
|
||
*async_iterable* must be an :term:`asynchronous iterable`, | ||
and :func:`aiter` returns an asynchronous iterator for it. | ||
``aiter(aiter(x))`` is the same as ``aiter(x)``. | ||
(``aiter(x)`` itself has an ``__aiter__()`` method that returns ``self``.) | ||
|
||
Unlike the :func:`iter` builtin, :func:`aiter` has no 2-argument variant. | ||
Often, this variant can be replaced with assignment expressions:: | ||
Formally, given an :term:`asynchronous iterable`, | ||
return an :term:`asynchronous iterator`. |
gvanrossum
Mar 22, 2021
Member
Much better! Here is how I would rearrange this:
Return an :term:`asynchronous iterator`
for an :term:`asynchronous iterable`.
Equivalent to calling ``x.__aiter__()``.
``aiter(x)`` itself has an ``__aiter__()`` method that returns ``x``,
so ``aiter(aiter(x))`` is the same as ``aiter(x)``.
jab
Mar 22, 2021
Author
Contributor
Incorporated in the latest revision, thanks!
(((
Come to think of it, iter
also has the iter(iter(x)) == iter(x)
property, which is not currently mentioned in the iter
docs. The iter
docs have a lot more work to do though, to cover the 1- and 2-arg variants (and they're already doing this very well).
And now that I'm looking at those again, I notice the only code example there is for the 2-arg variant:
from functools import partial
with open('mydata.db', 'rb') as f:
for block in iter(partial(f.read, 64), b''):
process_block(block)
If this pattern is now obsoleted by assignment expressions[1], is it worth updating the iter
docs to (1) mention the iter(iter(x)) == iter(x)
property, and (2) remove the obsoleted example code? If so, happy to submit a separate PR for that.
[1] as in the following:
with open('mydata.db', 'rb') as f:
while block := f.read(64):
process_block(block)
)))
gvanrossum
Mar 22, 2021
Member
I don't want to personally get into the weeds about the iter() docs, sorry. Something for the docs WG perhaps.
brettcannon
Mar 23, 2021
Member
I don't think it's worth updating the docs to promise iter(iter(x)) == iter(x)
as it isn't a specific benefit to the user beyond logically doing the right thing.
jab
Mar 23, 2021
•
Author
Contributor
I can see that.
@gvanrossum, do you think @brettcannon's rationale applies equally to the aiter
docs too? I did add a test_aiter_idempotent()
for this in 6ee8824 to go along with the promise that the aiter
docs are now making, but can remove that test along with that part of the docs if that's better.
gvanrossum
Mar 23, 2021
Member
I think there's a different bar for updating the iter()
docs than for the initial version of the aiter()
docs. And they don't have to match precisely.
FWIW the main reason the idempotency property is important is because of the implicit [a]iter()
call in a for-loop, since in
for i in x: ...
the for-loop calls iter(x)
, so that in
for i in iter(x): ...
the for-loop ends up calling iter(iter(x))
. It's the same for async for
and aiter()
.
Thanks for your patience. This all looks good to me now! I'll merge after applying my own suggestion. |
@gvanrossum: Please replace |
Congrats! |
Thanks so much, @gvanrossum! We’re honored to have been able to contribute! |
|
I believe test_asyncio is unstable or broken on this platform.
|
I proposed to PR #25266 to rename PyAnextAwaitable_Type to _PyAnextAwaitable_Type, and to initialize the type at Python startup: can someone please have a look? |
Co-authored-by: jab <jab@users.noreply.github.com> Co-authored-by: Daniel Pope <mauve@mauveweb.co.uk> Co-authored-by: Justin Wang <justin39@gmail.com>
This is the C implementation for bpo-31861 requested as an alternative to the Python implementation provided in #8895.
For a more direct translation of this into Python (in case it makes reviewing easier), see jab@ce35092.
Patch by @justin39, @lordmauve, and me.
https://bugs.python.org/issue31861