Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-31861: Add aiter and anext to builtins #23847

Merged
merged 41 commits into from Mar 23, 2021
Merged

Conversation

@jab
Copy link
Contributor

@jab jab commented Dec 18, 2020

This is the C implementation for bpo-31861 requested as an alternative to the Python implementation provided in #8895.

For a more direct translation of this into Python (in case it makes reviewing easier), see jab@ce35092.

Patch by @justin39, @lordmauve, and me.

https://bugs.python.org/issue31861

@the-knights-who-say-ni
Copy link

@the-knights-who-say-ni the-knights-who-say-ni commented Dec 18, 2020

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

Recognized GitHub username

We couldn't find a bugs.python.org (b.p.o) account corresponding to the following GitHub usernames:

@justin39

This might be simply due to a missing "GitHub Name" entry in one's b.p.o account settings. This is necessary for legal reasons before we can look at this contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

Objects/iterobject.c Outdated Show resolved Hide resolved
Doc/library/functions.rst Outdated Show resolved Hide resolved
@justin39
Copy link
Contributor

@justin39 justin39 commented Feb 26, 2021

Resolved the conflicts - hopefully we can get this reviewed before a new conflict pops up!

jab added 3 commits Mar 20, 2021
* master: (129 commits)
  bpo-43452: Micro-optimizations to PyType_Lookup (GH-24804)
  bpo-43517: Fix false positive in detection of circular imports (#24895)
  bpo-43494: Make some minor changes to lnotab notes (GH-24861)
  Mention that code.co_lnotab is deprecated in what's new for 3.10. (#24902)
  bpo-43244: Remove symtable.h header file (GH-24910)
  bpo-43466: Add --with-openssl-rpath configure option (GH-24820)
  Fix a typo in c-analyzer (GH-24468)
  bpo-41561: Add workaround for Ubuntu's custom security level (GH-24915)
  bpo-43521: Allow ast.unparse with empty sets and NaN (GH-24897)
  bpo-43244: Remove the PyAST_Validate() function (GH-24911)
  bpo-43541: Fix PyEval_EvalCodeEx() regression (GH-24918)
  bpo-43244: Fix test_peg_generators on Windows (GH-24913)
  bpo-39342: Expose X509_V_FLAG_ALLOW_PROXY_CERTS in ssl module (GH-18011)
  bpo-43244: Fix test_peg_generator for PyAST_Validate() (GH-24912)
  bpo-42128: Add 'missing :' syntax error message to match statements (GH-24733)
  bpo-43244: Add pycore_ast.h header file (GH-24908)
  bpo-43244: Rename pycore_ast.h to pycore_ast_state.h (GH-24907)
  Remove unnecessary imports in the grammar parser (GH-24904)
  bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters (GH-24843)
  Add PEP 626 to what's new in 3.10. (#24892)
  ...
@jab
Copy link
Contributor Author

@jab jab commented Mar 20, 2021

Merged in latest master and fixed some minor nits. All checks have passed against the latest revision.

UPDATE (ICYMI): There is now ongoing discussion in this recent python-dev thread about whether to merge this PR (favored by @1st1, and perhaps others who've ❤️'d and participated in this PR), or whether to instead resurrect #8895, my PR from 2018 that added operator.aiter and operator.anext (favored by @ramalho, @pbryan, @gvanrossum, @brettcannon, @terryjreedy). Looking forward to seeing what the core developers decide, and then hopefully being able to merge one of these two PRs before the upcoming 3.10 feature freeze on May 3. Thank you!

Copy link
Member

@gvanrossum gvanrossum left a comment

The basic code is solid. Some questions and suggestions about how much needs to be public and documented.

Doc/glossary.rst Outdated Show resolved Hide resolved
Return an :term:`asynchronous iterator`. This is the async variant
of the :func:`iter` builtin, and behaves similarly.
Comment on lines 66 to 67

This comment has been minimized.

@gvanrossum

gvanrossum Mar 22, 2021
Member

Frankly, this doesn't tell me much. The description doesn't even state that aiter(x) is equivalent to x.__aiter__(), which to me is the key point. Certainly it shouldn't start by stating the type of what it returns; it should describe how the return value relates to the input. (Compare the entry for abs(x) above, "Return the absolute value of a number." This clearly references the input and what the function does to that value.)

Also, state explicitly that aiter(aiter(x)) is the same as aiter(x) (IOW that aiter(x) itself has an __aiter__() method that returns self.)

This comment has been minimized.

@jab

jab Mar 22, 2021
Author Contributor

Incorporated in the latest revision, but please let me know if it needs further refinement. Thanks for the great feedback!

Doc/library/functions.rst Outdated Show resolved Hide resolved
/* Takes an AsyncIterable object and returns an AsyncIterator for it.
This is typically a new iterator but if the argument is an AsyncIterator,
this returns itself. */
PyAPI_FUNC(PyObject *) PyObject_GetAiter(PyObject *);

This comment has been minimized.

@gvanrossum

gvanrossum Mar 22, 2021
Member

Is it necessary to add these to the public C API? Just because PyObject_GetIter() is public I'm not sure that the Aiter variant needs to be. @vstinner tends to push back on adding new things to the C API. @1st1 what do you think?

This comment has been minimized.

@1st1

1st1 Mar 23, 2021
Member

I don't mind adding this function -- while somewhat trivial, it's something that projects like Cython (and potentially our own modules like _asynciomodule.c) have to reimplement.

@@ -7,6 +7,8 @@ extern "C" {

PyAPI_DATA(PyTypeObject) PySeqIter_Type;
PyAPI_DATA(PyTypeObject) PyCallIter_Type;
PyAPI_DATA(PyTypeObject) PyAsyncCallAwaitable_Type;

This comment has been minimized.

@gvanrossum

gvanrossum Mar 22, 2021
Member

Maybe these types and the function below should remain CPython implementation details? Just because they're returned by builtins doesn't mean all the implementation types need to be in the C-level API.

This comment has been minimized.

@1st1

1st1 Mar 23, 2021
Member

+1. I'd make them internal.

This comment has been minimized.

@jab

jab Mar 23, 2021
Author Contributor

Makes sense, thanks. Done in the latest revision. Another look?

Objects/abstract.c Outdated Show resolved Hide resolved
Copy link
Member

@gvanrossum gvanrossum left a comment

Almost there!

Equivalent to calling ``x.__aiter__()``.

*async_iterable* must be an :term:`asynchronous iterable`,
and :func:`aiter` returns an asynchronous iterator for it.
``aiter(aiter(x))`` is the same as ``aiter(x)``.
(``aiter(x)`` itself has an ``__aiter__()`` method that returns ``self``.)

Unlike the :func:`iter` builtin, :func:`aiter` has no 2-argument variant.
Often, this variant can be replaced with assignment expressions::
Formally, given an :term:`asynchronous iterable`,
return an :term:`asynchronous iterator`.
Comment on lines 66 to 72

This comment has been minimized.

@gvanrossum

gvanrossum Mar 22, 2021
Member

Much better! Here is how I would rearrange this:

Return an :term:`asynchronous iterator`
for an :term:`asynchronous iterable`. 
Equivalent to calling ``x.__aiter__()``.

``aiter(x)`` itself has an ``__aiter__()`` method that returns ``x``,
so ``aiter(aiter(x))`` is the same as ``aiter(x)``.

This comment has been minimized.

@jab

jab Mar 22, 2021
Author Contributor

Incorporated in the latest revision, thanks!

(((
Come to think of it, iter also has the iter(iter(x)) == iter(x) property, which is not currently mentioned in the iter docs. The iter docs have a lot more work to do though, to cover the 1- and 2-arg variants (and they're already doing this very well).

And now that I'm looking at those again, I notice the only code example there is for the 2-arg variant:

from functools import partial
with open('mydata.db', 'rb') as f:
    for block in iter(partial(f.read, 64), b''):
        process_block(block)

If this pattern is now obsoleted by assignment expressions[1], is it worth updating the iter docs to (1) mention the iter(iter(x)) == iter(x) property, and (2) remove the obsoleted example code? If so, happy to submit a separate PR for that.

[1] as in the following:

with open('mydata.db', 'rb') as f:
    while block := f.read(64):
        process_block(block)

)))

This comment has been minimized.

@gvanrossum

gvanrossum Mar 22, 2021
Member

I don't want to personally get into the weeds about the iter() docs, sorry. Something for the docs WG perhaps.

This comment has been minimized.

@brettcannon

brettcannon Mar 23, 2021
Member

I don't think it's worth updating the docs to promise iter(iter(x)) == iter(x) as it isn't a specific benefit to the user beyond logically doing the right thing.

This comment has been minimized.

@jab

jab Mar 23, 2021
Author Contributor

I can see that.

@gvanrossum, do you think @brettcannon's rationale applies equally to the aiter docs too? I did add a test_aiter_idempotent() for this in 6ee8824 to go along with the promise that the aiter docs are now making, but can remove that test along with that part of the docs if that's better.

This comment has been minimized.

@gvanrossum

gvanrossum Mar 23, 2021
Member

I think there's a different bar for updating the iter() docs than for the initial version of the aiter() docs. And they don't have to match precisely.

FWIW the main reason the idempotency property is important is because of the implicit [a]iter() call in a for-loop, since in

for i in x: ...

the for-loop calls iter(x), so that in

for i in iter(x): ...

the for-loop ends up calling iter(iter(x)). It's the same for async for and aiter().

Copy link
Member

@gvanrossum gvanrossum left a comment

Thanks for your patience. This all looks good to me now! I'll merge after applying my own suggestion.

Doc/library/functions.rst Outdated Show resolved Hide resolved
@gvanrossum gvanrossum merged commit f0a6fde into python:master Mar 23, 2021
11 checks passed
11 checks passed
@github-actions
Docs
Details
@github-actions
Check for source changes
Details
@github-actions
Check if generated files are up to date
Details
@github-actions
Windows (x86)
Details
@github-actions
Windows (x64) Windows (x64)
Details
@github-actions
macOS
Details
@github-actions
Ubuntu
Details
Azure Pipelines PR #20210323.44 succeeded
Details
@travis-ci
Travis CI - Pull Request Build Passed
Details
@bedevere-bot
bedevere/issue-number Issue number 31861 found
Details
@bedevere-bot
bedevere/news News entry found in Misc/NEWS.d
@bedevere-bot
Copy link

@bedevere-bot bedevere-bot commented Mar 23, 2021

@gvanrossum: Please replace # with GH- in the commit message next time. Thanks!

@gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented Mar 23, 2021

Congrats!

@jab
Copy link
Contributor Author

@jab jab commented Mar 23, 2021

Thanks so much, @gvanrossum! We’re honored to have been able to contribute!

@bedevere-bot
Copy link

@bedevere-bot bedevere-bot commented Mar 23, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL8 3.x has failed when building commit f0a6fde.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/509/builds/900) and take a look at the build logs.
  4. Check if the failure is related to this commit (f0a6fde) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/509/builds/900

Summary of the results of the build (if available):

== Tests result: ENV CHANGED ==

412 tests OK.

10 slowest tests:

  • test_concurrent_futures: 3 min 9 sec
  • test_multiprocessing_spawn: 1 min 36 sec
  • test_peg_generator: 1 min 26 sec
  • test_capi: 1 min 17 sec
  • test_gdb: 1 min 14 sec
  • test_unparse: 1 min 12 sec
  • test_multiprocessing_forkserver: 1 min 10 sec
  • test_asyncio: 1 min 5 sec
  • test_tokenize: 1 min 3 sec
  • test_multiprocessing_fork: 1 min

1 test altered the execution environment:
test_asyncio

14 tests skipped:
test_devpoll test_ioctl test_kqueue test_msilib test_nis
test_ossaudiodev test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

Total duration: 5 min 18 sec

Click to see traceback logs
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py", line 321, in __del__
    self.close()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py", line 316, in close
    self._ssl_protocol._start_shutdown()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py", line 590, in _start_shutdown
    self._abort()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py", line 731, in _abort
    self._transport.abort()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/selector_events.py", line 680, in abort
    self._force_close(None)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/selector_events.py", line 731, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/base_events.py", line 745, in call_soon
    self._check_closed()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
@gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented Mar 23, 2021

@vstinner
Copy link
Member

@vstinner vstinner commented Apr 7, 2021

I proposed to PR #25266 to rename PyAnextAwaitable_Type to _PyAnextAwaitable_Type, and to initialize the type at Python startup: can someone please have a look?

kreathon added a commit to kreathon/cpython that referenced this pull request May 2, 2021
Co-authored-by: jab <jab@users.noreply.github.com>
Co-authored-by: Daniel Pope <mauve@mauveweb.co.uk>
Co-authored-by: Justin Wang <justin39@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment