Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-77714: Implement as_completed as an asynchronous generator #10251

Closed
wants to merge 5 commits into from

Conversation

mivade
Copy link
Contributor

@mivade mivade commented Oct 31, 2018

This implements asyncio.as_completed with async for support while keeping the old iteration method for backwards compatibility.

This obviously needs new tests. I was also hoping for some additional feedback regarding the implementation.

https://bugs.python.org/issue33533

Lib/asyncio/tasks.py Outdated Show resolved Hide resolved
exceptions!) of the original Futures (or coroutines), in the order
in which and as soon as they complete.
This differs from PEP 3148; results are yielded from asynchronous
iteration rather than futures.
Copy link
Contributor

@hniksic hniksic Oct 31, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new implementation should also return futures, just like the one from concurrent.futures does. The difference is that with async iteration we can return the original futures, which are guaranteed to have completed.

Changing yield future.result() to yield future in __aiter__ makes it correct, and compatible with PEP 3148.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The existing implementation of as_completed already yields results. I also think yielding results rather than futures is more natural with async for.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The existing implementation yields futures which you must await.

The idea is to yield futures which are guaranteed to have completed, as they complete. This is how concurrent.futures.as_completed works, but the benefit goes beyond consistency with concurrent.futures. It allows code to be robust in the face of exceptions raised by futures, and to associate side data with the futures and retrieve it from as_completed. For example, when yielding futures, this usage example immediately becomes usable with asyncio.as_completed, just changing for to async for.

The sync version doesn't support that usage because it gives you a different future than any of those passed to as_completed. Among other things, that's what prompted the BPO issue.

As far as I can tell, your implementation can easily support that, just by removing .result() at the yield point.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I misread what was going on before. Thanks for the explanation!

Copy link
Contributor

@willingc willingc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @mivade, Thanks for the PR. Since this PR has been open for a while, would you prefer to close the PR or resolve the CI errors? Thanks.

@mivade
Copy link
Contributor Author

mivade commented May 1, 2023

It has indeed been quite some time! I should be able to take a look at resolving conflicts later this week if there is not a competing PR that would be ready to merge.

@gvanrossum gvanrossum changed the title bpo-33533: Implement as_completed as an asynchronous generator gh-77714: Implement as_completed as an asynchronous generator May 7, 2023
@gvanrossum
Copy link
Member

@kumaraditya303 Can you review this? I just don't have the time, and you expressed interest in the GH issue before.

@kumaraditya303 kumaraditya303 added the 3.13 bugs and security fixes label May 10, 2023
@bvd0
Copy link

bvd0 commented Dec 16, 2023

What needs to happen for this to get merged?

@gvanrossum
Copy link
Member

I’ve asked @serhiy-storchaka for help.

@serhiy-storchaka serhiy-storchaka self-requested a review December 16, 2023 17:35
# This is *not* a @coroutine! It is just an iterator (yielding Futures).
def as_completed(fs, *, timeout=None):
"""Return an iterator whose values are coroutines.
class as_completed(object):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
class as_completed(object):
class as_completed:

Inheriting from object isn't necessary

@serhiy-storchaka
Copy link
Member

Thank you for your PR, but #22491 is more complete. It returns an iterator (not simply an iterable) and contains tests. So we focused on it.

@mivade
Copy link
Contributor Author

mivade commented Dec 24, 2023

Glad to see there's movement on addressing this issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.