Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Why doesn't asyncio.as_completed cancel unfinished tasks if it timeout? #99933

Open
NewUserHa opened this issue Dec 1, 2022 · 3 comments
Open

Why doesn't asyncio.as_completed cancel unfinished tasks if it timeout? #99933

NewUserHa opened this issue Dec 1, 2022 · 3 comments
Labels
docs Documentation in the Doc dir expert-asyncio

Comments

@NewUserHa
Copy link
Contributor

NewUserHa commented Dec 1, 2022

Feature or enhancement

asyncio.as_completed should cancel unfinished tasks if it timeout, or even maybe have the ability to cancel no needed task

Pitch

cpython/Lib/asyncio/tasks.py

Lines 588 to 609 in 0563be2

def _on_timeout():
for f in todo:
f.remove_done_callback(_on_completion)
done.put_nowait(None) # Queue a dummy value for _wait_for_one().
todo.clear() # Can't do todo.remove(f) in the loop.
def _on_completion(f):
if not todo:
return # _on_timeout() was here first.
todo.remove(f)
done.put_nowait(f)
if not todo and timeout_handle is not None:
timeout_handle.cancel()
async def _wait_for_one():
f = await done.get()
if f is None:
# Dummy value from _on_timeout().
raise exceptions.TimeoutError
return f.result() # May raise f.exception().

Why doesn't asyncio.as_completed cancel unfinished tasks if it timeout?
Maybe say there may be many network requests pending, since that already raised exception then future requests should probably be canceled to save resources.
Probably because the concurrent.futures.as_complete didn't cancel unfinished tasks? But it's local resources than asyncio is usually network resources.

More, I know the asyncio.wait() can do cancel the pending job. But asyncio.as_complete() probably should also be able to cancel the next coros if got the wanted result. But currently, the returned is a coroutine that can't cancel.
@NewUserHa NewUserHa added the type-feature A feature request or enhancement label Dec 1, 2022
@gvanrossum
Copy link
Member

gvanrossum commented Dec 1, 2022

This is indeed unfortunate, but we cannot change it without breaking backwards compatibility. I recommend trying to work around it using task groups.

@gvanrossum gvanrossum closed this as not planned Won't fix, can't repro, duplicate, stale Dec 1, 2022
@NewUserHa
Copy link
Contributor Author

NewUserHa commented Dec 1, 2022

But that can definitely waste unnecessary network resources if users input many awaitables to the asyncio.as_completed() isn't it?

Should the document mention that situation for the users who never read the source code? Or recommend using 'task group' in the document of asyncio.as_completed() as well? (currently, only asyncio.gather() recommend using 'task group')

@gvanrossum
Copy link
Member

gvanrossum commented Dec 1, 2022

I agree that we could add a note to the docs for as_completed. If we wanted to recommend task groups we'd at least have to give a code sample showing how to do that -- it's not completely trivial.

@gvanrossum gvanrossum added docs Documentation in the Doc dir and removed type-feature A feature request or enhancement labels Dec 1, 2022
@gvanrossum gvanrossum reopened this Dec 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
docs Documentation in the Doc dir expert-asyncio
Projects
asyncio
Status: In Progress
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gvanrossum @NewUserHa @AlexWaygood