bpo-26552: Fixed case where failing `asyncio.ensure_future` did not close the coroutine #30288

kumaraditya303 · 2021-12-29T13:40:57Z

https://bugs.python.org/issue26552

Lib/asyncio/base_events.py

carlosdamazio · 2021-12-29T20:02:34Z

In Lib/asyncio/tasks.py, there's an ~~_enclose_future()~~_ensure_future(), you might be able to find out whether the function has called wrap_awaitable or not. I couldn't get the edge case when there's a task with a loop closed, of which was stated on the bug report. You might be able to flag out the specific edge case and then call create_task with that flag and corner it.

kumaraditya303 · 2022-01-24T07:26:33Z

(Rebased to main)

arhadthedev · 2022-01-24T07:40:29Z

*with hope* Does it fix test_asyncio random failures on a Windows runner so it can be enabled back?

kumaraditya303 · 2022-01-26T11:57:43Z

with hope Does it fix test_asyncio random failures on a Windows runner so it can be enabled back?

I fixed that issue separately.

gvanrossum

Hi Kumar, I had procrastinated on reviewing this because there's some complicated stuff going on. I finally found the time and noticed that the bpo issue was written somewhat misleadingly, so I've added a clarification there.

I now finally figured out what the real issue is and how to deal with it, and your code is correct:

If the argument is a Future, nothing more needs to be done
If it is a coroutine (i.e. async def), we need to close it if create_task() raises RuntimeError (which it only does for one reason, i.e. if the event loop is closed)
If it is an awaitable (i.e. something with an __await__() method), it may not have a close() method; the wrapper created by _wrap_awaitable() doesn't need to be closed

So I'm going to land this now. Thanks for your patience!

bedevere-bot · 2022-01-28T22:24:38Z

@gvanrossum: Please replace # with GH- in the commit message next time. Thanks!

bedevere-bot · 2022-01-28T22:59:20Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot PPC64LE CentOS9 3.x has failed when building commit 24cc641.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/852/builds/10) and take a look at the build logs.
Check if the failure is related to this commit (24cc641) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/852/builds/10

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 13, done.        
remote: Counting objects:   9% (1/11)        
remote: Counting objects:  18% (2/11)        
remote: Counting objects:  27% (3/11)        
remote: Counting objects:  36% (4/11)        
remote: Counting objects:  45% (5/11)        
remote: Counting objects:  54% (6/11)        
remote: Counting objects:  63% (7/11)        
remote: Counting objects:  72% (8/11)        
remote: Counting objects:  81% (9/11)        
remote: Counting objects:  90% (10/11)        
remote: Counting objects: 100% (11/11)        
remote: Counting objects: 100% (11/11), done.        
remote: Total 13 (delta 11), reused 11 (delta 11), pack-reused 2        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to '24cc6411adbfe5555ecd8901f1ea50caa414c908'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 24cc6411ad [bpo-26552](https://bugs.python.org/issue26552): Fixed case where failing `asyncio.ensure_future` did not close the coroutine (#30288)
Switched to and reset branch 'main'

Cannot create temporary file in /tmp/: No space left on device
../Python/thread.c:257:1: fatal error: error writing to /tmp/ccTo9nrX.s: No space left on device
  257 | }
      | ^
compilation terminated.
make: *** [Makefile:2392: Python/thread.o] Error 1
make: *** Waiting for unfinished jobs....
make: *** [Makefile:2392: Python/pystrtod.o] Aborted (core dumped)
../Python/pystrcmp.c:30:1: fatal error: error writing to /tmp/ccodR3vP.s: No space left on device
   30 | }
      | ^
compilation terminated.
make: *** [Makefile:2392: Python/pystrcmp.o] Error 1
../Parser/parser.c:35925:1: fatal error: error writing to /tmp/cc3DnvC8.s: No space left on device
35925 | }
      | ^
compilation terminated.
make: *** [Makefile:2392: Parser/parser.o] Error 1
../Python/symtable.c:2188:1: fatal error: error writing to /tmp/cc0vWBNO.s: No space left on device
 2188 | }
      | ^
compilation terminated.
make: *** [Makefile:2392: Python/symtable.o] Error 1
../Python/traceback.c:1334:1: fatal error: error writing to /tmp/cc6eT91n.s: No space left on device
 1334 | }
      | ^
compilation terminated.
make: *** [Makefile:2392: Python/traceback.o] Error 1
../Python/sysmodule.c:3383:1: fatal error: error writing to /tmp/ccjLnafA.s: No space left on device
 3383 | }
      | ^
compilation terminated.
make: *** [Makefile:1373: Python/sysmodule.o] Error 1
../Objects/unicodeobject.c:16167:1: fatal error: error writing to /tmp/ccmwQWIw.s: No space left on device
16167 | }
      | ^
compilation terminated.
make: *** [Makefile:2392: Objects/unicodeobject.o] Error 1

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make: [Makefile:2472: clean-retain-profile] Error 1 (ignored)

kumaraditya303 · 2022-01-29T03:44:13Z

@gvanrossum I think this should be backported to 3.10 and 3.9.

gvanrossum · 2022-01-29T04:33:28Z

Yeah, makes sense. Adding the labels now.

miss-islington · 2022-01-29T04:33:41Z

Thanks @kumaraditya303 for the PR, and @gvanrossum for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

miss-islington · 2022-01-29T04:33:42Z

Thanks @kumaraditya303 for the PR, and @gvanrossum for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

miss-islington · 2022-01-29T04:33:53Z

Sorry, @kumaraditya303 and @gvanrossum, I could not cleanly backport this to 3.9 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 24cc6411adbfe5555ecd8901f1ea50caa414c908 3.9

miss-islington · 2022-01-29T04:33:55Z

Sorry @kumaraditya303 and @gvanrossum, I had trouble checking out the 3.10 backport branch.
Please backport using cherry_picker on command line.
cherry_picker 24cc6411adbfe5555ecd8901f1ea50caa414c908 3.10

…lose the coroutine (python#30288)

kumaraditya303 · 2022-01-29T05:18:43Z

I am leaving 3.9 as it is too diverged from main.

…lose the coroutine (#30288) (#31003)

bedevere-bot · 2022-01-29T06:59:42Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot PPC64LE CentOS9 3.10 has failed when building commit a5451c9.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/843/builds/5) and take a look at the build logs.
Check if the failure is related to this commit (a5451c9) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/843/builds/5

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 13, done.        
remote: Counting objects:   8% (1/12)        
remote: Counting objects:  16% (2/12)        
remote: Counting objects:  25% (3/12)        
remote: Counting objects:  33% (4/12)        
remote: Counting objects:  41% (5/12)        
remote: Counting objects:  50% (6/12)        
remote: Counting objects:  58% (7/12)        
remote: Counting objects:  66% (8/12)        
remote: Counting objects:  75% (9/12)        
remote: Counting objects:  83% (10/12)        
remote: Counting objects:  91% (11/12)        
remote: Counting objects: 100% (12/12)        
remote: Counting objects: 100% (12/12), done.        
remote: Compressing objects:   9% (1/11)        
remote: Compressing objects:  18% (2/11)        
remote: Compressing objects:  27% (3/11)        
remote: Compressing objects:  36% (4/11)        
remote: Compressing objects:  45% (5/11)        
remote: Compressing objects:  54% (6/11)        
remote: Compressing objects:  63% (7/11)        
remote: Compressing objects:  72% (8/11)        
remote: Compressing objects:  81% (9/11)        
remote: Compressing objects:  90% (10/11)        
remote: Compressing objects: 100% (11/11)        
remote: Compressing objects: 100% (11/11), done.        
remote: Total 13 (delta 1), reused 11 (delta 1), pack-reused 1        
From https://github.com/python/cpython
 * branch                  3.10       -> FETCH_HEAD
Note: switching to 'a5451c96a14ac0c3ee7cef7b5c52ab1d783ec613'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at a5451c96a1 [bpo-26552](https://bugs.python.org/issue26552): Fixed case where failing `asyncio.ensure_future` did not close the coroutine (#30288) (#31003)
Switched to and reset branch '3.10'

../Modules/_io/textio.c:3338:1: fatal error: error writing to /tmp/cc6yGOcE.s: No space left on device
 3338 | };
      | ^
compilation terminated.
make: *** [Makefile:2085: Modules/textio.o] Error 1
make: *** Waiting for unfinished jobs....
../Programs/_testembed.c:1899:1: fatal error: error writing to /tmp/cclxhfYW.s: No space left on device
 1899 | }
      | ^
compilation terminated.
make: *** [Makefile:816: Programs/_testembed.o] Error 1
/tmp/ccqYhb34.s: Assembler messages:
/tmp/ccqYhb34.s: Fatal error: can't close Objects/unicodeobject.o: No space left on device
make: *** [Makefile:1855: Objects/unicodeobject.o] Error 1
../Modules/posixmodule.c:15828:1: fatal error: error writing to /tmp/ccmkQztW.s: No space left on device
15828 | }
      | ^
compilation terminated.
make: *** [Makefile:2046: Modules/posixmodule.o] Error 1

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make: [Makefile:1928: clean-retain-profile] Error 1 (ignored)

kumaraditya303 requested review from 1st1 and asvetlov as code owners Dec 29, 2021

the-knights-who-say-ni added the CLA signed label Dec 29, 2021

bedevere-bot added the awaiting review label Dec 29, 2021

kumaraditya303 changed the title ~~bpo-26552: fix failing ensure_future which created a Task~~ bpo-26552: Fixed case where failing asyncio.ensure_future did not close the coroutine Dec 29, 2021

gvanrossum reviewed Dec 29, 2021

View changes

Lib/asyncio/base_events.py Outdated Show resolved Hide resolved

bpo-26552: fix failing ensure_future still creates a Task

ea3a9b4

kumaraditya303 force-pushed the issue26552 branch from 0d0fd3f to ea3a9b4 Compare Jan 24, 2022

new approach

d4ecd36

kumaraditya303 requested a review from gvanrossum Jan 26, 2022

gvanrossum approved these changes Jan 28, 2022

View changes

bedevere-bot added awaiting merge and removed awaiting review labels Jan 28, 2022

gvanrossum merged commit 24cc641 into python:main Jan 28, 2022
12 checks passed

bedevere-bot removed the awaiting merge label Jan 28, 2022

kumaraditya303 deleted the issue26552 branch Jan 29, 2022

gvanrossum added needs backport to 3.9 needs backport to 3.10 labels Jan 29, 2022

miss-islington assigned gvanrossum Jan 29, 2022

kumaraditya303 added a commit to kumaraditya303/cpython that referenced this pull request Jan 29, 2022

bpo-26552: Fixed case where failing asyncio.ensure_future did not c…

e514f9f

…lose the coroutine (python#30288)

kumaraditya303 mentioned this pull request Jan 29, 2022

[3.10] bpo-26552: Fixed case where failing asyncio.ensure_future did not c… …lose the coroutine #31003

Merged

gvanrossum pushed a commit that referenced this pull request Jan 29, 2022

bpo-26552: Fixed case where failing asyncio.ensure_future did not c…

a5451c9

…lose the coroutine (#30288) (#31003)

bpo-26552: Fixed case where failing `asyncio.ensure_future` did not close the coroutine #30288

bpo-26552: Fixed case where failing `asyncio.ensure_future` did not close the coroutine #30288

kumaraditya303 commented Dec 29, 2021 •

edited by bedevere-bot

carlosdamazio commented Dec 29, 2021 •

edited by gvanrossum

kumaraditya303 commented Jan 24, 2022

arhadthedev commented Jan 24, 2022

kumaraditya303 commented Jan 26, 2022

gvanrossum left a comment

bedevere-bot commented Jan 28, 2022

bedevere-bot commented Jan 28, 2022 •

edited

kumaraditya303 commented Jan 29, 2022

gvanrossum commented Jan 29, 2022

miss-islington commented Jan 29, 2022

miss-islington commented Jan 29, 2022

miss-islington commented Jan 29, 2022

miss-islington commented Jan 29, 2022

kumaraditya303 commented Jan 29, 2022

bedevere-bot commented Jan 29, 2022 •

edited

bpo-26552: Fixed case where failing asyncio.ensure_future did not close the coroutine #30288

bpo-26552: Fixed case where failing asyncio.ensure_future did not close the coroutine #30288

Conversation

kumaraditya303 commented Dec 29, 2021 • edited by bedevere-bot

carlosdamazio commented Dec 29, 2021 • edited by gvanrossum

kumaraditya303 commented Jan 24, 2022

arhadthedev commented Jan 24, 2022

kumaraditya303 commented Jan 26, 2022

gvanrossum left a comment

bedevere-bot commented Jan 28, 2022

bedevere-bot commented Jan 28, 2022 • edited

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

kumaraditya303 commented Jan 29, 2022

gvanrossum commented Jan 29, 2022

miss-islington commented Jan 29, 2022

miss-islington commented Jan 29, 2022

miss-islington commented Jan 29, 2022

miss-islington commented Jan 29, 2022

kumaraditya303 commented Jan 29, 2022

bedevere-bot commented Jan 29, 2022 • edited

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bpo-26552: Fixed case where failing `asyncio.ensure_future` did not close the coroutine #30288

bpo-26552: Fixed case where failing `asyncio.ensure_future` did not close the coroutine #30288

kumaraditya303 commented Dec 29, 2021 •

edited by bedevere-bot

carlosdamazio commented Dec 29, 2021 •

edited by gvanrossum

bedevere-bot commented Jan 28, 2022 •

edited

bedevere-bot commented Jan 29, 2022 •

edited