bpo-32528: Make asyncio.CancelledError a BaseException. #13528

1st1 · 2019-05-23T21:38:07Z

This will address the common mistake many asyncio users make:
an "except Exception" clause breaking Tasks cancellation.

In addition to this change, we stop inheriting asyncio.TimeoutError
and asyncio.InvalidStateError from their concurrent.futures.*
counterparts. There's no point for these exceptions to share the
inheritance chain.

https://bugs.python.org/issue32528

asvetlov · 2019-05-24T13:01:27Z

I see a small problem with the PR in test_base_events.py:

    def test_run_until_complete_loop_orphan_future_close_loop(self):
        class ShowStopper(BaseException):
            pass

        async def foo(delay):
            await asyncio.sleep(delay)

        def throw():
            raise ShowStopper

        self.loop._process_events = mock.Mock()
        self.loop.call_soon(throw)
        try:
            self.loop.run_until_complete(foo(0.1))
        except ShowStopper:
            pass

        # This call fails if run_until_complete does not clean up
        # done-callback for the previous future.
        self.loop.run_until_complete(foo(0.2))

The test passes but a message is printed:

test_run_until_complete_loop_orphan_future_close_loop (test.test_asyncio.test_base_events.BaseEventLoopTests) ... Exception in callback BaseEventLoopTests.test_run_until_complete_loop_orphan_future_close_loop.<locals>.throw() at /home/andrew/projects/cpython/Lib/test/test_asyncio/test_base_events.py:485
handle: <Handle BaseEventLoopTests.test_run_until_complete_loop_orphan_future_close_loop.<locals>.throw() at /home/andrew/projects/cpython/Lib/test/test_asyncio/test_base_events.py:485>
Traceback (most recent call last):
  File "/home/andrew/projects/cpython/Lib/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/home/andrew/projects/cpython/Lib/test/test_asyncio/test_base_events.py", line 486, in throw
    raise ShowStopper
test.test_asyncio.test_base_events.BaseEventLoopTests.test_run_until_complete_loop_orphan_future_close_loop.<locals>.ShowStopper
ok

I suggest a little different implementation.
On catching exceptions let's don't change except Exception to except BaseException but replace it with except (Exception, asyncio.CancelledError).

By this, we keep exception handling logic in asyncio unchanged but put CacelledError at the proper place in exceptions hierarchy.
I don't know if users use custom BaseException derived classes in their code. This PR has a potential regression risk, while my proposal eliminates it (I hope).

asvetlov

After offline talk @1st1 convinced me that we should process all exceptions as proposed by the PR.
Eventually in Python 3.9 we will get rid of custom handling of SystemExit and KeyboardInterrupt.

This will address the common mistake many asyncio users make: an "except Exception" clause breaking Tasks cancellation. In addition to this change, we stop inheriting asyncio.TimeoutError and asyncio.InvalidStateError from their concurrent.futures.* counterparts. There's no point for these exceptions to share the inheritance chain.

python/cpython#13528 broke us in two ways: asyncio.CancelledError is no longer an alias for concurrent.futures.CancelledError and it's now a BaseException. Fixes tornadoweb#2677 Closes tornadoweb#2681

* See also: https://bugs.python.org/issue32528 * This change makes actxmgr cancellation behaviors in Python 3.6/3.7/3.8 consistent.

This will address the common mistake many asyncio users make: an "except Exception" clause breaking Tasks cancellation. In addition to this change, we stop inheriting asyncio.TimeoutError and asyncio.InvalidStateError from their concurrent.futures.* counterparts. There's no point for these exceptions to share the inheritance chain. In 3.9 we'll focus on implementing supervisors and cancel scopes, which should allow better handling of all exceptions, including SystemExit and KeyboardInterrupt

@1st1

…or (GH-21474) #msg373510 [bpo-32528]()/#13528 changed `asyncio.CancelledError` such that it no longer inherits from `concurrent.futures.CancelledError`. As this affects existing code, specifically when catching the latter instead of the former in exception handling, it should be documented in the "What's new in 3.8?" document. Automerge-Triggered-By: @1st1

@1st1

…or (pythonGH-21474) GH-msg373510 [bpo-32528]()/pythonGH-13528 changed `asyncio.CancelledError` such that it no longer inherits from `concurrent.futures.CancelledError`. As this affects existing code, specifically when catching the latter instead of the former in exception handling, it should be documented in the "What's new in 3.8?" document. Automerge-Triggered-By: @1st1 (cherry picked from commit 2a51818) Co-authored-by: JustAnotherArchivist <JustAnotherArchivist@users.noreply.github.com>

@1st1

…or (pythonGH-21474) GH-msg373510 [bpo-32528]()/pythonGH-13528 changed `asyncio.CancelledError` such that it no longer inherits from `concurrent.futures.CancelledError`. As this affects existing code, specifically when catching the latter instead of the former in exception handling, it should be documented in the "What's new in 3.8?" document. Automerge-Triggered-By: @1st1 (cherry picked from commit 2a51818) Co-authored-by: JustAnotherArchivist <JustAnotherArchivist@users.noreply.github.com>

@1st1

…or (GH-21474) GH-msg373510 [bpo-32528]()/GH-13528 changed `asyncio.CancelledError` such that it no longer inherits from `concurrent.futures.CancelledError`. As this affects existing code, specifically when catching the latter instead of the former in exception handling, it should be documented in the "What's new in 3.8?" document. Automerge-Triggered-By: @1st1 (cherry picked from commit 2a51818) Co-authored-by: JustAnotherArchivist <JustAnotherArchivist@users.noreply.github.com>

@1st1

…or (pythonGH-21474) #msg373510 [bpo-32528]()/python#13528 changed `asyncio.CancelledError` such that it no longer inherits from `concurrent.futures.CancelledError`. As this affects existing code, specifically when catching the latter instead of the former in exception handling, it should be documented in the "What's new in 3.8?" document. Automerge-Triggered-By: @1st1

@1st1

…or (pythonGH-21474) GH-msg373510 [bpo-32528]()/pythonGH-13528 changed `asyncio.CancelledError` such that it no longer inherits from `concurrent.futures.CancelledError`. As this affects existing code, specifically when catching the latter instead of the former in exception handling, it should be documented in the "What's new in 3.8?" document. Automerge-Triggered-By: @1st1 (cherry picked from commit 2a51818) Co-authored-by: JustAnotherArchivist <JustAnotherArchivist@users.noreply.github.com>

@1st1

…or (GH-21474) GH-msg373510 [bpo-32528]()/GH-13528 changed `asyncio.CancelledError` such that it no longer inherits from `concurrent.futures.CancelledError`. As this affects existing code, specifically when catching the latter instead of the former in exception handling, it should be documented in the "What's new in 3.8?" document. Automerge-Triggered-By: @1st1 (cherry picked from commit 2a51818) Co-authored-by: JustAnotherArchivist <JustAnotherArchivist@users.noreply.github.com>

* fix for 3.8 python pull 13528 - python/cpython#13528 * changes * flake8

@1st1

…or (pythonGH-21474) #msg373510 [bpo-32528]()/python#13528 changed `asyncio.CancelledError` such that it no longer inherits from `concurrent.futures.CancelledError`. As this affects existing code, specifically when catching the latter instead of the former in exception handling, it should be documented in the "What's new in 3.8?" document. Automerge-Triggered-By: @1st1

@1st1

…or (pythonGH-21474) #msg373510 [bpo-32528]()/python#13528 changed `asyncio.CancelledError` such that it no longer inherits from `concurrent.futures.CancelledError`. As this affects existing code, specifically when catching the latter instead of the former in exception handling, it should be documented in the "What's new in 3.8?" document. Automerge-Triggered-By: @1st1

@1st1

…or (pythonGH-21474) #msg373510 [bpo-32528]()/python#13528 changed `asyncio.CancelledError` such that it no longer inherits from `concurrent.futures.CancelledError`. As this affects existing code, specifically when catching the latter instead of the former in exception handling, it should be documented in the "What's new in 3.8?" document. Automerge-Triggered-By: @1st1

@1st1

…or (pythonGH-21474) #msg373510 [bpo-32528]()/python#13528 changed `asyncio.CancelledError` such that it no longer inherits from `concurrent.futures.CancelledError`. As this affects existing code, specifically when catching the latter instead of the former in exception handling, it should be documented in the "What's new in 3.8?" document. Automerge-Triggered-By: @1st1

1st1 requested a review from asvetlov May 23, 2019

1st1 requested a review from gpshead as a code owner May 23, 2019

the-knights-who-say-ni added the CLA signed label May 23, 2019

bedevere-bot added the awaiting core review label May 23, 2019

1st1 removed the request for review from gpshead May 23, 2019

asvetlov added the expert-asyncio label May 24, 2019

asvetlov reviewed May 24, 2019

View changes

asvetlov approved these changes May 24, 2019

View changes

bedevere-bot added awaiting merge and removed awaiting core review labels May 24, 2019

1st1 added 2 commits May 27, 2019

Fix test_run_until_complete_loop_orphan_future_close_loop

04bd65e

1st1 force-pushed the exc branch from 02aa811 to 04bd65e Compare May 27, 2019

1st1 merged commit 431b540 into python:master May 27, 2019
5 checks passed

bedevere-bot removed the awaiting merge label May 27, 2019

1st1 deleted the exc branch May 27, 2019

bdarnell mentioned this pull request Jun 22, 2019

gen: Adapt to py38 CancelledError changes tornadoweb/tornado#2683

Merged

achimnol added a commit to achimnol/aiotools that referenced this issue Aug 28, 2019

Catch cancellation as part of BaseException (python/cpython#13528)

deb5a15

* See also: https://bugs.python.org/issue32528 * This change makes actxmgr cancellation behaviors in Python 3.6/3.7/3.8 consistent.

jrbourbeau mentioned this pull request Jan 27, 2020

Add support for Python 3.8 dask/distributed#3249

Merged

JustAnotherArchivist mentioned this pull request Jul 14, 2020

bpo-32528: Document the change in inheritance of asyncio.CancelledError #21474

Merged

miss-islington mentioned this pull request Jul 14, 2020

[3.9] bpo-32528: Document the change in inheritance of asyncio.CancelledError (GH-21474) #21475

Closed

miss-islington mentioned this pull request Jul 14, 2020

[3.8] bpo-32528: Document the change in inheritance of asyncio.CancelledError (GH-21474) #21476

Merged

miss-islington mentioned this pull request Jul 24, 2020

[3.9] bpo-32528: Document the change in inheritance of asyncio.CancelledError (GH-21474) #21606

Merged

hoffmang9 pushed a commit to Chia-Network/chia-blockchain that referenced this issue Jul 27, 2020

fix for 3.8 python pull 13528 - 38exceptwjb (#333)

4fe1806

* fix for 3.8 python pull 13528 - python/cpython#13528 * changes * flake8

hoffmang9 pushed a commit to Chia-Network/chia-blockchain that referenced this issue Jul 27, 2020

fix for 3.8 python pull 13528 - 38exceptwjb (#333)

2563053

* fix for 3.8 python pull 13528 - python/cpython#13528 * changes * flake8

vxgmichel referenced this issue in vxgmichel/aioconsole Jan 26, 2022

Explicitely wrap base exceptions in run_as_daemon

1b7717f

python / cpython Public

bpo-32528: Make asyncio.CancelledError a BaseException. #13528

bpo-32528: Make asyncio.CancelledError a BaseException. #13528

1st1 commented May 23, 2019 •

edited by bedevere-bot

asvetlov commented May 24, 2019

asvetlov left a comment

python / cpython Public

bpo-32528: Make asyncio.CancelledError a BaseException. #13528

bpo-32528: Make asyncio.CancelledError a BaseException. #13528

Conversation

1st1 commented May 23, 2019 • edited by bedevere-bot

asvetlov commented May 24, 2019

asvetlov left a comment

1st1 commented May 23, 2019 •

edited by bedevere-bot