New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-39104 Fix hanging ProcessPoolExcutor on shutdown nowait with pickling failure #17670

Open

tomMoral wants to merge 2 commits into python:master from tomMoral:FIX_hanging_interpreter_executor_shutdown_nowait

+48 −17

Contributor

tomMoral commented Dec 20, 2019 •

edited by bedevere-bot

As reported initially by @rad-pat in #6084, the following script causes a deadlock.

from concurrent.futures import ProcessPoolExecutor


class ObjectWithPickleError():
    """Triggers a RuntimeError when sending job to the workers"""

    def __reduce__(self):
        raise RuntimeError()


if __name__ == "__main__":
    e = ProcessPoolExecutor()
    f = e.submit(id, ObjectWithPickleError())
    e.shutdown(wait=False)
    f.result()  # Deadlock on get

This is caused by the fact that the main process is closing communication channels that might be necessary to the queue_management_thread later. To avoid this, this PR let the queue_management_thread manage all the closing.

https://bugs.python.org/issue39104

tomMoral added 2 commits


        FIX hang on executor shutdown no wait with failure

44f70fd


        TST non regression test for #6084

38d8c98

the-knights-who-say-ni added the CLA signed label

bedevere-bot added the awaiting review label

tomMoral mentioned this pull request

bpo-33056 FIX leaking fd in concurrent.futures.ProcessPoolExecutor #6084

Merged

rad-pat reviewed

rad-pat left a comment

Just a comment. I had also started trying to make a solution, but perhaps I didn't grasp the entire problem. Please see rad-pat@188c222, including a more basic test without the pickle error.

Lib/concurrent/futures/process.py

  
      @@ -87,7 +87,12 @@ def close(self):
    
                self._reader.close()

            def wakeup(self):

                self._writer.send_bytes(b"")

                try:

rad-pat Dec 20, 2019

Maybe a _closed flag might be better than catching the exception? You could also then check the flag in the clear method?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

You can’t perform that action at this time.