bpo-43142: Do not add duplicate FDs to list in duplicate_for_child() #24461
Conversation
While spawning, if the code built on top of python's multiprocessing module calls DupFd (in /Lib/multiprocessing/reduction.py) multiple times on the same File descriptor by mistake, then it ends up invoking duplicate_for_child(fd)) of the class Popen, which, in turn, adds a duplicate FD to a list. This list is then used in spawnv_passfds() in multiprocessing/util.py, which uses that list as an argument in a call of _posixsubprocess.fork_exec(). In Modules/_posixsubprocess.c, checks exist to ensure that all the FDs in the list are unique, positive, and in a sorted order. If duplicate entries are found, a 'ValueError: bad value(s) in fds_to_keep' exception is raised, and the execution is terminated because this exception is not handled by Python. Even if it were, a developer can't change the aforementioned FD list in Python without breaking modularity. It's better to let developers get away with calling DupFd (in /Lib/multiprocessing/reduction.py) by not accepting duplicate entries in the first place, so that spawning a process can be successful.
|
Hello, and thanks for your contribution! I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA). CLA MissingOur records indicate the following people have not signed the CLA: For legal reasons we need all the people listed to sign the CLA before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue. If you have recently signed the CLA, please wait at least one business day You can check yourself to see if the CLA has been received. Thanks again for the contribution, we look forward to reviewing it! |
imaginary-person
changed the title
All tests except the news item addition had passed the first time around, so need to re-run CI.
|
This PR is stale because it has been open for 30 days with no activity. |
github-actions
bot
removed
the
stale
bpo-43142: On Linux, while spawning another process, if a process that has subclassed Python's multiprocessing module calls
multiprocessing.reduction.DupFd()(in/Lib/multiprocessing/reduction.py) multiple times on the same File descriptor by mistake, then it ends up invokingduplicate_for_child(fd))of the classPopen, which, in turn, adds a duplicate FD to a list.This list is then used in
spawnv_passfds()in/Lib/multiprocessing/util.py, which uses that list as an argument in a call of_posixsubprocess.fork_exec().In
Modules/_posixsubprocess.c, checks exist to ensure that all the FDs in the list are unique, positive, and in a sorted order.If duplicate entries are found, a
ValueError: bad value(s) in fds_to_keepexception is raised, and the execution is terminated.Even if this exception were somehow handled, a developer can't change the aforementioned FD list in Python without breaking modularity. Moreover, the multiprocessing library should be robust enough to not pass duplicate FDs while calling
_posixsubprocess.fork_exec().It's better to let developers get away with calling
multiprocessing.reduction.DupFd()(in /Lib/multiprocessing/reduction.py) multiple times on some file descriptor by not adding duplicate entries to the list that's passed as a parameter in the call to_posixsubprocess.fork_exec(), so that spawning another process would be successful.Developers can then handle any other situations related to using FDs at their end, because the reason they would call
multiprocessing.reduction.DupFd()multiple times on the same FD would be related to their particular use-case, and they can use a workaround for it.https://bugs.python.org/issue43142