subprocess gained the ability to use close_range() in #84603, however, it's only used as a fallback if procfs is not available, which means that at least on Linux it's almost never used. Its performance benefits are significant (see numbers at the end). I propose to use it by default, with procfs being the second option.
This requires some extra changes because _Py_closerange() was designed to be standalone and tries all available methods, including brute force, and also because it's not required to be async-signal-safe, but we must preserve async-signal-safety of fd closing code on Linux. I'll open a PR with a proposed implementation.
Performance comparison that I did on a test version with the following script (may require you to raise the fd limit in your shell):
import os, subprocess, sys, timeit
from resource import *
soft, hard = getrlimit(RLIMIT_NOFILE)
setrlimit(RLIMIT_NOFILE, (hard, hard))
num_fds, num_iter = map(int, sys.argv[1:3])
for i in range(num_fds):
os.open('/dev/null', os.O_RDONLY)
print(timeit.timeit(lambda: subprocess.run('/bin/true'), number=num_iter))
…#92303)
#92301: subprocess: Prefer `close_range()` to procfs-based fd closing.
`close_range()` is much faster for large number of file descriptors, e.g.
4 times faster for 1000 descriptors in a Linux 5.16-based environment.
We prefer close_range() only if it's known to be async-signal-safe.
subprocess gained the ability to use close_range() in #84603, however, it's only used as a fallback if procfs is not available, which means that at least on Linux it's almost never used. Its performance benefits are significant (see numbers at the end). I propose to use it by default, with procfs being the second option.
This requires some extra changes because _Py_closerange() was designed to be standalone and tries all available methods, including brute force, and also because it's not required to be async-signal-safe, but we must preserve async-signal-safety of fd closing code on Linux. I'll open a PR with a proposed implementation.
Performance comparison that I did on a test version with the following script (may require you to raise the fd limit in your shell):
Before:
After:
The text was updated successfully, but these errors were encountered: