The problem comes from pip bootstrap when running the test_with_pip() test of test_venv.

It seems like pip creates a .pem file in the temporary directory (like /tmp), but only when pip is imported from a ZIP file, which is the case when I run get-pip.py: https://bootstrap.pypa.io/get-pip.py

I can reproduced the issue in the 3.9, 3.10, 3.11 and main branches (I didn't test older branches).

• In Python 3.9, I cannot reproduce the issue in Python 3.9 at commit 4b4d60f: pip 20.2.3.
• But I can reproduce the issue at commit d962b00: pip 21.1.

It seems to be a change between pip 20.2.3 and pip 21.1.

In get-pip.py, from pip._internal.commands.install import InstallCommand is enough to create the .pem file.

Shell script to reproduce the issue:

set -e -x

DIR=$PWD/TMP/
rm -rf $DIR
mkdir $DIR

rm -rf env
TMPDIR=$DIR TEMPDIR=$DIR ./python -m venv env --without-pip
TMPDIR=$DIR TEMPDIR=$DIR env/bin/python -m ensurepip -v

echo
echo
echo "=== TMPDIR ==="
find $DIR

Output:

(...)
=== TMPDIR ===
++ find /home/vstinner/python/main/TMP/
/home/vstinner/python/main/TMP/
/home/vstinner/python/main/TMP/tmpmb5ndxc7cacert.pem

The PEM file is created indirectly by importlib.resources on import pip._vendor.requests.

It's created by this code:

>>> import sys; sys.path.insert(0, 'pip.zip')
>>> import pip._vendor.certifi
>>> pip._vendor.certifi.where()
pip.zip/pip/_vendor/certifi/core.py:50: DeprecationWarning: path is deprecated. Use files() instead. Refer to https://importlib-resources.readthedocs.io/en/latest/using.html#migrating-from-legacy for migration advice.
'/home/vstinner/python/main/TMP/tmp9osgb6wrcacert.pem'

Debugger:

Lib/importlib/resources/_common.py, line 84, in _tempfile (suffix='cacert.pem', fd=4, raw_path='/home/vstinner/python/main/TMP/tmpv7fwzaoocacert.pem')

If Python exits normally, the garbage collector deletes the temporary PEM file. When running ensurepip / get-pip.py, it's not deleted. I don't know why.

importlib.resources only creates a temporary file if pip is imported from a ZIP file.

Workaround to manually delete the temporaryfile:

pip._vendor.certifi.core._CACERT_CTX = None

OpenSSL requires a filename to a PEM file. certifi uses importlib.resources to create a temporary named file if pip._vendor.certifi is imported from a ZIP file. That's convenient.

The problem is more that the pip._vendor.certifi.core._CACERT_CTX context manager which keeps the temporary file alive until it's destroyed. Maybe sometimes _CACERT_CTX is part of a complex reference cycle, it's never destroyed, and so the temporary file is not deleted.

Nice. How did you find this?

I used a shell script similar to #93353 (comment) and I ran manually a bisection on the test list (./python -m test --list-tests) using a script taking a random.sample() of half of tests. I repeated the operation until I found a single test file, then I looked at the code.

The leaked files was reported by more and more owners of buildbot workers.

Once the PEM issue is solved, my plan is to enhance test.regrtest to do something similar than the shell script: define TMPFILE env var. Maybe in the main process spawning test processes.

For pip/certifi, I hacked get-pip.py to add breakpoint() and I followed the code flow. It's not easy to identify which part of pip created the file. It's not just an import. It's an import + call to where() function, done in the module body that I failed to find which module. Maybe requests. It would be nice to be able to postpone when where() is called to workaround the issue: only call it when a filename to cacert.pem is needed.

Reproducer without pip nor certifi, attached run.py and bug.py scripts:

$ ./python run.py 
bug.py: temporary directory: ['tmp1wkab03ycacert.pem']
run.py: temporary directory: ['tmp1wkab03ycacert.pem']

The expected output is an empty list in run.py (parent process). bug.py never calls _tempfile() finally block.

I'm still bisecting the issue. It smells like logging prevents deleting an object somehow.

• bug.py.txt
• run.py.txt

Ok, this problem is quite complicated:

• pip imports requests
• requests imports certifi
• requests calls certifi.where()
• certifi.where() uses importlib.resources to create a temporary file using a context manager.
• logging is imported by pip
• logging calls os.register_at_fork() which keeps the whole logging namespace alive
• somehow, the logging module keeps the whole pip package (!) and all of its sub-modules (!) alive until Python clears os.register_at_fork(): that happens VERY LATE, like the last function call before the last GC collection.
• Python clears os.register_at_fork() callbacks: the context manager finally: block is called as expected....
• Oops, os.remove is None at this point, the call fails.

Fixing importlib.resources is simple: keep a reference to os.remove() in importlib.resources implementation to make sure that we can remove the file very late during Python finalization.

This problem is tricky because:

• importlib.resources only creates a named temporary file if pip is imported from a ZIP file. When pip is installed on disk as a file, the bug is gone.
• the code is valid.
• certifi relies on the garbage collector to gently calls the context manager finalizer which removes the temporary file.
• Python doesn't log any error when a bug occurs very late during Python finalization.

I'm working on a fix for importlib.resources.

related: pypa/pip#10753

Add parameter _remove=os.remove in def _tempfile() and use it instead of os.remove. The leak will gone.

I suppose that the finally code is executed after the module os has been cleared, so os.remove raises an exception. You do not get a report because the standard streams have already been cleared too. Perhaps the logging module creates reference loops and keeps reference to this module dict.

It is the answer of the puzzle. In general, you should not rely on the code executed at the interpreter shutdown stage (except registered with atexit). It may be much worse in PyPy.

I created the PR #93377 to fix importlib.resources.