[WASI] test cases with sys.path.insert(0, os.curdir) fail on WASI #93461

tiran · 2022-06-03T10:33:21Z

Bug report

Test cases that insert os.curdir ('.') into sys.path fail on WASI platforms. I think it is related how wasmtime maps directories into the runtime environment and how it sets up the current working directory. The test passes if I replace '.' with an empty string '' or remove the cached FileFinder instance for '.' with sys.path_importer_cache.pop('.').

 ./test-wasi.sh test_support -m test_forget
0:00:00 Run tests sequentially
0:00:00 [1/1] test_support
test test_support failed -- Traceback (most recent call last):
  File "/Lib/test/test_support.py", line 116, in test_forget
    mod = __import__(TESTFN)
          ^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '@test_66600_tmpæ'

test_support failed (1 error)

== Tests result: FAILURE ==

The culprit seems to be an outdated FileFinder instance: {'.': FileFinder('/.')}. The CWD is '/python-wasm/cpython/builddir/wasi/build/test_python_66600æ' when the test case is executed.

0:00:00 [1/1] test_support
> /Lib/test/test_support.py(114)test_forget()
-> sys.path.insert(0, os.curdir)
(Pdb) p sys.path
['.', '/usr/local/lib/python312.zip', '/Lib', '/build/lib.wasi-wasm32-3.12']
(Pdb) pp sys.path_importer_cache
{'.': FileFinder('/.'),
 '/Lib': FileFinder('/Lib'),
 '/Lib/collections': FileFinder('/Lib/collections'),
 '/Lib/encodings': FileFinder('/Lib/encodings'),
 '/Lib/importlib': FileFinder('/Lib/importlib'),
 '/Lib/re': FileFinder('/Lib/re'),
 '/Lib/test': FileFinder('/Lib/test'),
 '/Lib/test/libregrtest': FileFinder('/Lib/test/libregrtest'),
 '/Lib/test/support': FileFinder('/Lib/test/support'),
 '/Lib/unittest': FileFinder('/Lib/unittest'),
 '/Lib/urllib': FileFinder('/Lib/urllib'),
 '/build/lib.wasi-wasm32-3.12': FileFinder('/build/lib.wasi-wasm32-3.12'),
 '/usr/local/lib/python312.zip': None}
(Pdb) p os.getcwd()
'/python-wasm/cpython/builddir/wasi/build/test_python_66600æ'
(Pdb) n
> /Lib/test/test_support.py(115)test_forget()
-> importlib.invalidate_caches()
(Pdb) n
> /Lib/test/test_support.py(116)test_forget()
-> try:
(Pdb) p sys.path
['.', '.', '/usr/local/lib/python312.zip', '/Lib', '/build/lib.wasi-wasm32-3.12']
(Pdb) pp sys.path_importer_cache
{'.': FileFinder('/.'),
 '/Lib': FileFinder('/Lib'),
 '/Lib/collections': FileFinder('/Lib/collections'),
 '/Lib/encodings': FileFinder('/Lib/encodings'),
 '/Lib/importlib': FileFinder('/Lib/importlib'),
 '/Lib/re': FileFinder('/Lib/re'),
 '/Lib/test': FileFinder('/Lib/test'),
 '/Lib/test/libregrtest': FileFinder('/Lib/test/libregrtest'),
 '/Lib/test/support': FileFinder('/Lib/test/support'),
 '/Lib/unittest': FileFinder('/Lib/unittest'),
 '/Lib/urllib': FileFinder('/Lib/urllib'),
 '/build/lib.wasi-wasm32-3.12': FileFinder('/build/lib.wasi-wasm32-3.12'),
 '/usr/local/lib/python312.zip': None}

A build on Linux has an absolute path to build directory as first sys.path entry.

reproducer

$ podman run --rm -ti -v $(pwd):/python-wasm/cpython:Z quay.io/tiran/cpythonbuild:emsdk3
# ./build-python-build.sh
# ./build-python-wasi.sh
# ./test-wasi.sh test_support -m test_forget

The text was updated successfully, but these errors were encountered:

…ive paths

tiran · 2022-06-09T18:43:51Z

GH-93461 is the simplest solution I could come up with. importlib.invalidate_caches now drops all FileFinder instances if the cache key is not an absolute path.

brettcannon · 2022-06-09T20:08:14Z

Seems like a reasonable solution to the problem.

…ths (GH-93653)

…ive paths (pythonGH-93653) (cherry picked from commit 09243b8) Co-authored-by: Christian Heimes <christian@python.org>

tiran added type-bug 3.11 3.12 labels Jun 3, 2022

tiran assigned brettcannon Jun 3, 2022

brettcannon added the expert-importlib label Jun 3, 2022

tiran added a commit to tiran/cpython that referenced this issue Jun 9, 2022

pythongh-93461: Invalidate sys.path_importer_cache entries with relat…

983288b

…ive paths

ambv pushed a commit that referenced this issue Jun 10, 2022

gh-93461: Invalidate sys.path_importer_cache entries with relative pa…

09243b8

…ths (GH-93653)

[WASI] test cases with sys.path.insert(0, os.curdir) fail on WASI #93461

[WASI] test cases with sys.path.insert(0, os.curdir) fail on WASI #93461

tiran commented Jun 3, 2022

tiran commented Jun 9, 2022

brettcannon commented Jun 9, 2022

[WASI] test cases with sys.path.insert(0, os.curdir) fail on WASI #93461

[WASI] test cases with sys.path.insert(0, os.curdir) fail on WASI #93461

Comments

tiran commented Jun 3, 2022

tiran commented Jun 9, 2022

brettcannon commented Jun 9, 2022