Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move all stdlib tests in the test package #93839

Open
vstinner opened this issue Jun 15, 2022 · 11 comments
Open

Move all stdlib tests in the test package #93839

vstinner opened this issue Jun 15, 2022 · 11 comments
Labels
type-bug

Comments

@vstinner
Copy link
Member

@vstinner vstinner commented Jun 15, 2022

Currently, Python tests are scattered in multiple directories:

  • Lib/test/ (main one)
  • Lib/ctypes/test/
  • Lib/distutils/tests/
  • Lib/idlelib/idle_test/
  • Lib/lib2to3/tests/
  • Lib/tkinter/test/
  • Lib/unittest/test/

idlelib, ctypes, lib2to3, tkinter, unittest and distutils have their tests under their own packages. It causes multiple issues:

  • As a maintainer of Python, I have to look into these directories manually when refactoring "Python tests" (I cannot just look into Lib/test/).
  • Linux distributions have to manually put these directories into a separated packages if they want to make the main "python" package smaller, to put python tests into a separated package.
  • Makefile is more complicated since tests are installed separately, and so these directories should be listed explicitly in the TESTSUBDIRS variable.
  • Tools/scripts/generate_stdlib_module_names.py excludes manually these modules, since sys.stdlib_module_names excludes test modules.

I propose to move ctypes, idlelib, tkinter and unittest tests under Lib/test/.

For distutils and lib2to3, maybe we can just leave them where they are, since both modules are deprecated. distutils should be removed in Python 3.12. lib2to3 is scheduled for removal in Python 3.13.

Do you think that it's worth it to move these directories?
@vstinner vstinner added the type-bug label Jun 15, 2022
@vstinner
Copy link
Member Author

@vstinner vstinner commented Jun 15, 2022

It seems like the Windows installer only treat Lib/test/ separately: see Tools/msi/test/test.wixproj. I would be nice to also exclude the other test directories to make Python installation smaller on Windows.

$ du -sh ctypes/test/ distutils/tests/ idlelib/idle_test/ lib2to3/tests/ tkinter/test/ unittest/test/
348K	ctypes/test/
336K	distutils/tests/
624K	idlelib/idle_test/
440K	lib2to3/tests/
284K	tkinter/test/
700K	unittest/test/

The total is about 2 732 KB.

On Fedora 36, the Python 3.10 tests (python3-test package) take around 54 MB in total (it includes idle_test, ctypes tests, etc.).

$ sudo dnf install python3-test
(...)
Total download size: 9.7 M
Installed size: 54 M

@rhettinger
Copy link
Contributor

@rhettinger rhettinger commented Jun 15, 2022

Do you think that it's worth it to move these directories?

Yes, yes, yes!

@zware
Copy link
Member

@zware zware commented Jun 15, 2022

This appears to duplicate #54781.

I'm still in favor :)

vstinner added a commit to vstinner/cpython that referenced this issue Jun 20, 2022
@vstinner
pythongh-93839: Move Lib/ctypes/test/ to Lib/test/test_ctypes/
dd050ba 
* Remove Lib/test/test_ctypes.py
vstinner added a commit to vstinner/cpython that referenced this issue Jun 20, 2022
@vstinner
pythongh-93839: Move Lib/unttest/test/ to Lib/test/test_unittest/
0216f69 
* Remove Lib/test/test_unittest.py
@vstinner
Copy link
Member Author

@vstinner vstinner commented Jun 20, 2022

I started with:

@vstinner vstinner mentioned this issue Jun 20, 2022
Open
@terryjreedy
Copy link
Member

@terryjreedy terryjreedy commented Jun 21, 2022

Please leave idlelib/idle_test alone. Following gives some of reasons:
#54781 (comment)
There are also several references to files in this directory in idlelib and idlelib.test_idle files.

@merwok
Copy link
Member

@merwok merwok commented Jun 21, 2022 •
edited

Could you exclude idle and distutils from your list, and attach the new PRs to the other ticket?

vstinner added a commit that referenced this issue Jun 21, 2022
@vstinner
gh-93839: Move Lib/ctypes/test/ to Lib/test/test_ctypes/ (#94041)
d82e0bf 
* Move Lib/ctypes/test/ to Lib/test/test_ctypes/
* Remove Lib/test/test_ctypes.py
* Update imports and build system.
vstinner added a commit that referenced this issue Jun 21, 2022
@vstinner
gh-93839: Move Lib/unttest/test/ to Lib/test/test_unittest/ (#94043)
c735d54 
* Move Lib/unittest/test/ to Lib/test/test_unittest/
* Remove Lib/test/test_unittest.py
* Replace unittest.test with test.test_unittest
* Remove unittest.load_tests()
* Rewrite unittest __init__.py and __main__.py
* Update build system, CODEOWNERS, and wasm_assets.py
@vstinner
Copy link
Member Author

@vstinner vstinner commented Jun 21, 2022

I merged my ctypes and unittest PRs. Ok, let's continue in issue #54781.

@tiran
Copy link
Member

@tiran tiran commented Jun 21, 2022

PR #94043 broke testing on all WebAssembly platforms.

@tiran tiran reopened this Jun 21, 2022
@vstinner
Copy link
Member Author

@vstinner vstinner commented Jun 21, 2022

PR #94043 broke testing on all WebAssembly platforms.

Please comment #54781. This issue is a duplicate.

@tiran
Copy link
Member

@tiran tiran commented Jun 21, 2022

Commit #94043 used this ticket as reference.

@tiran
Copy link
Member

@tiran tiran commented Jun 21, 2022

There are actually two problems:

  • Lib/test/test_unittest/init.py fails when a submodule raises SkipTest. This was supported before.
  • Re-running uses wrong interpreter to re-run the test, reported as #94052
# wasmtime run --mapdir .::../../ -- python.wasm -m test -v test_unittest
== CPython 3.12.0a0 (heads/main-dirty:c735d545343, Jun 21 2022, 09:30:50) [Clang 14.0.4 (https://github.com/llvm/llvm-project 29f1039a7285a5c3a9c353d05414
== wasi-0.0.0-wasm32-32bit little-endian
== cwd: /python-wasm/cpython/builddir/wasi/build/test_python_859108æ
== CPU count: 1
== encodings: locale=UTF-8, FS=utf-8
0:00:00 Run tests sequentially
0:00:00 [1/1] test_unittest
Failed to call load_tests:
Traceback (most recent call last):
  File "/Lib/unittest/loader.py", line 130, in loadTestsFromModule
    return load_tests(self, tests, pattern)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Lib/test/test_unittest/testmock/__init__.py", line 14, in load_tests
    __import__(modname)
    ^^^^^^^^^^^^^^^^^^^
  File "/Lib/test/test_unittest/testmock/testasync.py", line 9, in <module>
    support.requires_working_socket(module=True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Lib/test/support/__init__.py", line 543, in requires_working_socket
    raise unittest.SkipTest(msg)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
unittest.case.SkipTest: requires socket support

test test_unittest crashed -- Traceback (most recent call last):
  File "/Lib/test/libregrtest/runtest.py", line 360, in _runtest_inner
    refleak = _runtest_inner2(ns, test_name)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Lib/test/libregrtest/runtest.py", line 317, in _runtest_inner2
    test_runner()
    ^^^^^^^^^^^^^
  File "/Lib/test/libregrtest/runtest.py", line 280, in _test_module
    raise Exception("errors while loading tests")
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: errors while loading tests

test_unittest failed (uncaught exception)

== Tests result: FAILURE ==

1 test failed:
    test_unittest

Total duration: 132 ms
Tests result: FAILURE

tiran added a commit to tiran/cpython that referenced this issue Jun 21, 2022
@tiran
pythongh-93839: Use load_package_tests() for testmock
4b3670c 
Fixes failing tests on WebAssembly platforms.
miss-islington pushed a commit that referenced this issue Jun 21, 2022
@tiran
gh-93839: Use load_package_tests() for testmock (GH-94055)
50ebd72 
Fixes failing tests on WebAssembly platforms.

Automerge-Triggered-By: GH:tiran
@vstinner vstinner mentioned this issue Jun 21, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type-bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@vstinner @tiran @merwok @rhettinger @zware @terryjreedy