Right now, test_venv fails without distutils, because pip fails on importing the distutils module in pip/_internal/locations/_distutils.py

I reported this issue to pip: pypa/pip#11103

The distutils package was deprecated by the issue #85454.

PEP 632 "Deprecate distutils module": https://peps.python.org/pep-0632/

That seems like a good plan. Thanks.

My fear with renaming it is that it lingers indefinitely, and diverging in subtle ways from pypa/distutils. Why not address the dependencies and fully remove the functionality?

Regardless, I agree renaming is preferable to not removing the module at all.

Why not address the dependencies and fully remove the functionality?

Someone has to modify Python to no longer depend on distutils. The issue description lists code using it. It doesn't seem trivial.

Would it be reasonable to keep it around in the repo while we don't drop the dependency but not ship it as part of Python installations?

Would it be reasonable to keep it around in the repo while we don't drop the dependency but not ship it as part of Python installations?

With my PR, the Lib/_distutils/ directory is not installed by make install.

Do note that we have buildbots that run from an installed Python. The tests would need to also be modified to fail gracefully
(or skip) when _distutils is not available.

Do note that we have buildbots that run from an installed Python. The tests would need to also be modified to fail gracefully (or skip) when _distutils is not available.

I propose skipping tests relying on _distutils if _distutils is not available.

I wrote two PRs to avoid the deprecated distutils module in two tests:

• gh-92584: test_cppext uses setuptools #92639: test_cppext uses setuptools
• gh-92584: test_decimal uses shutil.which() #92640: test_decimal uses shutil.which() -- IMO an uncontroversial change

Hi @vstinner

I think #92639 is causing a test failure, details here #92820

PS: Commented here as I thought those changes are related to this PEP.

Marking as 3.12.0a1 release blocker. We want to have public-facing distutils gone in the first alpha, to have as much time as possible to deal with the fallout.

Actually we can get rid of distutils entirely. I have a working PR that gets rid of our setup.py and builds all extension modules with configure and make.

For the curious, the PR that @tiran is refering to is #94474

Python 3.12 alpha 2 was released without distutils.

test_check_c_globals and test_peg_generator are currently skipped because of this removal. These tests should be rewritten without distutils.

@ericsnowcurrently: FYI test_check_c_globals is simply skipped in the main branch, since the distutils package was removed. This test should be modified to use something else to build C code, like setuptools, or simply create a virtual environment to get the distutils packages from setuptools: trick used by test_cppext.

@pablogsal @lysnikolaou: test_peg_generator has the same issue, the whole test is also simply skipped in the main branch.

You may create new issues for these two currently skipped tests. As you want.

test_peg_generator has the same issue, the whole test is also simply skipped in the main branch.

What do you mean is skipped? I don't feel comfortable with just skipping this test without thinking how to fix it. I think this aspect should have taken into account when merging the original PR.

@vstinner I am sure you understand that is not good to just do something without thinking how to fix the problems you are creating and then throwing the issue to someone else.

To me it looks like peg_generator and check_c_globals are in the same situation as the rest of the ecosystem will be with 3.12: they broke and the maintainers need to fix them.

IMO, PEP 632 underestimated the amount of work needed, and failed to mention that no one was really interested in actually implementing it. (AFAIK we have an implicit assumption that a PEP author will manage the work if the PEP is accepted, we might want to add that to the PEP process docs.)

Thanks for picking up the slack at the last moment, Victor.

To me it looks like peg_generator and check_c_globals are in the same situation as the rest of the ecosystem will be with 3.12: they broke and the maintainers need to fix them.

IMO, PEP 632 underestimated the amount of work needed, and failed to mention that no one was really interested in actually implementing it. (AFAIK we have an implicit assumption that a PEP author will manage the work if the PEP is accepted, we might want to add that to the PEP process docs.)

Thanks for picking up the slack at the last moment, Victor.

I was under the impression that the plan was to have distutils outside the standard library but available until we figure out what to do with the tests that depend on it. The PEP suggests this as well:

The final dependency on distutils is CPython itself, which uses it to build native extension modules in the standard library (except on Windows). Because this is a CPython build-time dependency, it is possible to continue to use distutils for this specific case without it being part of the standard library.

Using alternatives within the test suite of CPython differs from using alternatives outside because installing packages like pip and setuptools is not that straightforward.

If this was the plan all along and I was under the wrong impression I apologize for the confusion but I still think having tests deactivated in main is just dangerous and we should have though about this first.

Reverting the removal would be some churn, but there are other options:

• should be quick to make the testsuite install setuptools, import distutils from setuptools and adjust code if needed
• a bit longer to replace the distutils code in tests with setuptools code, but hopefully not too hard (Extension class should have the same interface, various compiler functions should exist too even if renamed, etc)

FWIW, test_check_c_globals basically covers Tools/c-analyzer/check-c-globals.py. We shouldn't be losing any coverage of CPython with that test disabled. Plus, there are a few things that need to be fixed with that test that I'd been putting off. It having been disabled is an extra motivator. 😄

That said, I agree it is surprising that we merged something requiring any tests to be disabled and then fixed after the fact by someone other than the author of that PR. At the very least there should have been some discussion beforehand with the maintainers of the to-be-disabled tests and a clear agreement on how to proceed, all before the PR was merged. While what happened with distutils may seem relatively benign and innocuous, it's a risky precedent to set.

AFAIK we have an implicit assumption that a PEP author will manage the work if the PEP is accepted, we might want to add that to the PEP process docs.

There was an explicit agreement that the author of this PEP (me) was just putting it in writing and wasn't obligated to do all the work. But feel free to make that an explicit requirement - it just means that I won't volunteer for PEPs that we all want but nobody is willing to do (and will simply withdraw from maintaining the module that we all want to remove) ;)

But yes, I agree that breaking/disabling the tests is not right. Moving distutils into Tools and pointing the tests at it there would have been fine - they'd have eventually broken with bitrot rather than immediately, but that was happening anyway.

Is distutils required to build the docs? I see uses of it in Doc/includes/setup.py and Doc/includes/test.py.

Also, I'm taking care of gh-100237 (c-analyzer) via gh-102506. However, separately I'm vendoring a subset of the [reverted] distutils code under Tools/c-analyzer, in gh-102505. However, we also need distutils for Tools/peg_generator/pegen/build.py (and possibly for the docs). So it may make sense to vendor distutils in one common place (or use some other mechanism, e.g. pip install in a venv, that the reliant tools can share).

Doc builds already rely on a venv with packages from PyPI, so making them use/assume setuptools should be okay.

If we were to vendor it, I'd prefer we just vendor https://github.com/pypa/distutils directly under Tools.

Is distutils required to build the docs? I see uses of it in Doc/includes/setup.py and Doc/includes/test.py.

As far as I can tell via git archaeology, those two files have never actually been used and can just be removed, along with the others that go with them. Currently testing that theory by ripping them out and seeing if things fail :)

along with the others that go with them.

Turns out that the other files (custom*.c, sublist.c) are referenced by Doc/extending/newtypes_tutorial.rst, which itself needs a distutils-ectomy. The setup.py and test.py files were apparently meant for building and testing those modules, but that was never made automatic and they are currently quite broken.

@pablogsal:

What do you mean is skipped? I don't feel comfortable with just skipping this test without thinking how to fix it. I think this aspect should have taken into account when merging the original PR.

My PR clearly says that it skips 2 tests. It was approved by @zware.

As I wrote previously, my plan was to run test_check_c_globals and test_peg_generator in a venv so distutils is provided by setuptools installed in the venv. In the meanwhile, I got catched by more urgent issues and I didn't find time to update these two tests.

Removing distutils is a significant change in Python 3.12 and it's better to do it as soon as possible to give users more time to prepare their project for this change planned for a long time (PEP 632) before Python 3.12 final release.

Just in Fedora, there are already at least 250 packages known to be affected by this change!

Fixing the whole ecosystem for this change is tricky, there are many glitches, like this setuptools bug.

@vstinner I am sure you understand that is not good to just do something without thinking how to fix the problems you are creating and then throwing the issue to someone else.

I got a burnout for 3 months. I warned you and Eric before leaving so someone else can handle the issue.

There are still 6 months until Python 3.12 final version to decide how to fix the issue: add again distutils or update the 2 tests.

Is there a reason or a technical issue which prevents to run whole test_check_c_globals and test_peg_generator files in a separated process running in a venv? We already have many tests which fully run in a child process for various reasons (usually to be "isolated", run in a controlled environment, and fully reset side effects when tests complete). Just two examples: test_atexit and test_eintr.

This is still marked as a release blocker, so I decided to take a look. I am not familiar with the distutils situation, but I can see the following concrete issues:

• Lib/test/test_peg_generator/__init__.py is still disabled, and various code in Tools/peg_generator/pegen/build.py uses distutils. Perhaps it could be switched to Tools/c-analyzer/distutils, which is a bundled copy of distutils used by Tools/c-analyzer, but there is no PR for that yet.
• Some of the documentation still refers to distutils (e.g., Doc/library/zipapp.rst).
• At some point, perhaps the removal of distutils is too disruptive for the ecosystem and we should consider delaying it. I don't know if we're at that point. Victor above mentioned that 250 Fedora packages are affected.

We can probably afford to release the next beta without further changes here, but definitely the docs should be updated before 3.12 final.