Right now, test_venv fails without distutils, because pip fails on importing the distutils module in pip/_internal/locations/_distutils.py

I reported this issue to pip: pypa/pip#11103

The distutils package was deprecated by the issue #85454.

PEP 632 "Deprecate distutils module": https://peps.python.org/pep-0632/

That seems like a good plan. Thanks.

My fear with renaming it is that it lingers indefinitely, and diverging in subtle ways from pypa/distutils. Why not address the dependencies and fully remove the functionality?

Regardless, I agree renaming is preferable to not removing the module at all.

Why not address the dependencies and fully remove the functionality?

Someone has to modify Python to no longer depend on distutils. The issue description lists code using it. It doesn't seem trivial.

Would it be reasonable to keep it around in the repo while we don't drop the dependency but not ship it as part of Python installations?

Would it be reasonable to keep it around in the repo while we don't drop the dependency but not ship it as part of Python installations?

With my PR, the Lib/_distutils/ directory is not installed by make install.

Do note that we have buildbots that run from an installed Python. The tests would need to also be modified to fail gracefully
(or skip) when _distutils is not available.

Do note that we have buildbots that run from an installed Python. The tests would need to also be modified to fail gracefully (or skip) when _distutils is not available.

I propose skipping tests relying on _distutils if _distutils is not available.

I wrote two PRs to avoid the deprecated distutils module in two tests:

• gh-92584: test_cppext uses setuptools #92639: test_cppext uses setuptools
• gh-92584: test_decimal uses shutil.which() #92640: test_decimal uses shutil.which() -- IMO an uncontroversial change

Hi @vstinner

I think #92639 is causing a test failure, details here #92820

PS: Commented here as I thought those changes are related to this PEP.

Marking as 3.12.0a1 release blocker. We want to have public-facing distutils gone in the first alpha, to have as much time as possible to deal with the fallout.

Actually we can get rid of distutils entirely. I have a working PR that gets rid of our setup.py and builds all extension modules with configure and make.

For the curious, the PR that @tiran is refering to is #94474

Marking as 3.12.0a1 release blocker. We want to have public-facing distutils gone in the first alpha, to have as much time as possible to deal with the fallout.

It seems that 3.12.0a1 was released with distutils. Was that intentional? cc. @Yhg1s

I was going to ping the issue during the core dev sprint and then intentionally ignore it for the alpha 1 release (using it more as a practice run rather than trying to get everything right) -- but then I accidentally ignored it instead 😬. This should just get done for alpha 2.

Not sure why this got put on me. If nobody else wants to do it, I can get to it eventually, but I definitely don't want to prevent anyone else from taking the honours.

Since May, the situation evolved:

• Python setup.py has been removed: distutils is no longer needed to build Python
• pip has been fixed: it now works if there is no distutils package
• test_cppext now creates a virtual environment to build the C++ with setuptools
• test_decimal now uses shutil.which(), rather than distutils.spawn.find_executable()

Thanks to that, I was able to create PR #99061. This PR removes the distutils package. It always skips test_check_c_globals and test_peg_generator since they use distutils. I prefer to remove distutils now, and fix these tests later, rather than missing the next Python 3.12 alpha release.

Python 3.12 alpha 2 was released without distutils.

test_check_c_globals and test_peg_generator are currently skipped because of this removal. These tests should be rewritten without distutils.

@ericsnowcurrently: FYI test_check_c_globals is simply skipped in the main branch, since the distutils package was removed. This test should be modified to use something else to build C code, like setuptools, or simply create a virtual environment to get the distutils packages from setuptools: trick used by test_cppext.

@pablogsal @lysnikolaou: test_peg_generator has the same issue, the whole test is also simply skipped in the main branch.

You may create new issues for these two currently skipped tests. As you want.

test_peg_generator has the same issue, the whole test is also simply skipped in the main branch.

What do you mean is skipped? I don't feel comfortable with just skipping this test without thinking how to fix it. I think this aspect should have taken into account when merging the original PR.

@vstinner I am sure you understand that is not good to just do something without thinking how to fix the problems you are creating and then throwing the issue to someone else.

To me it looks like peg_generator and check_c_globals are in the same situation as the rest of the ecosystem will be with 3.12: they broke and the maintainers need to fix them.

IMO, PEP 632 underestimated the amount of work needed, and failed to mention that no one was really interested in actually implementing it. (AFAIK we have an implicit assumption that a PEP author will manage the work if the PEP is accepted, we might want to add that to the PEP process docs.)

Thanks for picking up the slack at the last moment, Victor.

To me it looks like peg_generator and check_c_globals are in the same situation as the rest of the ecosystem will be with 3.12: they broke and the maintainers need to fix them.

IMO, PEP 632 underestimated the amount of work needed, and failed to mention that no one was really interested in actually implementing it. (AFAIK we have an implicit assumption that a PEP author will manage the work if the PEP is accepted, we might want to add that to the PEP process docs.)

Thanks for picking up the slack at the last moment, Victor.

I was under the impression that the plan was to have distutils outside the standard library but available until we figure out what to do with the tests that depend on it. The PEP suggests this as well:

The final dependency on distutils is CPython itself, which uses it to build native extension modules in the standard library (except on Windows). Because this is a CPython build-time dependency, it is possible to continue to use distutils for this specific case without it being part of the standard library.

Using alternatives within the test suite of CPython differs from using alternatives outside because installing packages like pip and setuptools is not that straightforward.

If this was the plan all along and I was under the wrong impression I apologize for the confusion but I still think having tests deactivated in main is just dangerous and we should have though about this first.

Reverting the removal would be some churn, but there are other options:

• should be quick to make the testsuite install setuptools, import distutils from setuptools and adjust code if needed
• a bit longer to replace the distutils code in tests with setuptools code, but hopefully not too hard (Extension class should have the same interface, various compiler functions should exist too even if renamed, etc)

FWIW, test_check_c_globals basically covers Tools/c-analyzer/check-c-globals.py. We shouldn't be losing any coverage of CPython with that test disabled. Plus, there are a few things that need to be fixed with that test that I'd been putting off. It having been disabled is an extra motivator. 😄

That said, I agree it is surprising that we merged something requiring any tests to be disabled and then fixed after the fact by someone other than the author of that PR. At the very least there should have been some discussion beforehand with the maintainers of the to-be-disabled tests and a clear agreement on how to proceed, all before the PR was merged. While what happened with distutils may seem relatively benign and innocuous, it's a risky precedent to set.

AFAIK we have an implicit assumption that a PEP author will manage the work if the PEP is accepted, we might want to add that to the PEP process docs.

There was an explicit agreement that the author of this PEP (me) was just putting it in writing and wasn't obligated to do all the work. But feel free to make that an explicit requirement - it just means that I won't volunteer for PEPs that we all want but nobody is willing to do (and will simply withdraw from maintaining the module that we all want to remove) ;)

But yes, I agree that breaking/disabling the tests is not right. Moving distutils into Tools and pointing the tests at it there would have been fine - they'd have eventually broken with bitrot rather than immediately, but that was happening anyway.