Description
Following the discussion her pypa/wheel#660 (comment) - images relesed today started to break CI - Airflow, but also other people have similar issue.
I explained it in the issue but let me copy it here as well:
Example failure: https://github.com/apache/airflow/actions/runs/14347343802/job/40220981486#step:6:809
6 13.37 Building wheels for collected packages: pyspark
docker-library/official-images#6 13.37 Building wheel for pyspark (setup.py): started
docker-library/official-images#6 14.11 Building wheel for pyspark (setup.py): finished with status 'error'
docker-library/official-images#6 14.11 error: subprocess-exited-with-error
docker-library/official-images#6 14.11
docker-library/official-images#6 14.11 × python setup.py bdist_wheel did not run successfully.
docker-library/official-images#6 14.11 │ exit code: 1
docker-library/official-images#6 14.11 ╰─> [8 lines of output]
docker-library/official-images#6 14.11 /home/airflow/.local/lib/python3.9/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
docker-library/official-images#6 14.11 warnings.warn(
docker-library/official-images#6 14.11 usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
docker-library/official-images#6 14.11 or: setup.py --help [cmd1 cmd2 ...]
docker-library/official-images#6 14.11 or: setup.py --help-commands
docker-library/official-images#6 14.11 or: setup.py cmd --help
docker-library/official-images#6 14.11
docker-library/official-images#6 14.11 error: invalid command 'bdist_wheel'
docker-library/official-images#6 14.11 [end of output]
docker-library/official-images#6 14.11
docker-library/official-images#6 14.11 note: This error originates from a subprocess, and is likely not a problem with pip.
docker-library/official-images#6 14.11 ERROR: Failed building wheel for pyspark
docker-library/official-images#6 14.12 Running setup.py clean for pyspark
And after some (non-obvious investigation I found out what the root cause is and that's somethign you might want to talk to the maintainers of official Python images in dockerhub, because probably a lot of people using them will have similar issue after 0.46+ wheel that does not contain bdist_wheel any more and rely on setuptools > 70.1.
The problem is that some - latest - official python images available in dockerhub are causing this problem to happen.
If you look - for example - at latest Python debian bookworm slim images you will see this:
[jarek:~/code/airflow] main ± docker run -it python:3.9-slim-bookworm bash
root@fb321ef4723e:/# pip list
Package Version
---------- -------
pip 23.0.1
setuptools 58.1.0
wheel 0.45.1
[notice] A new release of pip is available: 23.0.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
root@fb321ef4723e:/# exit
exit
[jarek:~/code/airflow] main 14s ± docker run -it python:3.10-slim-bookworm bash
root@2234dee7aba0:/# pip list
Package Version
---------- -------
packaging 24.2
pip 23.0.1
setuptools 65.5.1
wheel 0.46.1
[notice] A new release of pip is available: 23.0.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
root@2234dee7aba0:/# exit
exit
[jarek:~/code/airflow] main 25s ± docker run -it python:3.11-slim-bookworm bash
root@b62343d129a6:/# pip list
Package Version
---------- -------
packaging 24.2
pip 24.0
setuptools 65.5.1
wheel 0.46.1
[jarek:~/code/airflow] main 1m5s ± docker run -it python:3.12-slim-bookworm bash
root@411574e67a7e:/# pip list
Package Version
------- -------
pip 25.0.1
root@411574e67a7e:/#
As you can see - default version of setuptools in all those images causes problem:
- Python 3.9 -> both wheel and setuptools are old, but if you just upgrade wheel but do not upgrade setuptools, sdist installations will fail
- Python 3.10, 3.11 -> wheel is already updated, but setuptools is < 70.1 - which mean that by default installing somethign from sdist wil fail there - even if you do not touch setuptools and wheel
- Python 3.12 -> all good, no setuptools, no wheel, newest versions of those will be installed
I am going to upgrade setuptools and wheels to latest versions in our images, so we will solve the problem in Airflow, but I think currently released "official images" need to be rebuild and latest setuptools should be installed in all of them.