bpo-43112: detect musl as a separate SOABI #24502
Conversation
musl libc and gnu libc are not ABI compatible so we need set different SOABI for musl and not simply assume that all linux is linux-gnu. Replace linux-gnu with the detected os for the build from config.guess for linux-musl*.
You also have to re-generate configure script with autoreconf and commit the new configure file.
|
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
|
I have made the requested changes; please review again |
|
Thanks for making the requested changes! @tiran: please review the changes made to this pull request. |
|
Looks ok to me. Let's wait for input on https://discuss.python.org/t/pre-pep-platform-tag-for-linux-distributions-using-musl/7165 |
|
Unfortunately I don’t have the required autoconf knowledge to understand what’s going on here :( |
|
I'd remove the runstatedir changes, as those are generated due to either a later version of autoconf, or from one where that change was backported from upstream. However when someone makes another change to the configure script it can be removed again depending on the autoconf capabilities of the developer's machine. See also: https://bugs.python.org/issue40216 |
The magic happens here: The shell code roughly translates to This changes the SOABI from |
|
Thanks for the explaination. Where does Am I correct to assume this needs to be done in the post-processing instead of |
|
We could modify conftest.c, but it would take much more effort. We would have add a dozen ifdef checks for musl. GCC uses a similar approach like this patch in https://github.com/gcc-mirror/gcc/blob/master/gcc/config/t-musl . |
|
Thanks. I still can’t say I really know how the code works, but kind of understand now what’s going on, and the approach makes sense to me. |
|
I think we may also need something like this: diff --git a/Lib/test/test_sysconfig.py b/Lib/test/test_sysconfig.py
index e279957e26..83ce74aa6f 100644
--- a/Lib/test/test_sysconfig.py
+++ b/Lib/test/test_sysconfig.py
@@ -387,10 +387,13 @@ def test_triplet_in_ext_suffix(self):
if re.match('(i[3-6]86|x86_64)$', machine):
if ctypes.sizeof(ctypes.c_char_p()) == 4:
self.assertTrue(suffix.endswith('i386-linux-gnu.so') or
- suffix.endswith('x86_64-linux-gnux32.so'),
+ suffix.endswith('x86_64-linux-gnux32.so') or
+ suffix.endswith('i386-linux-musl.so'),
suffix)
else: # 8 byte pointer size
- self.assertTrue(suffix.endswith('x86_64-linux-gnu.so'), suffix)
+ self.assertTrue(suffix.endswith('x86_64-linux-gnu.so') or
+ suffix.endswith('x86_64-linux-musl.so'),
+ suffix)
@unittest.skipUnless(sys.platform == 'darwin', 'OS X-specific test')
def test_osx_ext_suffix(self): |
|
This PR is stale because it has been open for 30 days with no activity. |
|
@ncopa I believe you are correct that those tests need to be updated |
|
|
|
|
|
|
|
|
|
|
|
|
| self.assertTrue(suffix.endswith(expected_suffixes), | ||
| f'unexpected suffix {suffix!r}') |
There was a problem hiding this comment.
I'm totally a neophyte here, but I think the buildbot failures might be due to this? This appears like it should've been indented to be inside the if re.match(... block, ala:
| self.assertTrue(suffix.endswith(expected_suffixes), | |
| f'unexpected suffix {suffix!r}') | |
| self.assertTrue(suffix.endswith(expected_suffixes), | |
| f'unexpected suffix {suffix!r}') |
There was a problem hiding this comment.
ah, that would be it, only one level of dedent was expected!
Accidentally introduced by pythonGH-24502
|
|
Thanks tianon, created #31001 |
|
Accidentally introduced by GH-24502
| @@ -5351,6 +5351,11 @@ EOF | |||
|
|
|||
| if $CPP $CPPFLAGS conftest.c >conftest.out 2>/dev/null; then | |||
| PLATFORM_TRIPLET=`grep -v '^#' conftest.out | grep -v '^ *$' | tr -d ' '` | |||
| case "$build_os" in | |||
There was a problem hiding this comment.
If I'm not mistaken, PLATFORM_TRIPLET should refer to the target platform on which cpython will run. If that is the case, the musl libc vs. glibc decision should be based on $host_os rather than $build_os as the former is based on autoconf's AC_CANONICAL_HOST macro1 which refers to the target platform that might differ from build platform in case of cross-compilation.
For example, the decision based on $build_os leads to
cpython 3.9.13 build failing2 when cross-compiling on AMD64 Linux with glibc for mpc8548 Linux (OpenWrt) with musl. (build runs correctly if $host_os is used)
Footnotes
-
https://www.gnu.org/software/autoconf/manual/autoconf-2.68/html_node/Canonicalizing.html
↩ -
"internal configure error for the platform triplet, please file a bug report"
↩
There was a problem hiding this comment.
Yes, the code is wrong. In fact the entire block should be revisited. We really should check for __GLIBC__ and use a heuristic to detect musl. Could you please open a new issue for the problem?
There was a problem hiding this comment.
No problem, but there's #87278 already. Is yet another issue needed?
Patch 030: Backported from Python main branch[^1] for Python to distinguish between glibc and musl libc SOABI. Patch 131: Changes PLATFORM_TRIPLET -gnu/-musl suffix detection (performed by the backported patch) to be based on the target OS instead of the building OS. See included patches for more detailed descriptions. Specifically this fixes cross-compilation for mpc8548 CPUs with SPE instructions[^2] enabled. [^1]: merged to python:main as python/cpython#24502 'bpo-43112: detect musl as a separate SOABI' [^2]: https://www.nxp.com/docs/en/reference-manual/SPEPEM.pdf Co-authored-by: Pali Rohár <pali@kernel.org> Signed-off-by: Šimon Bořek <simon.borek@nic.cz>
Patch 030: Backported from Python main branch[^1] for Python to distinguish between glibc and musl libc SOABI. Patch 131: Changes PLATFORM_TRIPLET -gnu/-musl suffix detection (performed by the backported patch) to be based on the target OS instead of the building OS. See included patches for more detailed descriptions. Specifically this fixes cross-compilation for mpc8548 CPUs with SPE instructions[^2] enabled. [^1]: merged to python:main as python/cpython#24502 'bpo-43112: detect musl as a separate SOABI' [^2]: https://www.nxp.com/docs/en/reference-manual/SPEPEM.pdf Co-authored-by: Pali Rohár <pali@kernel.org> Signed-off-by: Šimon Bořek <simon.borek@nic.cz>
Patch 030: Backported from Python main branch[^1] for Python to distinguish between glibc and musl libc SOABI. Patch 131: Changes PLATFORM_TRIPLET -gnu/-musl suffix detection (performed by the backported patch) to be based on the target OS instead of the building OS. See included patches for more detailed descriptions. Specifically this fixes cross-compilation for mpc8548 CPUs with SPE instructions[^2] enabled. [^1]: merged to python:main as python/cpython#24502 'bpo-43112: detect musl as a separate SOABI' [^2]: https://www.nxp.com/docs/en/reference-manual/SPEPEM.pdf Co-authored-by: Pali Rohár <pali@kernel.org> Signed-off-by: Šimon Bořek <simon.borek@nic.cz>
musl libc and gnu libc are not ABI compatible so we need set different
SOABI for musl and not simply assume that all linux is linux-gnu.
Replace linux-gnu with the detected os for the build from config.guess
for linux-musl*.
https://bugs.python.org/issue43112
Automerge-Triggered-By: GH:brettcannon