bpo-37916: distutils: allow overriding of the RANLIB command on macOS (darwin) #15387
Conversation
This is required to be able to crosscompile numpy (and probably some other packages too) on a macOS host where the system ranlib does not understand ELF files.
|
Hello, and thanks for your contribution! I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA). Unfortunately we couldn't find an account corresponding to your GitHub username on bugs.python.org (b.p.o) to verify you have signed the CLA (this might be simply due to a missing "GitHub Name" entry in your b.p.o account settings). This is necessary for legal reasons before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue. You can check yourself to see if the CLA has been received. Thanks again for your contribution, we look forward to reviewing it! |
| @@ -232,7 +234,8 @@ def customize_compiler(compiler): | |||
| compiler_cxx=cxx, | |||
| linker_so=ldshared, | |||
| linker_exe=cc, | |||
| archiver=archiver) | |||
| archiver=archiver, | |||
| ranlib=ranlib) | |||
There was a problem hiding this comment.
ranlib should be added here only if it is in compiler.executables. Otherwise this would be adding ranlib to other OS as well.
There was a problem hiding this comment.
Hey, thanks for the feedback. I may be misunderstanding something but this change would add ranlib only if it was explicitly passed as an environment variable (or set in the Python Makefile), right? If I am misunderstanding something I would love if you could sketch the code change since I know next to nothing about the distutils codebase (apart from what I have learned debugging this issue).
I think this change is really OS independent – it's just that it is easier to trigger the bug on macOS since it breaks many Linux-centric assumptions.
|
Hi, is there anything I could do to help get this merged? |
|
Thanks for submitting the PR. I just had time to take a quick look at it and that raised some questions. First, to be complete, Lib/distutils/test/test_sysconfig.py should be modified to test this change. Second, the PR should be renamed since its effects are not limited to macOS. But, most importantly, I wonder if the root cause of seeing problems with cross-compiling on macOS is some code in Lib/distutils/unixccompiler.py which is where all these customizations are used. 005dbb2 way back in 2002 added: with the explanation "on MacOSX/Darwin, use ranlib when building static libs.". So it appears that only on macOS is the use of ranlib forced. This was in the very early days of macOS so perhaps this is no longer needed. There are comments elsewhere associated with libtool that suggest ranlib might be needed for universal builds on macOS. I don't have a lot of experience with static libs on macOS so I really don't know whether this is still an issue or not. But before further complicating Distutils, we should figure out whether this forced use of ranlib is still necessary. I also note that the use of ranlib has been removed from the building of Python itself by bpo-31625 so that might be a clue. (@ronaldoussoren. any insights?) |
|
Please open a new issue: https://bugs.python.org/issue31625 is closed. |
|
The ticket was closed with an explanation that the code moved to https://github.com/pypa/distutils/ and issues should be reported there. |
On a macOS hosts the system ranlib does not understand ELF
files so using the "ranlib" command causes errors during cross-compilations.
The simplest way to fix it is to pass the RANLIB parameter provided to setup.py through to the distutils compiler machinery. This is analogous to the way the C/C++ cross-compiler is configured.
This change was required to proceed with crosscompiling numpy. It should help with other packages too (if they use distutils and need ranlib).
https://bugs.python.org/issue37916