Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

misconfiguration of c extensions flags on macos arm #100420

Closed
galchapman opened this issue Dec 22, 2022 · 1 comment
Closed

misconfiguration of c extensions flags on macos arm #100420

galchapman opened this issue Dec 22, 2022 · 1 comment
Labels
OS-mac type-bug An unexpected behavior, bug, or error

Comments

@galchapman
Copy link

galchapman commented Dec 22, 2022 •
edited

Bug report

When install trying to install pycryptodome with pip, I got the following message.
Firstly, It's CPython problem and not pycryptodome fault because The problem consisted between a few different packages.
Secondly, According to my findings it's a problem in the setuptools package, and more specifically the problem root is in a value inside the sysconfig package.

$ pip3 install pycryptodome
...
Building wheels for collected packages: pycryptodome
  Building wheel for pycryptodome (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for pycryptodome (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [336 lines of output]
...
      clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -I/opt/homebrew/opt/llvm/include -DHAVE_STDINT_H -DPYCRYPTO_LITTLE_ENDIAN -DSYS_BITS=64 -DLTC_NO_ASM -DHAVE_UINT128 -DHAVE_POSIX_MEMALIGN -Isrc/ -I/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c src/MD2.c -o build/temp.macosx-10.9-universal2-cpython-311/src/MD2.o
      clang -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64 -g -L/opt/homebrew/opt/llvm/lib -I/opt/homebrew/opt/llvm/include build/temp.macosx-10.9-universal2-cpython-311/src/MD2.o -o build/lib.macosx-10.9-universal2-cpython-311/Crypto/Hash/_MD2.abi3.so
      ld: warning: dylib (/opt/homebrew/opt/llvm/lib/libunwind.dylib) was built for newer macOS version (13.0) than being linked (11.0)
      ld: warning: dylib (/opt/homebrew/opt/llvm/lib/libunwind.dylib) was built for newer macOS version (13.0) than being linked (11.0)
      ld: in '/opt/homebrew/opt/llvm/lib/libunwind.dylib', building for macOS-x86_64 but attempting to link with file built for macOS-arm64
      clang-15: error: linker command failed with exit code 1 (use -v to see invocation)
      error: command '/opt/homebrew/opt/llvm/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pycryptodome
Failed to build pycryptodome
ERROR: Could not build wheels for pycryptodome, which is required to install pyproject.toml-based projects

Note that there is both a -arch x86_64 and a -arch arm64, this caused the linker to error because the compiled file and the library file where from different architectures.
The cause for this is that CFLAGS (and LDFLAGS) variables are containing both -arch arm64 and -arch x86_64.

To stop the problem from occurring in the future it is needed to figure out why _sysconfigdata__darwin_darwin.py
contained those flags in the first place (Maybe also remove the -g debug flag).
Because I wasn't able to recreate the problem when compiling cpython from source.
I think it's due to the way the macos package was compiled.

One more possible fix is to add some code that detect if there are multiple -arch flags and remove them.

My environment

  • CPython versions tested on: 3.11.0 (v3.11.0:deaf509e8f, Oct 24 2022, 14:43:23) [Clang 13.0.0 (clang-1300.0.29.30)]
  • Operating system and architecture: Darwin Macbook-Pro 22.2.0 Darwin Kernel Version 22.2.0: Fri Nov 11 02:03:51 PST 2022; root:xnu-8792.61.2~4/RELEASE_ARM64_T6000 arm64

Temporary Fix

A temporary fix to the problem is to set the ARCHFLAGS environment variable.

$ export ARCHFLAGS='-arch arm64'

Or modify the configuration directly, by modify the python package internal files. On my specific machine the file is
/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/_sysconfigdata__darwin_darwin.py
@galchapman galchapman added the type-bug An unexpected behavior, bug, or error label Dec 22, 2022
@galchapman galchapman changed the title misconfiguration of c extensions on macos arm misconfiguration of c extensions flags on macos arm Dec 22, 2022
@ronaldoussoren
Copy link
Contributor

ronaldoussoren commented Dec 22, 2022

Note that this is not a bug in CPython, but an incompatibility in your environment. The installer on www.python.org ships as a "Universal 2" binary supporting both arm64 and x86_64. Homebrew apparently only installs binaries targeting the current machine, which in your case means that it only supports arm64. This is what causes the link error, and that's not something we can fix on the CPython end.

BTW. You could consider to ask the pycryptodome maintainers if they are willing to ship arm64 or (IMHO preferably) universal2 wheels, they appear to use the "cibuildwheel" project to build wheels and project also supports building wheels supporting arm64.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
OS-mac type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ronaldoussoren @galchapman