@tiran Seems that having the pkg-config macros makes something fail on macos for openssl Do you know why this may be?

@ned-deily Any idea what could be happening in macOs?

I'm looking at it.

I did verify that the PR works in my environment as expected.

Wow, how has this been working so far!?!

> Run /usr/local/opt/openssl/bin/openssl version
/Users/runner/work/_temp/e33f99c9-59f2-4453-8b9b-0804b4843aa2.sh: line 1: /usr/local/opt/openssl/bin/openssl: No such file or directory

> ls: /usr/local/opt/openssl: No such file or directory

Well, I just looked at a run from a prior PR and it appears that the CI job hasn't been finding an OpenSSL and skipping building _ssl and _hashlib :( I'm not sure where that /usr/local/opt/openssl came from.

According to running brew --prefix openssl the correct path is: /usr/local/opt/openssl@3 not /usr/local/opt/openssl

Oh, they version bumped to OpenSSL 3. Sigh. We should work to avoid depending on Homebrew.

Oh, they version bumped to OpenSSL 3. Sigh. We should work to avoid depending on Homebrew.

Whay I am more concerned is that we have been not testing openssl for a while :S

According to running brew --prefix openssl the correct path is: /usr/local/opt/openssl@3 not /usr/local/opt/openssl

I think if we use brew --prefix openssl we should be fine

Perhaps the CI job should check and fail if _ssl doesn't build since it is so important?

Perhaps the CI job should check and fail if _ssl doesn't build since it is so important?

With this PR it does, because that's what was making it fail 😉

@ned-deily Ok, openssl works, now we fail with _tkinter on macos :(

With the PKG-CONFIG macros we are missing -I/Library ↪ /Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Tcl.framework/Headers -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Tk.framework/Headers from the link line :(

Seems a side effect of https://github.com/python/cpython/pull/23721/files

Without the pkg-config macro, it was picking up the system 8.5 Tcl/Tk which is next to useless but undoubtedly the tk-based tests were correctly skipped anyway because of the lack of a display environment. Now with the pkg-config, it is probably picking up a Homebrew Tcl/Tk from /usr/local which may not be built as expected; it works fine with a MacPorts installed Tcl/Tk. I'll pull down a Homebrew one and check.

The tests will be skipped in any case so it's just an issue of the build failure.

it is probably picking up a Homebrew Tcl/Tk from /usr/local which may not be built as expected

Apparently no:

gcc -Wno-unused-result -Wsign-compare -g -O0 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -DWITH_APPINIT=1 -I./Include -I. -I/usr/local/include -I/Users/runner/work/cpython/cpython/Include -I/Users/runner/work/cpython/cpython -c /Users/runner/work/cpython/cpython/Modules/_tkinter.c -o build/temp.macosx-10.15-x86_64-3.11-pydebug/Users/runner/work/cpython/cpython/Modules/_tkinter.o -I/usr/local/include

Is picking it from /usr/local/include which is not homebrew

Oh, it was Homebrew in the past, they moved away from /usr/local. Try removing it (and other cruft in /usr/local ?) and making sure Tcl and Tk are installed in the /opt Homebrew prefix.

Try removing it (and other cruft in /usr/local ?)

Removing what?

and making use Tcl and Tk are installed in the /opt Homebrew prefix.

I don't think those are installed. I am triying to install them to see what happens

From the date and the fact that current Homebrew does not link Tk into /usr/local/lib:

==> Pouring tcl-tk--8.6.11_1.big_sur.bottle.tar.gz
==> Caveats
tcl-tk is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have tcl-tk first in your PATH, run:
  echo 'export PATH="/usr/local/opt/tcl-tk/bin:$PATH"' >> ~/.zshrc

For compilers to find tcl-tk you may need to set:
  export LDFLAGS="-L/usr/local/opt/tcl-tk/lib"
  export CPPFLAGS="-I/usr/local/opt/tcl-tk/include"

For pkg-config to find tcl-tk you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/tcl-tk/lib/pkgconfig"

does not link Tk into /usr/local/lib:

But there are other libraries with symlinks there, no?

% ls -l /usr/local/lib
total 0
lrwxr-xr-x  1 sysadmin  admin  39 Oct  3 14:55 libgdbm.6.dylib -> ../Cellar/gdbm/1.21/lib/libgdbm.6.dylib
lrwxr-xr-x  1 sysadmin  admin  33 Oct  3 14:55 libgdbm.a -> ../Cellar/gdbm/1.21/lib/libgdbm.a
lrwxr-xr-x  1 sysadmin  admin  37 Oct  3 14:55 libgdbm.dylib -> ../Cellar/gdbm/1.21/lib/libgdbm.dylib

/usr/local/lib/libtk8.6.dylib (compatibility version 8.6.0, current version 8.6.6)

Yeah, that's an ancient version of Tk, the current is 8.6.11 and it was clearly built as an X11 version rather than the native Quartz version. The X11 version barely works on macOS and we don't support it.

Ok, so we need to figure out why PKG_CONFIG_PATH="$(brew --prefix openssl@1.1)/lib/pkgconfig:$(brew --prefix tcl-tk)/lib/pkgconfig" pkg-config tk --libs links against some bad version

But there are other libraries with symlinks there, no?

There are but that's one of the joys of Homebrew. They make a decision about which packages to link to /usr/local based on various criteria, primarily whether the package duplicates something supplied by Apple in the system. So, for example, Apple doesn't ship gdbm in macOS so there's no conflict, while they do ship a version of Tk, albeit a deprecated broken one. And Apple no longer ships an OpenSSL but it does ship LibreSSL pretty much strictly for the use of the system Python and other third-party open source packages; native Mac apps use Apple's built-in TLS stuff.

Ok, so we need to figure out why PKG_CONFIG_PATH="$(brew --prefix openssl@1.1)/lib/pkgconfig:$(brew --prefix tcl-tk)/lib/pkgconfig" pkg-config tk --libs links against some bad version

Probably /usr/local/ gets searched first in configure and setup.py in most cases, before the pkg-config supplied path.

Probably /usr/local/ gets searched first in configure and setup.py in most cases, before the pkg-config supplied path.

Yeah, apparently PKG_CONFIG returns the correct thing:

PKG_CONFIG_PATH="$(brew --prefix openssl@1.1)/lib/pkgconfig:$(brew --prefix tcl-tk)/lib/pkgconfig" pkg-config tk --libs
-L/usr/local/Cellar/tcl-tk/8.6.11_1/lib -ltk8.6 -ltkstub8.6 -ltcl8.6 -ltclstub8.6

Ah, is because -L/usr/local/lib is before in the link line:

gcc -bundle -undefined dynamic_lookup build/temp.macosx-10.15-x86_64-3.11-pydebug/Users/runner/work/cpython/cpython/Modules/_tkinter.o build/temp.macosx-10.15-x86_64-3.11-pydebug/Users/runner/work/cpython/cpython/Modules/tkappinit.o -L/usr/local/lib -o build/lib.macosx-10.15-x86_64-3.11-pydebug/_tkinter.cpython-311d-darwin.so -L/usr/local/Cellar/tcl-tk/8.6.11_1/lib -ltk8.6 -ltkstub8.6 -ltcl8.6 -ltclstub8.6

Ok, prepending LDFLAGS=$(-L$(brew --prefix tcl-tk)/lib) seems to do the trick

I will be landing this and the 3.10 version if the CI passes. What a day 😓

I don't think that's a particular good long term fix for the CI. There's definitely an issue here about /usr/local/lib, in general, overriding other selections. We should try to fix that. And we should get a more predictable CI environment. But we don't have to solve that tonight :)

overriding other selections. We should try to fix that.

We don't have control over that. That file is there from the files that the GitHub actions CI gives us. The only thing we can do is remove them. Basically we say "run this on MacOS" and GitHub gives us a VM where we can add more things but where the stuff in /usr/local/lib comes already there. We can remove them but there is no way to ask for something without it.

@ned-deily Maybe you are referring to something different when you mention "And we should get a more predictable CI environment" but unfortunately this is what GitHub actions gives us as base.

We could do some strategic rm -f at the start of the CI run and someone should probably ask GitHub to clean up the VM. I didn't see any notice of a Tk, for example, being there in the GitHub readme but I might have missed it. I can't imagine any project wanting to have a build and test environment with all that cruft. But we can do something about our builds preferring /usr/local; but that needs to be carefully thought through.

I can't imagine any project wanting to have a build and test environment with all that cruft.

Apparently CI image comes with a ton of stuff pre-installed and they seem to be a bit reticent to modify stuff :(

For example:

actions/virtual-environments#1088
actions/virtual-environments#771
actions/virtual-environments#275
actions/virtual-environments#150

They seem to be preinstalling all this:

https://github.com/actions/virtual-environments/tree/main/images/macos/provision/core

Ha, including our own Python 2.7 macOS installer, apparently. I'll try to take a look at what there's soon but not today, assuming the LDFLAGS workaround is successful. It should be. And we should also move to their macOS Big Sur 11 image soon since 12 Monterey will be released soon. Perhaps the Big Sur image isn't so crufty.