Port module setup to PY_STDLIB_MOD() macro and addext() #90005

tiran · 2021-11-19T16:56:30Z

BPO	45847
Nosy	@tiran, @ned-deily, @pablogsal, @miss-islington, @erlend-aasland, @arhadthedev
PRs	#29642 #29644 #29660 #29668 #29685 #29688 #29689 #29690 #29696 #29697 #29699 #29702 #29703 #29705 #29706 #29707 #29713 #29715 #29720 #29725 #29727 #29738 #29741 #29743 #29747 #29768 #29769 #29833 #29844 #29914 #29946 #29967 #29969 #31698 #32229 #32299 #32328

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-11-19.16:56:30.299>
labels = ['type-feature', 'build', '3.11']
title = 'Port module setup to PY_STDLIB_MOD() macro and addext()'
updated_at = <Date 2022-04-05.04:48:03.902>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2022-04-05.04:48:03.902>
actor = 'Anthony Sottile'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2021-11-19.16:56:30.299>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 45847
keywords = ['patch']
message_count = 61.0
messages = ['406597', '406618', '406646', '406662', '406703', '406714', '406746', '406751', '406754', '406755', '406763', '406771', '406777', '406779', '406781', '406782', '406783', '406784', '406785', '406788', '406801', '406822', '406823', '406842', '406890', '406891', '406892', '406905', '406913', '406914', '407077', '407199', '407249', '407255', '407274', '407287', '407654', '407668', '407861', '407949', '407958', '407959', '407962', '407981', '408008', '414642', '414645', '414648', '414649', '414656', '416425', '416565', '416566', '416567', '416568', '416569', '416570', '416575', '416600', '416712', '416750']
nosy_count = 6.0
nosy_names = ['christian.heimes', 'ned.deily', 'pablogsal', 'miss-islington', 'erlendaasland', 'arhadthedev']
pr_nums = ['29642', '29644', '29660', '29668', '29685', '29688', '29689', '29690', '29696', '29697', '29699', '29702', '29703', '29705', '29706', '29707', '29713', '29715', '29720', '29725', '29727', '29738', '29741', '29743', '29747', '29768', '29769', '29833', '29844', '29914', '29946', '29967', '29969', '31698', '32229', '32299', '32328']
priority = None
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45847'
versions = ['Python 3.11']

tiran · 2021-11-19T16:56:30Z

bpo-45573 added the PY_STDLIB_MOD() autoconf macro, Modules/Setup.stdlib.in template, and setup.py helper method addext().

The macro sets up

MODULE_{NAME}_TRUE/FALSE conditional
MODULE_{NAME} variable with values yes, disabled, missing, or n/a
MODULE_{NAME}_CFLAGS
MODULE_{NAME}_LDFLAGS

Additionally there is a MODULE_{NAME}_DEPS variable with dependency information.

The Modules/Setup.stdlib.in template and addext() method consume the variables and set up extension compilation and linking. There is no need to look for headers or libraries in setup.py any more.

Let's port all modules to the new approach.

tiran · 2021-11-19T19:20:40Z

New changeset c8c21bd by Christian Heimes in branch 'main':
bpo-45847: Port builtin hashlib extensions to PY_STDLIB_MOD (GH-29642)
c8c21bd

tiran · 2021-11-20T09:18:59Z

New changeset 5596909 by Christian Heimes in branch 'main':
bpo-45847: Port _scproxy to PY_STDLIB_MOD (GH-29644)
5596909

tiran · 2021-11-20T15:43:16Z

New changeset f36c69a by Christian Heimes in branch 'main':
bpo-45847: Port test modules to PY_STDLIB_MOD (GH-29660)
f36c69a

tiran · 2021-11-21T09:45:36Z

New changeset f201d26 by Christian Heimes in branch 'main':
bpo-45847: Port grp, spwd, termios, resource, syslog to PY_STDLIB_MOD (GH-29668)
f201d26

tiran · 2021-11-21T13:08:55Z

New changeset 2afa1a1 by Christian Heimes in branch 'main':
bpo-45847: Port codecs and unicodedata to PY_STDLIB_MOD (GH-29685)
2afa1a1

tiran · 2021-11-22T08:05:40Z

New changeset 133c65a by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port array, _contextvars, math, and cmath to PY_STDLIB_MOD_SIMPLE (GH-29688)
133c65a

miss-islington · 2021-11-22T08:45:50Z

New changeset 718cee0 by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port _bisect, _heapq, _json, _pickle, _random, and _zoneinfo to PY_STDLIB_MOD_SIMPLE (GH-29689)
718cee0

erlend-aasland · 2021-11-22T09:56:42Z

45 down, 35 to go.

miss-islington · 2021-11-22T09:57:58Z

New changeset 39f7d2f by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port _lfprof, _opcode, _asyncio, _queue, _statistics, and _typing to PY_STDLIB_MOD_SIMPLE (GH-29690)
39f7d2f

tiran · 2021-11-22T11:09:22Z

New changeset 29699a2 by Christian Heimes in branch 'main':
bpo-45847: Various PY_STDLIB_MOD cleanups (GH-29697)
29699a2

tiran · 2021-11-22T13:02:40Z

New changeset 5b946ca by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port fcntl to Py_STDLIB_MOD (GH-29696)
5b946ca

tiran · 2021-11-22T13:50:09Z

New changeset b451673 by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port mmap, select, and _xxsubinterpreters to Py_STDLIB_MOD (GH-29703)
b451673

tiran · 2021-11-22T13:52:38Z

New changeset d9cedab by Christian Heimes in branch 'main':
bpo-45847: Port compression libs to PY_STDLIB_MOD (GH-29702)
d9cedab

tiran · 2021-11-22T14:18:49Z

New changeset c6dec7e by Christian Heimes in branch 'main':
bpo-45847: Port nis module to PY_STDLIB_MOD (GH-29699)
c6dec7e

tiran · 2021-11-22T14:25:45Z

Not converted yet:

readline
_curses
_curses_panel
_crypt
_socket
_ssl
_hashlib
_dbm
_gdbm
_ctypes
_multiprocessing
_posixshmem
_tkinter
_uuid
xxlimited
xxlimited_35

erlend-aasland · 2021-11-22T14:28:17Z

_socket coming up...

miss-islington · 2021-11-22T14:37:34Z

New changeset eee683c by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port audioop, _csv, and _posixsubprocess to PY_STDLIB_MOD_SIMPLE (GH-29705)
eee683c

tiran · 2021-11-22T14:42:42Z

_crypt is powered by pkgconf libcrypt or libxcrypt on Linux. On my system and on Debian, libcrypt.pc is a symlink to libxcrypt.pc. Linux has crypt() and/or crypt_r() in <crypt.h>. On FreeBSD crypt_r() is in <unistd.h> and libc.

_uuid needs similar special handling. It's <uuid/uuid.h>, -luuid, and uuid.pc on Linux. On BSD it's <uuid.h> and symbols are in libc.

_readline uses either libreadline (readline.pc) or libeditline (libeditline.pc). It also needs special handling of tinfo and termcap on some platforms.

ndbm, gdbm, and libdb have no pkgconf providers. My PR #73720 has some code to detect them in configure.

tiran · 2021-11-22T14:58:53Z

New changeset 0e1c2f3 by Christian Heimes in branch 'main':
bpo-45847: port _struct to PY_STDLIB_MOD (GH-29706)
0e1c2f3

tiran · 2021-11-22T20:27:32Z

New changeset 2dc7d3d by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port xxlimited and xxlimited_35 to PY_STDLIB_MOD (GH-29707)
2dc7d3d

tiran · 2021-11-23T07:51:35Z

New changeset 758a23d by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Fix xxlimited and xxlimited_35 build conditions (GH-29715)
758a23d

tiran · 2021-11-23T07:52:10Z

New changeset d82f2ca by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port _socket to PY_STDLIB_MOD (GH-29713)
d82f2ca

tiran · 2021-11-23T14:41:06Z

We are down to 9 missing modules:

readline
_curses
_curses_panel
_dbm
_ctypes
_multiprocessing
_posixshmem
_tkinter
_uuid

erlend-aasland · 2022-03-07T11:11:07Z

Solved: Ubuntu is fine, we just need to treat LIBS and LDFLAGS with care in configure.ac. #75879 is updated with a fix.

miss-islington · 2022-03-31T10:19:20Z

New changeset b36d222 by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port _tkinter to PY_STDLIB_MOD (GH-31698)
b36d222

asottile · 2022-04-02T16:43:31Z

this appears to break the tkinter extension for ubuntu bionic (18.04) -- I'm not entirely sure on the correct fix here but I get the following from trying to build there:

2022-04-02T15:52:08.0910452Z Python build finished successfully!
2022-04-02T15:52:08.0911924Z The necessary bits to build these optional modules were not found:
2022-04-02T15:52:08.0913469Z _tkinter

this is the tk I have available:

root@f0dd06a3e87c:/# dpkg -l | grep libtk
ii libtk8.6:amd64 8.6.8-4 amd64 Tk toolkit for Tcl and X11 v8.6 - run-time files

tiran · 2022-04-02T16:54:15Z

Please attach your config.log file. Did configure tell you that you are missing pkg-config?

asottile · 2022-04-02T16:54:52Z

the tk-dev package on ubuntu bionic does not ship with a pkg-config file for tk so it does not build properly there:

root@f0dd06a3e87c:/cpython# dpkg -L tk8.6-dev | grep pc
root@f0dd06a3e87c:/cpython#

(a note: bionic reaches end of life in april 2023 so it is likely to still see significant use until then)

tiran · 2022-04-02T17:23:08Z

Could you please open a distro bug with Ubuntu and report the issue? We require a pkg-config file for TCL/TK.

asottile · 2022-04-02T17:33:04Z

I could, but it's very unlikely to get fixed given I believe 18.04 is in security-only fixes and backporting a pkg-config file seems unlikely

this worked two days ago before this patch

tiran · 2022-04-02T17:38:46Z

As I said already, we require a pkg-config file for TCL/TK detection. It turned out to be too painful and too complicated to implement manual TCL/TK detection in configure.ac. Erlend and I tried and eventually gave up.

If your platform does not provide the pkg-config files, then you need to work around the problem or contribute a patch.

tiran · 2022-04-02T18:44:13Z

PS: Check out ./configure --help

erlend-aasland · 2022-04-02T22:18:30Z

Anthony: you should be able to build the _tkinter by manually specifying the compiler and linker flags using the TCLTK_CFLAGS and TCLTK_LIBS environment variables. AFAICS, the flags provided by pkgconfig in Ubuntu 20.04 should work well in 18.04 (possibly with minor adjustments; I didn't check).

ned-deily · 2022-04-05T01:18:23Z

New changeset 5a7506d by Oleg Iarygin in branch 'main':
Fix "Contributed ... in bpo-bpo-45847" (GH-32299)
5a7506d

ned-deily · 2022-04-05T04:46:19Z

New changeset a0c7004 by Ned Deily in branch 'main':
bpo-45847: Adapt macOS installer build to use new tkinter configure vars (GH-32328)
a0c7004

erlend-aasland · 2022-04-14T20:08:41Z

Regarding _ctypes: are we actually using the bundled libffi anymore?

$ ls  Modules/_ctypes/libffi_osx 
LICENSE		README		README.pyobjc	ffi.c		include		powerpc		types.c		x86
$ grep -r libffi_osx *
Tools/c-analyzer/cpython/_parser.py:#Modules/_ctypes/libffi_osx/*.c
Tools/scripts/patchcheck.py:EXCLUDE_DIRS = [os.path.join('Modules', '_ctypes', 'libffi_osx'),

erlend-aasland · 2022-04-14T23:39:26Z

(cherry picked from commit 269e726) Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@protonmail.com>

Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no> Automerge-Triggered-By: GH:tiran

It makes it easier to look for module states in sysconfig without special casing suffixes "_CFLAGS|, "_DEPS", "_LDFLAGS", "_OBJS", and "CTYPES_MALLOC_CLOSURE".

It makes it easier to look for module states in sysconfig without special casing suffixes "_CFLAGS", "_DEPS", "_LDFLAGS", "_OBJS", and "CTYPES_MALLOC_CLOSURE".

tiran added 3.11 build type-feature labels Nov 19, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

erlend-aasland mentioned this issue Apr 10, 2022

Use pkg-config to find dependencies #72394

Closed

erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Apr 17, 2022

pythongh-90005: Cleanup after pythonGH-31698

62957a1

tiran mentioned this issue Apr 20, 2022

gh-76214: Disable _ctypes_test on all WASM targets #91750

Closed

miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 9, 2022

pythongh-90005: Cleanup after pythonGH-31698 (pythonGH-91642)

2f434cb

(cherry picked from commit 269e726) Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@protonmail.com>

erlend-aasland added a commit that referenced this issue May 9, 2022

gh-90005: Cleanup after GH-31698 (#91642)

269e726

miss-islington added a commit that referenced this issue May 9, 2022

gh-90005: Cleanup after GH-31698 (GH-91642)

5e2e9c4

(cherry picked from commit 269e726) Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@protonmail.com>

erlend-aasland mentioned this issue May 19, 2022

bpo-34449: HP aCC complains about invalid -fPIC on HP-UX #8847

Merged

AA-Turner added the 3.12 label Jun 7, 2022

tiran mentioned this issue Jun 17, 2022

Build shared extensions without distutils #93939

Open

bedevere-bot mentioned this issue Jun 26, 2022

gh-90005: Port _ctypes to PY_STDLIB_MOD #32229

Merged

miss-islington pushed a commit that referenced this issue Jun 26, 2022

gh-90005: Port _ctypes to PY_STDLIB_MOD (GH-32229)

bb8b931

Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no> Automerge-Triggered-By: GH:tiran

bedevere-bot mentioned this issue Jun 26, 2022

gh-90005: Rename MODULE_EGG variable to MODULE_EGG_STATE #94301

Merged

Port module setup to PY_STDLIB_MOD() macro and addext() #90005

Port module setup to PY_STDLIB_MOD() macro and addext() #90005

Comments

tiran commented Nov 19, 2021

tiran commented Nov 19, 2021

tiran commented Nov 19, 2021

tiran commented Nov 20, 2021

tiran commented Nov 20, 2021

tiran commented Nov 21, 2021

tiran commented Nov 21, 2021

tiran commented Nov 22, 2021

miss-islington commented Nov 22, 2021

erlend-aasland commented Nov 22, 2021

miss-islington commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

erlend-aasland commented Nov 22, 2021

miss-islington commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 23, 2021

tiran commented Nov 23, 2021

tiran commented Nov 23, 2021

erlend-aasland commented Mar 7, 2022

miss-islington commented Mar 31, 2022

asottile mannequin commented Apr 2, 2022

tiran commented Apr 2, 2022

asottile mannequin commented Apr 2, 2022

tiran commented Apr 2, 2022

asottile mannequin commented Apr 2, 2022

tiran commented Apr 2, 2022

tiran commented Apr 2, 2022

erlend-aasland commented Apr 2, 2022

ned-deily commented Apr 5, 2022

ned-deily commented Apr 5, 2022

erlend-aasland commented Apr 14, 2022

erlend-aasland commented Apr 14, 2022 • edited

erlend-aasland commented Apr 14, 2022 •

edited