Port module setup to PY_STDLIB_MOD() macro and addext() #90005

tiran · 2021-11-19T16:56:30Z

BPO	45847
Nosy	@tiran, @ned-deily, @pablogsal, @miss-islington, @erlend-aasland, @arhadthedev
PRs	bpo-45847: Port builtin hashlib extensions to PY_STDLIB_MOD (GH-29642) #29642 bpo-45847: Port _scproxy to PY_STDLIB_MOD (GH-29644) #29644 bpo-45847: Port test modules to PY_STDLIB_MOD (GH-29660) #29660 bpo-45847: Port grp, spwd, termios, resource, syslog to PY_STDLIB_MOD (GH-29668) #29668 bpo-45847: Port codecs and unicodedata to PY_STDLIB_MOD (GH-29685) #29685 bpo-45847: Port array, _contextvars, math, and cmath to PY_STDLIB_MOD_SIMPLE (GH-29688) #29688 bpo-45847: Port _bisect, _heapq, _json, _pickle, _random, and _zoneinfo to PY_STDLIB_MOD_SIMPLE #29689 bpo-45847: Port _lfprof, _opcode, _asyncio, _queue, _statistics, and _typing to PY_STDLIB_MOD_SIMPLE #29690 bpo-45847: Port fcntl to Py_STDLIB_MOD (GH-29696) #29696 bpo-45847: Various PY_STDLIB_MOD cleanups (GH-29697) #29697 bpo-45847: Port nis module to PY_STDLIB_MOD (GH-29699) #29699 bpo-45847: Port compression libs to PY_STDLIB_MOD (GH-29702) #29702 bpo-45847: Port mmap, select, and _xxsubinterpreters to Py_STDLIB_MOD (GH-29703) #29703 bpo-45847: Port audioop, _csv, and _posixsubprocess to PY_STDLIB_MOD_SIMPLE #29705 bpo-45847: port _struct to PY_STDLIB_MOD (GH-29706) #29706 bpo-45847: Port xxlimited and xxlimited_35 to PY_STDLIB_MOD (GH-29707) #29707 bpo-45847: Port _socket to PY_STDLIB_MOD (GH-29713) #29713 bpo-45847: Fix xxlimited and xxlimited_35 build conditions (GH-29715) #29715 bpo-45847: Port _gdbm to PY_STDLIB_MOD (GH-29720) #29720 bpo-45847: Port _crypt to PY_STDLIB_MOD (GH-29725) #29725 bpo-45847: Port _ssl and _hashlib to PY_STDLIB_MOD (GH-29727) #29727 bpo-45847: Port _posixshmem to PY_STDLIB_MOD (GH-29738) #29738 bpo-45847: Port _uuid to PY_STDLIB_MOD (GH-29741) #29741 bpo-45847: Fix _crypt detection on Ubuntu (GH-29743) #29743 bpo-45847: Port _ctypes partly to PY_STDLIB_MOD (GH-29747) #29747 bpo-45847: Port _multiprocessing to PY_STDLIB_MOD (GH-29768) #29768 bpo-45847: Make socket module conditional (GH-29769) #29769 bpo-45847: Fix _multiprocessing on platforms without sem_open (GH-29833) #29833 bpo-45847: PY_STDLIB_MOD_SIMPLE now checks py_stdlib_not_available (GH-29844) #29844 bpo-45847: Update whatsnew and add place holder entries for missing extensions (GH-29914) #29914 bpo-45847: Fix uuid detection on macOS (GH-29946) #29946 bpo-45847: Don't override user overrides for CFLAGS/LIBS (GH-29967) #29967 bpo-45847: Revert Port _ctypes partly to PY_STDLIB_MOD (GH-29747) (GH-29969) #29969 bpo-45847: Port _tkinter to PY_STDLIB_MOD #31698 gh-90005: Port _ctypes to PY_STDLIB_MOD #32229 Fix `Contributed ... in bpo-bpo-45847` typo #32299 bpo-45847: Adapt macOS installer build to use new tkinter configure vars #32328

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-11-19.16:56:30.299>
labels = ['type-feature', 'build', '3.11']
title = 'Port module setup to PY_STDLIB_MOD() macro and addext()'
updated_at = <Date 2022-04-05.04:48:03.902>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2022-04-05.04:48:03.902>
actor = 'Anthony Sottile'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2021-11-19.16:56:30.299>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 45847
keywords = ['patch']
message_count = 61.0
messages = ['406597', '406618', '406646', '406662', '406703', '406714', '406746', '406751', '406754', '406755', '406763', '406771', '406777', '406779', '406781', '406782', '406783', '406784', '406785', '406788', '406801', '406822', '406823', '406842', '406890', '406891', '406892', '406905', '406913', '406914', '407077', '407199', '407249', '407255', '407274', '407287', '407654', '407668', '407861', '407949', '407958', '407959', '407962', '407981', '408008', '414642', '414645', '414648', '414649', '414656', '416425', '416565', '416566', '416567', '416568', '416569', '416570', '416575', '416600', '416712', '416750']
nosy_count = 6.0
nosy_names = ['christian.heimes', 'ned.deily', 'pablogsal', 'miss-islington', 'erlendaasland', 'arhadthedev']
pr_nums = ['29642', '29644', '29660', '29668', '29685', '29688', '29689', '29690', '29696', '29697', '29699', '29702', '29703', '29705', '29706', '29707', '29713', '29715', '29720', '29725', '29727', '29738', '29741', '29743', '29747', '29768', '29769', '29833', '29844', '29914', '29946', '29967', '29969', '31698', '32229', '32299', '32328']
priority = None
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45847'
versions = ['Python 3.11']

The text was updated successfully, but these errors were encountered:

tiran · 2021-11-19T16:56:30Z

bpo-45573 added the PY_STDLIB_MOD() autoconf macro, Modules/Setup.stdlib.in template, and setup.py helper method addext().

The macro sets up

MODULE_{NAME}_TRUE/FALSE conditional
MODULE_{NAME} variable with values yes, disabled, missing, or n/a
MODULE_{NAME}_CFLAGS
MODULE_{NAME}_LDFLAGS

Additionally there is a MODULE_{NAME}_DEPS variable with dependency information.

The Modules/Setup.stdlib.in template and addext() method consume the variables and set up extension compilation and linking. There is no need to look for headers or libraries in setup.py any more.

Let's port all modules to the new approach.

tiran · 2021-11-19T19:20:40Z

New changeset c8c21bd by Christian Heimes in branch 'main':
bpo-45847: Port builtin hashlib extensions to PY_STDLIB_MOD (GH-29642)
c8c21bd

tiran · 2021-11-20T09:18:59Z

New changeset 5596909 by Christian Heimes in branch 'main':
bpo-45847: Port _scproxy to PY_STDLIB_MOD (GH-29644)
5596909

tiran · 2021-11-20T15:43:16Z

New changeset f36c69a by Christian Heimes in branch 'main':
bpo-45847: Port test modules to PY_STDLIB_MOD (GH-29660)
f36c69a

tiran · 2021-11-21T09:45:36Z

New changeset f201d26 by Christian Heimes in branch 'main':
bpo-45847: Port grp, spwd, termios, resource, syslog to PY_STDLIB_MOD (GH-29668)
f201d26

tiran · 2021-11-21T13:08:55Z

New changeset 2afa1a1 by Christian Heimes in branch 'main':
bpo-45847: Port codecs and unicodedata to PY_STDLIB_MOD (GH-29685)
2afa1a1

tiran · 2021-11-22T08:05:40Z

New changeset 133c65a by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port array, _contextvars, math, and cmath to PY_STDLIB_MOD_SIMPLE (GH-29688)
133c65a

miss-islington · 2021-11-22T08:45:50Z

New changeset 718cee0 by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port _bisect, _heapq, _json, _pickle, _random, and _zoneinfo to PY_STDLIB_MOD_SIMPLE (GH-29689)
718cee0

erlend-aasland · 2021-11-22T09:56:42Z

45 down, 35 to go.

miss-islington · 2021-11-22T09:57:58Z

New changeset 39f7d2f by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port _lfprof, _opcode, _asyncio, _queue, _statistics, and _typing to PY_STDLIB_MOD_SIMPLE (GH-29690)
39f7d2f

tiran · 2021-11-22T11:09:22Z

New changeset 29699a2 by Christian Heimes in branch 'main':
bpo-45847: Various PY_STDLIB_MOD cleanups (GH-29697)
29699a2

tiran · 2021-11-22T13:02:40Z

New changeset 5b946ca by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port fcntl to Py_STDLIB_MOD (GH-29696)
5b946ca

tiran · 2021-11-22T13:50:09Z

New changeset b451673 by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port mmap, select, and _xxsubinterpreters to Py_STDLIB_MOD (GH-29703)
b451673

tiran · 2021-11-22T13:52:38Z

New changeset d9cedab by Christian Heimes in branch 'main':
bpo-45847: Port compression libs to PY_STDLIB_MOD (GH-29702)
d9cedab

tiran · 2021-11-22T14:18:49Z

New changeset c6dec7e by Christian Heimes in branch 'main':
bpo-45847: Port nis module to PY_STDLIB_MOD (GH-29699)
c6dec7e

tiran · 2021-11-22T14:25:45Z

Not converted yet:

readline
_curses
_curses_panel
_crypt
_socket
_ssl
_hashlib
_dbm
_gdbm
_ctypes
_multiprocessing
_posixshmem
_tkinter
_uuid
xxlimited
xxlimited_35

erlend-aasland · 2021-11-22T14:28:17Z

_socket coming up...

miss-islington · 2021-11-22T14:37:34Z

New changeset eee683c by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port audioop, _csv, and _posixsubprocess to PY_STDLIB_MOD_SIMPLE (GH-29705)
eee683c

tiran · 2021-11-22T14:42:42Z

_crypt is powered by pkgconf libcrypt or libxcrypt on Linux. On my system and on Debian, libcrypt.pc is a symlink to libxcrypt.pc. Linux has crypt() and/or crypt_r() in <crypt.h>. On FreeBSD crypt_r() is in <unistd.h> and libc.

_uuid needs similar special handling. It's <uuid/uuid.h>, -luuid, and uuid.pc on Linux. On BSD it's <uuid.h> and symbols are in libc.

_readline uses either libreadline (readline.pc) or libeditline (libeditline.pc). It also needs special handling of tinfo and termcap on some platforms.

ndbm, gdbm, and libdb have no pkgconf providers. My PR #73720 has some code to detect them in configure.

tiran · 2021-11-22T14:58:53Z

New changeset 0e1c2f3 by Christian Heimes in branch 'main':
bpo-45847: port _struct to PY_STDLIB_MOD (GH-29706)
0e1c2f3

tiran · 2021-11-22T20:27:32Z

New changeset 2dc7d3d by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port xxlimited and xxlimited_35 to PY_STDLIB_MOD (GH-29707)
2dc7d3d

tiran · 2021-11-23T07:51:35Z

New changeset 758a23d by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Fix xxlimited and xxlimited_35 build conditions (GH-29715)
758a23d

tiran · 2021-11-23T07:52:10Z

New changeset d82f2ca by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port _socket to PY_STDLIB_MOD (GH-29713)
d82f2ca

tiran · 2021-11-23T14:41:06Z

We are down to 9 missing modules:

readline
_curses
_curses_panel
_dbm
_ctypes
_multiprocessing
_posixshmem
_tkinter
_uuid

…pythonGH-94301) It makes it easier to look for module states in sysconfig without special casing suffixes "_CFLAGS", "_DEPS", "_LDFLAGS", "_OBJS", and "CTYPES_MALLOC_CLOSURE".. (cherry picked from commit 71868a0) Co-authored-by: Christian Heimes <christian@python.org>

…4301) (GH-94317) Co-authored-by: Christian Heimes <christian@python.org>

The fallback path did not set ``LIBFFI_LIBS`` variable to link with ``-lffi``.

The fallback path did not set LIBFFI_LIBS variable to link with ``-lffi``.

Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@protonmail.com>

erlend-aasland · 2022-07-06T18:41:23Z

Looks like we're done! Thanks for the work with the remaining modules, Christian! I believe we can close this issue now.

erlend-aasland · 2022-07-06T19:13:41Z

See #93939 for the next step :)

tiran added 3.11 build The build process and cross-build type-feature A feature request or enhancement labels Nov 19, 2021

bedevere-bot mentioned this issue Jun 27, 2022

[3.11] gh-90005: Rename MODULE_EGG variable to MODULE_EGG_STATE (GH-94301) (GH-94317) #94317

Merged

tiran added a commit that referenced this issue Jun 27, 2022

[3.11] gh-90005: Rename MODULE_EGG variable to MODULE_EGG_STATE (GH-9…

4ec146c

…4301) (GH-94317) Co-authored-by: Christian Heimes <christian@python.org>

bedevere-bot mentioned this issue Jun 30, 2022

gh-90005: Port _dbm module to PY_STDLIB_MOD (GH-94433) #94433

Merged

tiran added a commit to tiran/cpython that referenced this issue Jun 30, 2022

pythongh-90005-ffi: Fix building _ctypes without pkg-config

87acdc1

The fallback path did not set ``LIBFFI_LIBS`` variable to link with ``-lffi``.

bedevere-bot mentioned this issue Jun 30, 2022

gh-90005-ffi: Fix building _ctypes without pkg-config (GH-94451) #94451

Merged

bedevere-bot mentioned this issue Jun 30, 2022

gh-90005: Port readline and curses to PY_STDLIB_MOD (GH-94452) #94452

Merged

tiran added a commit that referenced this issue Jul 1, 2022

gh-90005-ffi: Fix building _ctypes without pkg-config (GH-94451)

d6acdc1

The fallback path did not set LIBFFI_LIBS variable to link with ``-lffi``.

tiran added a commit to tiran/cpython that referenced this issue Jul 1, 2022

pythongh-90005: Port _dbm module to PY_STDLIB_MOD

ad90c1e

tiran added a commit that referenced this issue Jul 1, 2022

gh-90005: Port _dbm module to PY_STDLIB_MOD (GH-94433)

ec5e253

tiran added a commit to tiran/cpython that referenced this issue Jul 1, 2022

pythongh-90005: Port readline and curses to PY_STDLIB_MOD

4c97453

tiran added a commit that referenced this issue Jul 6, 2022

gh-90005: Port readline and curses to PY_STDLIB_MOD (GH-94452)

e925241

Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@protonmail.com>

erlend-aasland added the pending The issue will be closed if no feedback is provided label Jul 6, 2022

erlend-aasland closed this as completed Jul 6, 2022

tiran mentioned this issue Jul 12, 2022

gh-93939: Build C extensions without setup.py #94474

Merged

erlend-aasland removed the pending The issue will be closed if no feedback is provided label Jul 20, 2022

sharewax mentioned this issue Aug 13, 2022

_tkinter build issue on Centos7 due changing TK/TCL detection by configure script. #95957

Closed

Port module setup to PY_STDLIB_MOD() macro and addext() #90005

Port module setup to PY_STDLIB_MOD() macro and addext() #90005

tiran commented Nov 19, 2021

tiran commented Nov 19, 2021

tiran commented Nov 19, 2021

tiran commented Nov 20, 2021

tiran commented Nov 20, 2021

tiran commented Nov 21, 2021

tiran commented Nov 21, 2021

tiran commented Nov 22, 2021

miss-islington commented Nov 22, 2021

erlend-aasland commented Nov 22, 2021

miss-islington commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

erlend-aasland commented Nov 22, 2021

miss-islington commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 23, 2021

tiran commented Nov 23, 2021

tiran commented Nov 23, 2021

erlend-aasland commented Jul 6, 2022

erlend-aasland commented Jul 6, 2022

Port module setup to PY_STDLIB_MOD() macro and addext() #90005

Port module setup to PY_STDLIB_MOD() macro and addext() #90005

Comments

tiran commented Nov 19, 2021

tiran commented Nov 19, 2021

tiran commented Nov 19, 2021

tiran commented Nov 20, 2021

tiran commented Nov 20, 2021

tiran commented Nov 21, 2021

tiran commented Nov 21, 2021

tiran commented Nov 22, 2021

miss-islington commented Nov 22, 2021

erlend-aasland commented Nov 22, 2021

miss-islington commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

erlend-aasland commented Nov 22, 2021

miss-islington commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 22, 2021

tiran commented Nov 23, 2021

tiran commented Nov 23, 2021

tiran commented Nov 23, 2021

erlend-aasland commented Jul 6, 2022

erlend-aasland commented Jul 6, 2022