Implement PEP 652 -- Maintaining the Stable ABI #87961

encukou · 2021-04-09T15:43:51Z

BPO	43795
Nosy	@orsenthil, @vstinner, @encukou, @ambv, @willingc, @corona10, @pablogsal, @miss-islington, @erlend-aasland
PRs	#25312 #25315 #25482 #25483 #25668 #25716 #25920 #25925 #26034 #26101 #26123 #26241 #26353 #26354 #26479 #26603 #26740 #26753 #29148 #29956

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-04-09.15:43:50.940>
labels = ['3.10']
title = 'Implement PEP 652 -- Maintaining the Stable ABI'
updated_at = <Date 2021-12-10.23:34:50.985>
user = 'https://github.com/encukou'

bugs.python.org fields:

activity = <Date 2021-12-10.23:34:50.985>
actor = 'lukasz.langa'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2021-04-09.15:43:50.940>
creator = 'petr.viktorin'
dependencies = []
files = []
hgrepos = []
issue_num = 43795
keywords = ['patch']
message_count = 26.0
messages = ['390638', '391081', '391217', '391223', '391235', '391258', '391432', '391435', '391436', '391684', '391686', '392212', '392222', '392296', '393031', '393484', '393496', '393509', '393624', '394335', '395335', '395337', '395913', '395915', '404722', '408267']
nosy_count = 9.0
nosy_names = ['orsenthil', 'vstinner', 'petr.viktorin', 'lukasz.langa', 'willingc', 'corona10', 'pablogsal', 'miss-islington', 'erlendaasland']
pr_nums = ['25312', '25315', '25482', '25483', '25668', '25716', '25920', '25925', '26034', '26101', '26123', '26241', '26353', '26354', '26479', '26603', '26740', '26753', '29148', '29956']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue43795'
versions = ['Python 3.10']

encukou · 2021-04-09T15:43:51Z

This issue tracks implementation of PEP-652.

CPython's Limited C-API and Stable ABI, introduced in PEP-384, will be formalized in a single definitive file, tested, and documented.

encukou · 2021-04-14T14:38:38Z

New changeset 92eebf6 by Petr Viktorin in branch 'master':
bpo-43795: Sort PC/python3dll.c (GH-25312)
92eebf6

encukou · 2021-04-16T15:25:38Z

Opened as a separate issue: bpo-43868, Remove PyOS_ReadlineFunctionPointer from the stable ABI list

encukou · 2021-04-16T16:08:20Z

I unsure about the following symbols.
Victor, do you think they should be part of the stable ABI?

PyThreadState_DeleteCurrent
Py_GetArgcArgv
PyInterpreterState_GetID

vstinner · 2021-04-16T17:23:35Z

PyThreadState_DeleteCurrent

No.

bpo-37878 explicitly excluded it from the limited C API. IMO it's too close to the CPython implementation and so must not be added to the stable ABI.

Py_GetArgcArgv

No.

Python 3.10 now provides sys.orig_argv which can be read in C.
https://docs.python.org/dev/library/sys.html#sys.orig_argv

Py_GetArgcArgv() is part of the "legacy" C API to initialize Python. The "new" PyConfig API provides PyConfig.orig_argv.

FYI in the PyPI top 4000 projects, the function seems to be called by the following projects:

2021-02-18/uWSGI-2.0.19.1.tar.gz
2021-02-18/stem-1.8.0.tar.gz
2021-02-18/sphinx-gallery-0.8.2.tar.gz
2021-02-18/setproctitle-1.2.2.tar.gz
2021-02-18/pyuwsgi-2.0.19.1.post0.tar.gz
2021-02-18/mercurial-5.7.tar.gz
2021-02-18/httpretty-1.0.5.tar.gz
2021-02-18/CherryPy-18.6.0.tar.gz
2021-02-18/aws-encryption-sdk-cli-2.1.0.tar.gz

PyInterpreterState_GetID

Yes.

It's part of the limited C API. IMO technically, it's ok to add to the stable ABI. I added it to PC/python3dll.c on purpose. Sadly, it's not available on PyPy, but I don't think that the stable ABI must care about PyPy which is a different use case.

encukou · 2021-04-16T21:02:10Z

Thanks! I'll send PRs later.

encukou · 2021-04-20T12:07:25Z

Thanks for linking to the issues/documentation!

I see that Py_GetArgcArgv it was exported in python3.def (i.e. stable ABI) in bpo-40910, which got backported to 3.9.

And the removal of PyThreadState_DeleteCurrent was reverted: bpo-38266

While we would like to deprecate/remove them in the future, they are now part of the stable ABI.
They are now in internal headers, so I will exclude them from the limited API.

erlend-aasland · 2021-04-20T13:49:30Z

Should we mention PEP-652 in Include/README.rst, now that the PEP is accepted?

encukou · 2021-04-20T13:52:18Z

Should we mention PEP-652 in Include/README.rst, now that the PEP is accepted?

No, we should link to the documentation (when it's written). The PEP is a design document; it'll become outdated.

encukou · 2021-04-23T12:14:07Z

New changeset e7cc64e by Petr Viktorin in branch 'master':
bpo-43795: PEP-652: Simplify headers for easier static analysis (GH-25483)
e7cc64e

encukou · 2021-04-23T12:18:03Z

New changeset 9d6a2d0 by Petr Viktorin in branch 'master':
bpo-43795: PEP-652: Clean up the stable ABI/limited API (GH-25482)
9d6a2d0

vstinner · 2021-04-28T15:15:19Z

Right now, running "make regen-limited-abi" adds again functions which were removed from Doc/data/stable_abi.dat:
#25687 (comment)

diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat
index cdc7160250..6fe61743d6 100644
--- a/Doc/data/stable_abi.dat
+++ b/Doc/data/stable_abi.dat
@@ -351,11 +351,17 @@ PyMapping_Length
 PyMapping_SetItemString
 PyMapping_Size
 PyMapping_Values
+PyMarshal_ReadObjectFromString
+PyMarshal_WriteLongToFile
+PyMarshal_WriteObjectToFile
+PyMarshal_WriteObjectToString
 PyMem_Calloc
 PyMem_Free
 PyMem_Malloc
 PyMem_Realloc
 PyMemberDescr_Type
+PyMember_GetOne
+PyMember_SetOne
 PyMemoryView_FromMemory
 PyMemoryView_FromObject
 PyMemoryView_GetContiguous

I guess that you should not be added until this issue is solved.

encukou · 2021-04-28T15:37:59Z

I hope the PR fixes that. I plan to merge tomorrow if there ar no objections to it.

On April 28, 2021 5:15:19 PM GMT+02:00, STINNER Victor <report@bugs.python.org> wrote:

STINNER Victor <vstinner@python.org> added the comment:

Right now, running "make regen-limited-abi" adds again functions which
were removed from Doc/data/stable_abi.dat:
#25687 (comment)

diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat
index cdc7160250..6fe61743d6 100644
--- a/Doc/data/stable_abi.dat
+++ b/Doc/data/stable_abi.dat
@@ -351,11 +351,17 @@ PyMapping_Length
PyMapping_SetItemString
PyMapping_Size
PyMapping_Values
+PyMarshal_ReadObjectFromString
+PyMarshal_WriteLongToFile
+PyMarshal_WriteObjectToFile
+PyMarshal_WriteObjectToString
PyMem_Calloc
PyMem_Free
PyMem_Malloc
PyMem_Realloc
PyMemberDescr_Type
+PyMember_GetOne
+PyMember_SetOne
PyMemoryView_FromMemory
PyMemoryView_FromObject
PyMemoryView_GetContiguous

I guess that you should not be added until this issue is solved.

----------

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43795\>

encukou · 2021-04-29T09:33:23Z

New changeset f6ee4da by Petr Viktorin in branch 'master':
bpo-43795: Generate python3dll.c and doc data from manifest (PEP-652) (GH-25315)
f6ee4da

miss-islington · 2021-05-05T18:15:34Z

New changeset 8f3ef45 by Miss Islington (bot) in branch '3.10':
bpo-43795: Mark PyCodec_Unregister as a function, not data, in stable ABI (GH-25920)
8f3ef45

pablogsal · 2021-05-12T03:31:28Z

Unfortunately commit b05955d has broken all refleak buildbots:

https://buildbot.python.org/all/#/builders/259/builds/25/steps/5/logs/stdio

See example:

----------------------------------------------------------------------
Ran 53 tests in 5.629s
OK (skipped=1)
......
test_tools leaked [251, 251, 251] references, sum=753
test_tools leaked [91, 91, 91] memory blocks, sum=273
1 test failed again:
test_tools

Per our buildbot policy, it will be reverted if is not fixed in 24h to avoid masking other errors.

encukou · 2021-05-12T09:58:23Z

Thanks for the note!

This is due to csv. Adding a script in Tools/scripts that imports csv will also make the refleak check fail.
I'll investigate further.

encukou · 2021-05-12T13:36:39Z

I opened https://bugs.python.org/issue44116 for the leak when trying to unload _csv.
Looks like investigating that will take a while, so I'll send a small PR to unblock the buildbots.

willingc · 2021-05-14T05:29:13Z

New changeset 3739371 by Miss Islington (bot) in branch '3.10':
bpo-43795: PEP-652 user documentation (GH-25668) (GH-26034)
3739371

pablogsal · 2021-05-25T11:59:26Z

New changeset 68b9c20 by Miss Islington (bot) in branch '3.10':
bpo-43795: Remove Py_FrozenMain from the Limited API & Stable ABI (GH-26241) (GH-26353)
68b9c20

encukou · 2021-06-08T15:20:15Z

New changeset 257e400 by Petr Viktorin in branch 'main':
bpo-43795: Note Stable ABI PEP in What's New (GH-26479)
257e400

encukou · 2021-06-08T16:00:40Z

New changeset 7518556 by Miss Islington (bot) in branch '3.10':
bpo-43795: Note Stable ABI PEP in What's New (GH-26479) (GH-26603)
7518556

encukou · 2021-06-16T09:22:54Z

New changeset 7cad9cb by Petr Viktorin in branch 'main':
bpo-43795: Don't list private names in the limited API (GH-26740)
7cad9cb

miss-islington · 2021-06-16T09:53:24Z

New changeset 7fd4010 by Miss Islington (bot) in branch '3.10':
bpo-43795: Don't list private names in the limited API (GH-26740)
7fd4010

encukou · 2021-10-22T08:12:11Z

New changeset 276468d by Petr Viktorin in branch 'main':
bpo-43795: Add a test for Stable ABI symbol availability using ctypes (GH-26354)
276468d

ambv · 2021-12-10T23:34:51Z

New changeset 98e506a by Petr Viktorin in branch 'main':
bpo-43795: Document stable_abi.txt format and contents (GH-29956)
98e506a

encukou added 3.10 labels Apr 9, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

python / cpython Public

Implement PEP 652 -- Maintaining the Stable ABI #87961

Implement PEP 652 -- Maintaining the Stable ABI #87961

encukou commented Apr 9, 2021

encukou commented Apr 9, 2021

encukou commented Apr 14, 2021

encukou commented Apr 16, 2021

encukou commented Apr 16, 2021

vstinner commented Apr 16, 2021

encukou commented Apr 16, 2021

encukou commented Apr 20, 2021

erlend-aasland commented Apr 20, 2021

encukou commented Apr 20, 2021

encukou commented Apr 23, 2021

encukou commented Apr 23, 2021

vstinner commented Apr 28, 2021

encukou commented Apr 28, 2021

encukou commented Apr 29, 2021

miss-islington commented May 5, 2021

pablogsal commented May 12, 2021

encukou commented May 12, 2021

encukou commented May 12, 2021

willingc commented May 14, 2021

pablogsal commented May 25, 2021

encukou commented Jun 8, 2021

encukou commented Jun 8, 2021

encukou commented Jun 16, 2021

miss-islington commented Jun 16, 2021

encukou commented Oct 22, 2021

ambv commented Dec 10, 2021

python / cpython Public

Implement PEP 652 -- Maintaining the Stable ABI #87961

Implement PEP 652 -- Maintaining the Stable ABI #87961

Comments

encukou commented Apr 9, 2021

encukou commented Apr 9, 2021

encukou commented Apr 14, 2021

encukou commented Apr 16, 2021

encukou commented Apr 16, 2021

vstinner commented Apr 16, 2021

encukou commented Apr 16, 2021

encukou commented Apr 20, 2021

erlend-aasland commented Apr 20, 2021

encukou commented Apr 20, 2021

encukou commented Apr 23, 2021

encukou commented Apr 23, 2021

vstinner commented Apr 28, 2021

encukou commented Apr 28, 2021

encukou commented Apr 29, 2021

miss-islington commented May 5, 2021

pablogsal commented May 12, 2021

encukou commented May 12, 2021

encukou commented May 12, 2021

willingc commented May 14, 2021

pablogsal commented May 25, 2021

encukou commented Jun 8, 2021

encukou commented Jun 8, 2021

encukou commented Jun 16, 2021

miss-islington commented Jun 16, 2021

encukou commented Oct 22, 2021

ambv commented Dec 10, 2021