[C API] PEP 741: Add stable ABI to customize Python initialization #107954

vstinner · 2023-08-14T22:57:37Z

Links to previous discussion of this feature:

https://discuss.python.org/t/fr-allow-private-runtime-config-to-enable-extending-without-breaking-the-pyconfig-abi/18004

Summary of proposal:

Add a text-based configuration to Python initialization

Pitch:

An API should be added to the limited C API customize the Python initialization (PyConfig API, PEP 587).

The problem is that the PyConfig C API is excluded from the limited C API, whereas the legacy initialization API is deprecated (ex: Py_VerboseFlag in Python 3.12) and being removed in Python 3.13 (ex: PySys_SetPath()).

Linked PRs

LostTime76 · 2023-08-15T02:26:10Z

As discussed in discourse, I do not think the text file is the best option for embedding. Depending on the embedding scenario, I think we will run into issues of finding the file due to how python is currently resolving paths. I have a very specific scenario that may be solved in easier ways. Even if you by chance get python to find the file in my particular scenario, I don't want the user of my embedded python exe to have to specify package paths that my exe should be able to set itself.

Its not as easy as putting the file in the same directory as the embedded exe or the main script. Take the scenario that the embedding exe is using the global python install or the embedding exe is launched as a child process from an external process where the process path is not the path of the embedded exe.

I think there may be much easier ways to solve my specific scenario. I would like to use as few APIs as possible and not duplicate how python is already parsing the options from the command line.

What about adding additional command line options that can be set from argv? If we could set additional package paths from argv, it would allow us to directly point to where we need to without hassle.

Environment variables do not work nicely in the embedded scenario, even though we have PYTHONPATH, because there is no way to cleanly set an environment variable from an already running process to be accessible from a loaded python dll.

vstinner · 2023-08-15T12:25:23Z

I'm thinking about a function taking a string, not a file.

Add a specification of the PyConfig structure to factorize the code.

Add PyInitConfig functions: * PyInitConfig_Python_New() * PyInitConfig_Isolated_New() * PyInitConfig_Free(config) * PyInitConfig_SetInt(config, key, value) * PyInitConfig_SetString(config, key, value) * PyInitConfig_SetList(config, key, length, items) * PyInitConfig_GetErrorMsg(config) Add also functions using it: * Py_InitializeFromInitConfig(config) * Py_ExitWithInitConfig(config) Changes: * Add Include/initconfig.h header.

Add PyInitConfig functions: * PyInitConfig_Python_New() * PyInitConfig_Isolated_New() * PyInitConfig_Free(config) * PyInitConfig_SetInt(config, key, value) * PyInitConfig_SetStr(config, key, value) * PyInitConfig_SetWStr(config, key, value) * PyInitConfig_SetStrList(config, key, length, items) * PyInitConfig_SetWStrList(config, key, length, items) * PyInitConfig_GetErrorMsg(config) Add also functions using it: * Py_InitializeFromInitConfig(config) * Py_ExitWithInitConfig(config) Changes: * Add Include/initconfig.h header.

Add PyInitConfig functions: * PyInitConfig_Python_New() * PyInitConfig_Isolated_New() * PyInitConfig_Free(config) * PyInitConfig_SetInt(config, key, value) * PyInitConfig_SetStr(config, key, value) * PyInitConfig_SetWStr(config, key, value) * PyInitConfig_SetStrList(config, key, length, items) * PyInitConfig_SetWStrList(config, key, length, items) * PyInitConfig_GetErrorMsg(config) Add also functions using it: * Py_InitializeFromInitConfig(config) * Py_ExitWithInitConfig(config) Add these 11 functions to the limited C API. Changes: * Add Include/initconfig.h header.

Add PyInitConfig functions: * PyInitConfig_Python_New() * PyInitConfig_Isolated_New() * PyInitConfig_Free(config) * PyInitConfig_SetInt(config, key, value) * PyInitConfig_SetStr(config, key, value) * PyInitConfig_SetWStr(config, key, value) * PyInitConfig_SetStrList(config, key, length, items) * PyInitConfig_SetWStrList(config, key, length, items) * PyInitConfig_GetErrorMsg(config) Add also functions using it: * Py_InitializeFromInitConfig(config) * Py_ExitWithInitConfig(config) * Add functions to get runtime config: * PyConfig_GetInt() * PyConfig_GetStr() * PyConfig_GetStrList() Add these functions to the limited C API. Changes: * Add Doc/c-api/config.rst. * Add Include/initconfig.h header. * Add Modules/_testcapi/config.c * Add Lib/test/test_capi/test_config.py.

Add PyInitConfig functions: * PyInitConfig_Python_New() * PyInitConfig_Isolated_New() * PyInitConfig_Free(config) * PyInitConfig_SetInt(config, key, value) * PyInitConfig_SetStr(config, key, value) * PyInitConfig_SetWStr(config, key, value) * PyInitConfig_SetStrList(config, key, length, items) * PyInitConfig_SetWStrList(config, key, length, items) * PyInitConfig_Exception(config) * PyInitConfig_GetError(config, &err_msg) * PyInitConfig_GetExitCode(config, &exitcode) Add also functions using it: * Py_InitializeFromInitConfig(config) * Py_ExitWithInitConfig(config) * Add functions to get runtime config: * PyConfig_GetInt() * PyConfig_GetStr() * PyConfig_GetStrList() Add these functions to the limited C API. Changes: * Add Doc/c-api/config.rst. * Add Include/initconfig.h header. * Add Modules/_testcapi/config.c * Add Lib/test/test_capi/test_config.py.

Add PyConfig_Get() and PyConfig_GetInt() functions to get the current Python configuration.

Add PyConfig_Get() and PyConfig_GetInt() functions to get the current Python configuration. _PyConfig_AsDict() now converts PyConfig.xoptions as a dictionary.

vstinner · 2023-12-01T21:12:41Z

I wrote PR #112609 to read configuration options such as "verbose" or "argv" as a C int or as a Python object.

Add PyInitConfig functions: * PyInitConfig_Python_New() * PyInitConfig_Isolated_New() * PyInitConfig_Free(config) * PyInitConfig_SetInt(config, key, value) * PyInitConfig_SetStr(config, key, value) * PyInitConfig_SetWStr(config, key, value) * PyInitConfig_SetStrList(config, key, length, items) * PyInitConfig_SetWStrList(config, key, length, items) * PyInitConfig_Exception(config) * PyInitConfig_GetError(config, &err_msg) * PyInitConfig_GetExitCode(config, &exitcode) Add also functions using it: * Py_InitializeFromInitConfig(config) * Py_ExitWithInitConfig(config) Add these functions to the limited C API. Changes: * Add Doc/c-api/config.rst. * Add Include/initconfig.h header. * Add Modules/_testcapi/config.c * Add Lib/test/test_capi/test_config.py.

Add PyInitConfig functions: * PyInitConfig_Python_New() * PyInitConfig_Isolated_New() * PyInitConfig_Free(config) * PyInitConfig_SetInt(config, key, value) * PyInitConfig_SetStr(config, key, value) * PyInitConfig_SetWStr(config, key, value) * PyInitConfig_SetStrList(config, key, length, items) * PyInitConfig_SetWStrList(config, key, length, items) * PyInitConfig_Exception(config) * PyInitConfig_GetError(config, &err_msg) * PyInitConfig_GetExitCode(config, &exitcode) Add also functions using it: * Py_InitializeFromInitConfig(config) * Py_ExitWithInitConfig(config) Add these functions to the limited C API. Changes: * Add Doc/c-api/config.rst. * Add Include/initconfig.h header.

vstinner · 2023-12-20T13:05:21Z

I close the issue: #110176 (comment)

milianw · 2024-01-16T15:34:09Z

Hey @vstinner - IIUC then there's still no non-deprecated API in the limited C API to customize the initialization, right? Can you then please reopen this task to indicate that this? Or should I report a separate issue to track this?

Thank you

vstinner · 2024-01-16T15:41:22Z

Can you then please reopen this task to indicate that this?

Ok, I reopen the issue.

vstinner · 2024-01-24T16:48:08Z

Hey @vstinner - IIUC then there's still no non-deprecated API in the limited C API to customize the initialization, right? Can you then please reopen this task to indicate that this? Or should I report a separate issue to track this?

I wrote PEP 741 – Python Configuration C API. Can you please elaborate your use case to motivate the usage of the limited C API to customize the Python initialization, rather than using the non-limited PyConfig C API? I was asked for such rationale.

milianw · 2024-01-26T08:59:12Z

We have a large old code base with an embedded python/jupyter console in a graphical application.

We call Py_SetProgramName and Py_SetPythonHome - though to be frank our test suite seems to pass even without calls to these two functions, but maybe not in a packaged environment without any system python available, I have to double check that - but I believe this was the initial idea - to tell python where to look for its bundled libraries in a subfolder of our application directory.

Add PyConfig_Get() and PyConfig_GetInt() functions to get the current Python configuration. _PyConfig_AsDict() now converts PyConfig.xoptions as a dictionary.

_PyConfig_AsDict() now returns bool objects for options using the new PyConfig_MEMBER_BOOL type. Update tests for these changes.

vstinner · 2024-03-07T17:34:59Z

See also #99872 "Allow efficient read access to PyConfig options".

Add PyInitConfig functions: * PyInitConfig_Python_New() * PyInitConfig_Isolated_New() * PyInitConfig_Free(config) * PyInitConfig_SetInt(config, key, value) * PyInitConfig_SetStr(config, key, value) * PyInitConfig_SetWStr(config, key, value) * PyInitConfig_SetStrList(config, key, length, items) * PyInitConfig_SetWStrList(config, key, length, items) * PyInitConfig_Exception(config) * PyInitConfig_GetError(config, &err_msg) * PyInitConfig_GetExitCode(config, &exitcode) Add also functions using it: * Py_InitializeFromInitConfig(config) * Py_ExitWithInitConfig(config) Add these functions to the limited C API. Changes: * Add Doc/c-api/config.rst. * Add Include/initconfig.h header.

…n#116359) _PyConfig_AsDict() now returns bool objects for options using the new PyConfig_MEMBER_BOOL type. Update tests for these changes.

Add PyInitConfig functions: * PyInitConfig_Python_New() * PyInitConfig_Isolated_New() * PyInitConfig_Free(config) * PyInitConfig_SetInt(config, key, value) * PyInitConfig_SetStr(config, key, value) * PyInitConfig_SetWStr(config, key, value) * PyInitConfig_SetStrList(config, key, length, items) * PyInitConfig_SetWStrList(config, key, length, items) * PyInitConfig_Exception(config) * PyInitConfig_GetError(config, &err_msg) * PyInitConfig_GetExitCode(config, &exitcode) Add also functions using it: * Py_InitializeFromInitConfig(config) * Py_ExitWithInitConfig(config) Add these functions to the limited C API. Changes: * Add Doc/c-api/config.rst. * Add Include/initconfig.h header.

vstinner added type-feature A feature request or enhancement topic-C-API labels Aug 14, 2023

rokm mentioned this issue Aug 16, 2023

Configure and initialize python via PEP 587 API pyinstaller/pyinstaller#7847

Merged

vstinner mentioned this issue Sep 4, 2023

Issues with the Stable ABI capi-workgroup/problems#4

Open

vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023

pythongh-107954: Add _PyConfig_Parse()

fa98a1c

bedevere-app bot mentioned this issue Sep 30, 2023

[WIP] gh-107954: Add _PyConfig_Parse() #110145

Closed

vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023

pythongh-107954: Refactor initconfig.c, add CONFIG_SPEC

01fcc4c

Add a specification of the PyConfig structure to factorize the code.

vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023

pythongh-107954: Refactor initconfig.c: add CONFIG_SPEC

92d80d6

Add a specification of the PyConfig structure to factorize the code.

vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023

pythongh-107954: Refactor initconfig.c: add CONFIG_SPEC

c09b9a4

Add a specification of the PyConfig structure to factorize the code.

bedevere-app bot mentioned this issue Sep 30, 2023

gh-107954: Refactor initconfig.c: add CONFIG_SPEC #110146

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023

pythongh-107954: Add _PyConfig_Parse()

96cb693

vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023

pythongh-107954: Refactor initconfig.c: add CONFIG_SPEC

7191629

Add a specification of the PyConfig structure to factorize the code.

vstinner added a commit that referenced this issue Sep 30, 2023

gh-107954: Refactor initconfig.c: add CONFIG_SPEC (#110146)

f3bb00e

Add a specification of the PyConfig structure to factorize the code.

vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023

pythongh-107954: Add _PyConfig_Parse()

e6e4e5c

bedevere-app bot mentioned this issue Oct 1, 2023

[PEP 741] gh-107954: Add PyInitConfig C API #110176

Closed

vstinner added a commit to vstinner/cpython that referenced this issue Dec 1, 2023

pythongh-107954: Add PyConfig_Get() function

fba1c0f

vstinner added a commit to vstinner/cpython that referenced this issue Dec 1, 2023

pythongh-107954: Add PyConfig_Get() function

30af159

vstinner added a commit to vstinner/cpython that referenced this issue Dec 1, 2023

pythongh-107954: Add PyConfig_Get() function

3bc322f

Add PyConfig_Get() and PyConfig_GetInt() functions to get the current Python configuration.

bedevere-app bot mentioned this issue Dec 1, 2023

[PEP 741] gh-107954: Add PyConfig_Get() function #112609

Closed

vstinner closed this as completed Dec 20, 2023

vstinner mentioned this issue Jan 16, 2024

[C API] Deprecate legacy API for configure Python initialization #88279

Closed

vstinner reopened this Jan 16, 2024

vstinner mentioned this issue Jan 16, 2024

Should we make it hard for 3rd parties to use private functions? capi-workgroup/decisions#7

Closed

vstinner changed the title ~~[C API] No limited C API to customize Python initialization (PyConfig, PEP 587)~~ [C API] PEP 741: No limited C API to customize Python initialization (PyConfig, PEP 587) Jan 24, 2024

vstinner changed the title ~~[C API] PEP 741: No limited C API to customize Python initialization (PyConfig, PEP 587)~~ [C API] PEP 741: Add stable ABI to customize Python initialization (PyConfig, PEP 587) Feb 15, 2024

vstinner changed the title ~~[C API] PEP 741: Add stable ABI to customize Python initialization (PyConfig, PEP 587)~~ [C API] PEP 741: Add stable ABI to customize Python initialization Feb 15, 2024

vstinner added a commit to vstinner/cpython that referenced this issue Mar 5, 2024

pythongh-107954: Add PyConfig_MEMBER_BOOL type to PyConfigSpec

87fadde

_PyConfig_AsDict() now returns bool objects for options using the new PyConfig_MEMBER_BOOL type. Update tests for these changes.

bedevere-app bot mentioned this issue Mar 5, 2024

gh-107954: Add PyConfig_MEMBER_BOOL type to PyConfigSpec #116359

Merged

vstinner added a commit that referenced this issue Mar 6, 2024

gh-107954: Add PyConfig_MEMBER_BOOL type to PyConfigSpec (#116359)

2b37996

_PyConfig_AsDict() now returns bool objects for options using the new PyConfig_MEMBER_BOOL type. Update tests for these changes.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[C API] PEP 741: Add stable ABI to customize Python initialization #107954

[C API] PEP 741: Add stable ABI to customize Python initialization #107954

vstinner commented Aug 14, 2023 •

edited by bedevere-app bot

LostTime76 commented Aug 15, 2023 •

edited

vstinner commented Aug 15, 2023

vstinner commented Dec 1, 2023

vstinner commented Dec 20, 2023

milianw commented Jan 16, 2024

vstinner commented Jan 16, 2024

vstinner commented Jan 24, 2024

milianw commented Jan 26, 2024

vstinner commented Mar 7, 2024

[C API] PEP 741: Add stable ABI to customize Python initialization #107954

[C API] PEP 741: Add stable ABI to customize Python initialization #107954

Comments

vstinner commented Aug 14, 2023 • edited by bedevere-app bot

Links to previous discussion of this feature:

Summary of proposal:

Pitch:

Linked PRs

LostTime76 commented Aug 15, 2023 • edited

vstinner commented Aug 15, 2023

vstinner commented Dec 1, 2023

vstinner commented Dec 20, 2023

milianw commented Jan 16, 2024

vstinner commented Jan 16, 2024

vstinner commented Jan 24, 2024

milianw commented Jan 26, 2024

vstinner commented Mar 7, 2024

vstinner commented Aug 14, 2023 •

edited by bedevere-app bot

LostTime76 commented Aug 15, 2023 •

edited