Skip to content

/ cpython

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bpo-43244: Remove ast.h, asdl.h, Python-ast.h headers #24933

Merged
merged 1 commit into from Mar 23, 2021
Merged

bpo-43244: Remove ast.h, asdl.h, Python-ast.h headers #24933

merged 1 commit into from Mar 23, 2021
+931 −933

Conversation

@vstinner
Copy link
Member

@vstinner vstinner commented Mar 19, 2021 •
edited

These fuctions were undocumented and excluded from the limited C API.

Most names defined by these header files were not prefixed by "Py"
and so could create names conflicts. For example, Python-ast.h
defined a "Yield" macro which was conflict with the "Yield" name used
by the Windows <winbase.h> header.

Use the Python ast module instead.

  • Move Include/asdl.h to Include/internal/pycore_asdl.h.
  • Move Include/Python-ast.h to Include/internal/pycore_ast.h.
  • Remove ast.h header file.
  • pycore_symtable.h no longer includes Python-ast.h.

https://bugs.python.org/issue43244
@vstinner
Copy link
Member Author

@vstinner vstinner commented Mar 19, 2021

I searched for asdl.h in the top PyPI 4000 projects, I found 3 projects using it: 2 can be ignored, the only interesting one is typed_ast. IMO typed_ast should not block this PR.

typed_ast-1.4.2.tar.gz: this project contains copies of C and header files of CPython.

ast3/Include/Python-ast.h
ast3/Include/asdl.h
ast3/Include/ast.h
ast3/Include/bitset.h
ast3/Include/errcode.h
ast3/Include/graminit.h
ast3/Include/grammar.h
ast3/Include/node.h
ast3/Include/parsetok.h
ast3/Include/pgenheaders.h
ast3/Include/token.h
ast3/Parser/acceler.c
ast3/Parser/bitset.c
ast3/Parser/grammar.c
ast3/Parser/grammar1.c
ast3/Parser/node.c
ast3/Parser/parser.c
ast3/Parser/parser.h
ast3/Parser/parsetok.c
ast3/Parser/tokenizer.c
ast3/Parser/tokenizer.h
ast3/Python/Python-ast.c
ast3/Python/asdl.c
ast3/Python/ast.c
ast3/Python/graminit.c

This project can be modified to copy Python 3.10 internal header files and be built with Py_BUILD_CORE_MODULE macro defined.

Pygments-2.8.0.tar.gz: it's used in example files to test the project, these files are not built, Pygments colorizes text files.

./tests/examplefiles/make/Makefile:		Include/asdl.h \
./tests/examplefiles/make/Makefile.output:'Include/asdl.h' Text

odfpy-1.4.1.tar.gz: the tarball contains many virtual environment in an hidden .tox/ subdirectory.
@vstinner
Copy link
Member Author

@vstinner vstinner commented Mar 19, 2021

I searched for ast.h using \bast\.h\b regex and for Python-ast.h in the top PyPI 4000 projects, I found:

  • the 3 same projects than when I search for asdl.h (typed_ast, Pygments, odfpy)
  • jsonnet which contains its own ast.h file: core/ast.h

In short, in the 4000 projects, typed_ast should be the only project impacted by this PR.

typed_ast copies/paste code directly from CPython, so I'm not surprised that it's impacted by changes in the CPython ASDL and AST implementation :-) typed_ast can continue copying code from CPython, but may need to be built with Py_BUILD_CORE_MODULE macro defined.
@vstinner
Copy link
Member Author

@vstinner vstinner commented Mar 19, 2021

Ah, test_asyncio failed on the Windows x64 CI, known random test failure: https://bugs.python.org/issue43539
@pablogsal pablogsal requested a review from gvanrossum Mar 19, 2021
@vstinner
vstinner reviewed Mar 19, 2021
View changes
Copy link
Member Author

@vstinner vstinner left a comment

pycore_ast.h now contains definitions of functions defined in Python-ast.c and in ast.c. Is that a good thing, or do you prefer to keep two header files separated? I failed to find a better name of the generated header file (old Python-ast.h).
@shihai1991
Copy link
Member

@shihai1991 shihai1991 commented Mar 20, 2021

pycore_ast.h now contains definitions of functions defined in Python-ast.c and in ast.c. Is that a good thing, or do you prefer to keep two header files separated? I failed to find a better name of the generated header file (old Python-ast.h).

Hm, Can we merge pycore_ast_state.h in pycore_ast.h? There just only have a ast_state in pycore_ast_state.h.
@vstinner
Copy link
Member Author

@vstinner vstinner commented Mar 20, 2021

Hm, Can we merge pycore_ast_state.h in pycore_ast.h? There just only have a ast_state in pycore_ast_state.h.

Yes, but I would prefer to not do that.

pycore_ast_state.h is included by pycore_interp.h and this header file is included by many C files. I would prefer to avoid including pycore_ast.h in C files since this header file is problematic, for example it requires #undef Yield to avoid conflict with <winbase.h>.
@vstinner vstinner force-pushed the vstinner:internal_pycore_ast branch from e68284f to 93680ee Mar 22, 2021
@vstinner
Copy link
Member Author

@vstinner vstinner commented Mar 22, 2021

PR rebased on master to fix a conflict on compile.h.
@vstinner vstinner force-pushed the vstinner:internal_pycore_ast branch from 93680ee to 3058baf Mar 22, 2021
@vstinner
Copy link
Member Author

@vstinner vstinner commented Mar 22, 2021

Removed functions are undocumented. They are also excluded from the limited C API (and from the stable ABI).

I updated the PR to mention that explicitly.

I cannot find removed functions in Doc/c-api/:

  • asdl.h: asdl
  • Python-ast.h: PyAST_obj2mod, PyAST_Check, PyAST_mod2obj, _Py_Module, mod_ty
  • ast.h: PyAST_Validate, PyAST_FromNode, PyAST_FromNodeObject, _PyAST_ExprAsUnicode, _PyAST_GetDocString
@vstinner
bpo-43244: Remove ast.h, asdl.h, Python-ast.h headers
Loading status checks…
4c68ac5 
These functions were undocumented and excluded from the limited C
API.

Most names defined by these header files were not prefixed by "Py"
and so could create names conflicts. For example, Python-ast.h
defined a "Yield" macro which was conflict with the "Yield" name used
by the Windows <winbase.h> header.

Use the Python ast module instead.

* Move Include/asdl.h to Include/internal/pycore_asdl.h.
* Move Include/Python-ast.h to Include/internal/pycore_ast.h.
* Remove ast.h header file.
* pycore_symtable.h no longer includes Python-ast.h.
@vstinner vstinner force-pushed the vstinner:internal_pycore_ast branch from 3058baf to 4c68ac5 Mar 23, 2021
@vstinner
Copy link
Member Author

@vstinner vstinner commented Mar 23, 2021

Oops, I fixed a typo: fuctions => functions.
@vstinner vstinner merged commit 94faa07 into python:master Mar 23, 2021
11 checks passed
11 checks passed
@github-actions
Docs
Details
@github-actions
Check for source changes
Details
@github-actions
Check if generated files are up to date
Details
@github-actions
Windows (x86)
Details
@github-actions
Windows (x64) Windows (x64)
Details
@github-actions
macOS
Details
@github-actions
Ubuntu
Details
Azure Pipelines PR #20210323.33 succeeded
Details
@travis-ci
Travis CI - Pull Request Build Passed
Details
@bedevere-bot
bedevere/issue-number Issue number 43244 found
Details
@bedevere-bot
bedevere/news News entry found in Misc/NEWS.d
@vstinner vstinner deleted the vstinner:internal_pycore_ast branch Mar 23, 2021
@vstinner vstinner mentioned this pull request Mar 23, 2021
Merged
kreathon added a commit to kreathon/cpython that referenced this pull request May 2, 2021
@vstinner @kreathon
bpo-43244: Remove ast.h, asdl.h, Python-ast.h headers (pythonGH-24933)
c9c96fc 
These functions were undocumented and excluded from the limited C
API.

Most names defined by these header files were not prefixed by "Py"
and so could create names conflicts. For example, Python-ast.h
defined a "Yield" macro which was conflict with the "Yield" name used
by the Windows <winbase.h> header.

Use the Python ast module instead.

* Move Include/asdl.h to Include/internal/pycore_asdl.h.
* Move Include/Python-ast.h to Include/internal/pycore_ast.h.
* Remove ast.h header file.
* pycore_symtable.h no longer includes Python-ast.h.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@brettcannon brettcannon

@encukou encukou

@ericsnowcurrently ericsnowcurrently

@isidentical isidentical

@lysnikolaou lysnikolaou

@ncoghlan ncoghlan

@pablogsal pablogsal

@warsaw warsaw

@python/windows-team windows-team

@gvanrossum gvanrossum

Assignees
No one assigned
Labels
CLA signed
Projects
None yet
Milestone
No milestone
4 participants
@vstinner @shihai1991 @the-knights-who-say-ni @bedevere-bot