bpo-35975: Support parsing earlier minor versions of Python 3 #12086

gvanrossum · 2019-02-28T00:44:51Z

This adds a feature_version flag to ast.parse() (documented) and compile() (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if feature_version is 5 or 6, the hacks for the async and await keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than NAME tokens that the parser driver recognizes.)

https://bugs.python.org/issue35975

gvanrossum · 2019-02-28T04:08:08Z

I could use some help finding the cause of the one remaining test failure -- test_parser.py computes the size of tree nodes differently somehow. [UPDATE: Fixed it, I think.]

gvanrossum · 2019-02-28T23:56:28Z

The main thing left to do is adding tests. There are some tests in typed_ast that I can migrate over. But first I think we should complete the debate over on bpo about whether this is desirable at all.

ilevkivskyi

Thanks! This looks very clean, I just have few minor suggestions.

(I also there is a merge conflict now.)

ilevkivskyi · 2019-03-03T00:11:29Z

Lib/ast.py

-    return compile(source, filename, mode, flags)
+    return compile(source, filename, mode, flags,
+                   dont_inherit=False,
+                   optimize=-1,


Why we now need to pass these two extra arguments? Maybe add a comment?

Hm, those are the default values that I somehow copied when cleaning this up. I'll remove them -- we just need feature_version.

ilevkivskyi · 2019-03-03T00:11:29Z

Python/ast.c

+        /* Async comprehensions only allowed in Python 3.6 and greater */
+        if (is_async && c->c_feature_version < 6) {
+            ast_error(c, n,
+                    "Async comprehensions are only supported in Python 3.6 and greater");


Indentation here and in several other places is a bit unusual, I would rather justify it after (.

OK, fixed. My guess is that at some point in the past that function had had a name that was 2 characters shorter. :-)

ilevkivskyi · 2019-03-03T00:11:29Z

Python/ast.c

@@ -1746,6 +1758,12 @@ ast_for_funcdef_impl(struct compiling *c, const node *n0,
    node *tc;
    string type_comment = NULL;

+    if (is_async && c->c_feature_version < 5) {
+        ast_error(c, n,
+                  "Async functions are only supported in Python 3.5 and greater");


Here indentation looks good.

Visible behavior: - Issue error for `X @ Y` (matrix multiply) if c_feature_version < 5 - Add optional feature_version kw arg to ast.parse() (default -1 which implies PY_MINOR_VERSION) - Add feature_version: int = -1 to compile() (via Argument Clinic); this sets cf_feature_version to the given value if >= 0, else defaults to PY_MINOR_VERSION Implementation: - Add PyAST_obj2mod_ex(): like PyAST_obj2mod() but with feature_version arg; the latter calls the former with PY_MINOR_VERSION - Add cf_feature_version to PyCompilerFlags structure; initialized to PY_MINOR_VERSION everywhere - Add c_feature_version to struct compiling; initialize from cf_feature_version - Add 'c' argument to get_operator() - In builtin eval() and exec(), default to PY_MINOR_VERSION TODO: - Put version-dependent ASYNC/AWAIT keyword scanning back - Reject async functions, await expressions, and async for/with in minor versions < 5 - Reject async comprehensions in minor versions < 6 - Reject underscores in numeric literals in minor versions < 6 - Reject variable annotations in minor versions < 6 - Reject `X @= Y` in minor versions < 5

This is everything currently in typeshed except await expressions (but it does reject async functions etc.): - Reject async functions and async for/with in minor versions < 5 - Reject async comprehensions in minor versions < 6 - Reject underscores in numeric literals in minor versions < 6 - Reject variable annotations in minor versions < 6 - Reject `X @= Y` in minor versions < 5

This adds: - Add ASYNC/AWAIT tokens back to Grammar and regenerate - Recognize async/await keywords conditionally if feature_version < 7 - Reject await expressions if feature_version < 5 - Docs for ASYNC/AWAIT tokens and for ast.parse(..., feature_version=N)

The PyST_Object header in parsermodule.c became one int larger because it contains a PyCompilerFlags struct, which grew extra space for the st_feature_version field. Took me long enough!

… not regular keywords

gvanrossum · 2019-03-05T19:38:27Z

OK, this is ready for final review and merge. (Sorry that the rebase lost some of the review history, I'm used to different tooling.)

gvanrossum · 2019-03-05T19:50:12Z

(Well, I promised tests. Upcoming. Docs are already done.)

ilevkivskyi

Thanks for the updates! All looks good, I have few optional suggestions.

ilevkivskyi · 2019-03-07T14:06:10Z

Doc/library/ast.rst

@@ -145,13 +145,18 @@ and classes for traversing abstract syntax trees:
   modified to correspond to :pep:`484` "signature type comments",
   e.g. ``(str, int) -> List[str]``.

+   Also, setting ``feature_version`` to the minor version of an
+   earlier Python 3 version will attempt to parse using that version's
+   grammar.  For example, setting ``feature_version=4`` will allow


Maybe add that 4 is the lowest supported value?

ilevkivskyi · 2019-03-07T14:06:10Z

Lib/test/test_type_comments.py

+        return ast.parse(source, type_comments=True,
+                         feature_version=feature_version)
+
+    def parses(self, source, minver=lowest, maxver=highest, expected_regex=""):


parses is not very descriptive and is easy to confuse with parse, maybe parse_all, or parse_all_versions?

ilevkivskyi · 2019-03-07T14:06:10Z

Lib/test/test_type_comments.py

+fstring = """\
+a = 42
+f"{a}"
+"""


Maybe add underscores in numeric literals for completeness?

…GH-12086) This adds a `feature_version` flag to `ast.parse()` (documented) and `compile()` (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if `feature_version` is 5 or 6, the hacks for the `async` and `await` keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than `NAME` tokens that the parser driver recognizes.) https://bugs.python.org/issue35975

gvanrossum added the DO-NOT-MERGE label Feb 28, 2019

the-knights-who-say-ni added the CLA signed label Feb 28, 2019

bedevere-bot added the awaiting merge label Feb 28, 2019

ilevkivskyi approved these changes Mar 3, 2019

View changes

gvanrossum added 10 commits Mar 5, 2019

Add blurb

1851eaf

Check flags isn't NULL before dereferencing (this broke regen-importlib)

00fdd08

'async' and 'await' are no longer plain keywords -- fix test_keyword.py

543ab75

Fix test_parser.py

9f4cb23

The PyST_Object header in parsermodule.c became one int larger because it contains a PyCompilerFlags struct, which grew extra space for the st_feature_version field. Took me long enough!

Manually add async and await to kwlist in keywords.py, since they are…

855e07b

… not regular keywords

Simplify compile() call in ast.parse()

ee95fef

Fix indentation of many ast_error() calls

ae36b7b

gvanrossum force-pushed the feature-version branch from 87c80c1 to ae36b7b Compare Mar 5, 2019

gvanrossum changed the title ~~bpo-35975: [WIP] Support parsing earlier minor versions of Python 3~~ bpo-35975: Support parsing earlier minor versions of Python 3 Mar 5, 2019

gvanrossum removed the DO-NOT-MERGE label Mar 5, 2019

Add tests for feature_version

bae211c

ilevkivskyi approved these changes Mar 7, 2019

View changes

gvanrossum added 3 commits Mar 7, 2019

Clarify range of feature_version in docs.

dbb08cc

Rename parses() to parse_all()

c8c84f4

Add test for underscore in number

e925175

gvanrossum added the 🤖 automerge label Mar 7, 2019

miss-islington merged commit 495da29 into master Mar 7, 2019
5 checks passed

bedevere-bot removed the awaiting merge label Mar 7, 2019

zware deleted the feature-version branch Apr 10, 2019

python / cpython Public

bpo-35975: Support parsing earlier minor versions of Python 3 #12086

bpo-35975: Support parsing earlier minor versions of Python 3 #12086

gvanrossum commented Feb 28, 2019 •

edited by bedevere-bot

gvanrossum commented Feb 28, 2019 •

edited

gvanrossum commented Feb 28, 2019

ilevkivskyi left a comment

ilevkivskyi Mar 3, 2019

gvanrossum Mar 5, 2019

ilevkivskyi Mar 3, 2019

gvanrossum Mar 5, 2019

ilevkivskyi Mar 3, 2019

gvanrossum commented Mar 5, 2019

gvanrossum commented Mar 5, 2019

ilevkivskyi left a comment

ilevkivskyi Mar 7, 2019

ilevkivskyi Mar 7, 2019

ilevkivskyi Mar 7, 2019

python / cpython Public

bpo-35975: Support parsing earlier minor versions of Python 3 #12086

bpo-35975: Support parsing earlier minor versions of Python 3 #12086

Conversation

gvanrossum commented Feb 28, 2019 • edited by bedevere-bot

gvanrossum commented Feb 28, 2019 • edited

gvanrossum commented Feb 28, 2019

ilevkivskyi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gvanrossum commented Mar 5, 2019

gvanrossum commented Mar 5, 2019

ilevkivskyi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gvanrossum commented Feb 28, 2019 •

edited by bedevere-bot

gvanrossum commented Feb 28, 2019 •

edited