The implementation is running compile() with flags=PyCF_ALLOW_INCOMPLETE_INPUT for each potential matched word preceded by the console code buffer (in the above example, source would be match x:\n case ), and iif it raises no exceptions or it raises _IncompleteInputError, the match is considered to be syntactically valid and kept.

It seems that the new REPL can run several statements together, e.g.

>>> if True:
...     "foo"
... "bar"
... 
'bar'
>>> 

Contrary to the old REPL,

>>> if True:
...     "foo"
... "bar"
  File "<stdin>", line 3
    "bar"
    ^^^^^
SyntaxError: invalid syntax
>>> 

I see the implementation of the new REPL is (hide irrelevant details):

class InteractiveColoredConsole(code.InteractiveConsole):
    ...
    def runsource(self, source, filename="<input>", symbol="single"):
        ...
        for stmt in tree.body:
            ...
            the_symbol = symbol if stmt is last_stmt else "exec"
            ...
                code = self.compile.compiler(item, filename, the_symbol, dont_inherit=True)
            ...

It runs the first several statements with "exec" except the last one with "single".

This means that I have to use mode="exec" for InteractiveColoredConsole and mode="single" for regular ones. mode="single" with PyCF_ALLOW_INCOMPLETE_INPUT works as expected, but mode="exec" with PyCF_ALLOW_INCOMPLETE_INPUT` can give unexpected results. For example,

>>> import codeop
>>> compile("if True", "<test>", "exec", codeop.PyCF_ALLOW_INCOMPLETE_INPUT)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<test>", line 1
    if True
           ^
SyntaxError: expected ':'
>>> 

This is because as in helpers.c _PyTokenizer_translate_newlines(),

    /* If this is exec input, add a newline to the end of the string if
       there isn't one already. */
    if (exec_input && c != '\n' && c != '\0') {
        *current = '\n';
        current++;
    }

which makes "if True" become "if True\n" which is not incomplete.

In my PR, I changed it to not adding a newline for exec input only when flag includes PyCF_ALLOW_INCOMPLETE_INPUT and excludes PyCF_DONT_IMPLY_DEDENT. I am not sure if this change is okay. codeop is using PyCF_ALLOW_INCOMPLETE_INPUT | PyCF_DONT_IMPLY_DEDENT and I have make this case unchanged. The backward-incompatible case would be using PyCF_ALLOW_INCOMPLETE_INPUT without PyCF_DONT_IMPLY_DEDENT, which I didn't find any in the standard library. Or should I add a new flag to make it completely backward-compatible, or should it be considered as a bug so that it is okay t fix it?

I am sorry for the long paragraphs above, and I hope I described the problem clearly.

• Issue: Make REPL completions match only syntactically valid keywords and values #122379