What do you think of removing the lines like testing protobuf (37 files)..? If there's no error in that library, it's just noise that makes it harder to find the error.

We could adapt the format I used for stubtest: https://github.com/python/typeshed/runs/6480640648?check_suite_focus=true (open the "Run stubtest" section). I think that's well readable and confirms that the tests (and which tests) were run.

What do you think of removing the lines like testing protobuf (37 files)..? If there's no error in that library, it's just noise that makes it harder to find the error.

Hmm, I quite like those when running the test locally. Testing the stubs directory can take a while; I kinda like the reassurance that something is actually happening :)

We could adapt the format I used for stubtest: https://github.com/python/typeshed/runs/6480640648?check_suite_focus=true (open the "Run stubtest" section). I think that's well readable and confirms that the tests (and which tests) were run.

Actually yes, this is very pretty. And the coloured output is very helpful in identifying errors as opposed to other kinds of output!

What do you think of removing the lines like testing protobuf (37 files)..? If there's no error in that library, it's just noise that makes it harder to find the error.

Relatedly, does anybody have any idea why I get a lot more output whenever I run the test locally than we seem to get in CI? When I run the test locally I get repeated spam about there being "no [mypy] section in config file":

A couple things that might be causing it:

• In mypy/config_parser.py, mypy calls config_parser.read(config_file). This method silences errors, and returns a list of successfully read files. Try printing what it returns (after installing mypy from source with e.g. pip install -e .\mypy).
• I wonder if temp.flush() in tests/mypy_test.py is enough to get the content showing there correctly. You could try:

  --- a/tests/mypy_test.py
  +++ b/tests/mypy_test.py
  @@ -174,6 +174,8 @@ def run_mypy(
                   temp.write("{} = {}\n".format(k, v))
           temp.flush()
   
  +        print(Path(temp.name).read_text())
  +
           flags = get_mypy_flags(args, major, minor, temp.name, custom_typeshed=custom_typeshed)
           mypy_args = [*flags, *files]
           if args.verbose: