Name already in use

We can sometimes simplify a mypyc RType union, even if the mypy union
couldn't be simplified. A typical example is `list[x] | list[y]` which
can be simplified to just `list`. Previously this would generate a
redundant union `union[list, list]`.

#14360)

This helps mypyc.

(Various small optimizations, including this, together netted a 6%
performance improvement in self check.)

Address dependabot alert about security vulnerability
(https://nvd.nist.gov/vuln/detail/CVE-2022-40897).

…4356)

Use lazy initialization to avoid method calls in `__init__`. This allows
mypyc to infer more always defined attributes.

(Various small optimizations, including this, together netted a 6%
performance improvement in self check.)

This makes mypy a bit faster and the implementation seems a little
cleaner as well.

(Various small optimizations, including this, together netted a 6%
performance improvement in self check.)

These errors are already reported by the (new) semantic analyzer. 

I've discovered this while updating unit tests for new semanal in
#14166.

Tests are included.

This one would occur because we set the errors module with global
options, instead of per-module override ones.
It only mattered for checks that happened after the partially undefined
checks, which (I believe) is only the unused `type: ignore` checks.

This was discovered when updating tests for #14166.

I've also cleaned up the function signature a little.

Enums are exploded into Union of Literal when narrowed.

Conditional branches on enum values can result in multiple distinct
narrowing of the same enum which are later subject to subtype checks 
(most notably via `is_same_type`, when exiting frame context in the binder). 
Such checks would have quadratic complexity: `O(N*M)` where `N` and `M` 
are the number of entries in each narrowed enum variable, and led to drastic 
slowdown if any of the enums involved has a large number of values.

Implement a linear-time fast path where literals are quickly filtered, with
a fallback to the slow path for more complex values.

In our codebase there is one method with a chain of a dozen `if`
statements operating on instances of an enum with a hundreds of values. 
Prior to the regression it was typechecked in less than 1s. After the regression 
it takes over 13min to typecheck. This patch fully fixes the regression for us.

Fixes #13821.

…#14354)

I want to get rid of all the bogus types eventually.

…14352)

Resolves the following warning:

![image](https://user-images.githubusercontent.com/1350584/209748053-dc33f93c-5c34-4cd2-863f-6e54207d9b4f.png)

By using https://github.com/kanga333/comment-hider/releases/tag/v0.4.0

This helps with await-related errors introduced in #12958, in
particular, which are expensive to generate. If errors are ignored (e.g.
in third-party libraries) or we don't care about the error message, use
simpler error message logic. We also often filter out error messages
temporarily, so any effort in constructing a nice error message is
wasted.

We could skip even more logic, but this should cover many of the
important code paths.

This speeds up self check by about 2%.

This makes the implementation of hasattr() checks faster (introduced in
#13544).

In particular, since the `extra_attrs` attribute used for hasattr()
checks is usually None, I micro-optimized the codepaths to avoid
expensive operations whenever there are no hasattr() checks.

Also avoid expensive operations on simple unions and order `isinstance`
checks so that common types are checked first.

I measured a 2% performance uplift in self-check.

… names are imported (#14270)

Stubtest currently has both false-positives and false-negatives when it
comes to verifying constants in the global namespace of a module.

This PR fixes the false positive by using `inspect.getsourcelines()` to
dynamically retrieve the module source code. It then uses `symtable` to
analyse that source code to gather a list of names which are known to be
imported.

The PR fixes the false negative by only using the `__module__` heuristic
on objects which are callable. The vast majority of callable objects
will be types or functions. For these objects, the `__module__`
attribute will give a good indication of whether the object originates
from another module or not; for other objects, it's less useful.

The docs build is currently failing on `master`: see e.g.
https://github.com/python/mypy/actions/runs/3748461486/jobs/6365827914.

It looks like it was broken by the release of `attrs` 22.2.0 earlier
today. Specifically, it looks like
python-attrs/attrs@1bb2864
broke mypy's docs build.

I think this fixes things.

…14332)

The script compiles some mypy commits in parallel and then measures how
long each version takes to self-check a specific mypy commit. It
measures the performance 15 times for each commit and takes the average.

Based on some experiments, the noise floor on my Linux desktop is about
0.5% to 1.0%. Any difference above 1.0% is likely significant, I
believe. For differences between 0.5% and 1.0% it makes sense to repeat
the measurement a few times.

The interesting part of the output looks something like this:
```
...
=== Results ===

145d8a4 8.207s (0.0%)
145d8a4~1 8.105s (-1.2%)
```

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

Mypyc can't optimize context managers yet, so it's best to avoid them in
hot code paths.

This sacrifices some code quality for a considerable perf gain. This
improved self-check performance by 4%.

This helps mypyc, since accessing mutable attributes of singleton
instances is faster than accessing class variables. The implementation
is also arguably a bit cleaner.

This seems performance-neutral or a very minor optimization, but if we
continue to add attributes to TypeState, this can help.

I measured a 1% performance improvement in self check.

Switch :py:func:type to py:class:type in type_narrowing.rst. The former does not render properly in the docs

This is a tiny follow up from
#14246 (comment)

#14152)

This reverts commit dbcbb3f.

The indentation in generated code is now inconsistent if using 4-space
tabs. We should either use tabs or spaces consistently everywhere, since
we can't expect everybody to have the same tab width.

The broken indentation can be seen by compiling a hello world program
and opening it in an editor configured to use 4-space tabs.

Since there is a lot of code in mypyc that assumes a 4-space
indentation, fixing it all is probably only feasible by normalizing the
indentation during the emit phase. However, the normalization step might
actually slow down compilation slightly, whereas the intent of the
original change to improve efficiency, so this change may ultimately be
impractical. In the future we should make it possible to normalize tabs
without any significant cost, but I'm not sure if that's possible right
now.

`in` could narrow unions of TypeDicts, e.g.
```python
class A(TypedDict)
   foo: int

@Final
class B(TypedDict):
   bar: int

union: Union[A, B] = ...
value: int
if 'foo' in union:
  # Cannot be a B as it is final and has no "foo" field, so must be an A
  value = union['foo']
else:
  # Cannot be an A as those went to the if branch
  value = union['bar']
```

Use a faster type query visitor and reuse visitor across calls.

This should speed up type checking slightly. My measurements show a
~0.5% improvement, but it may be below the noise floor.

Mypyc isn't good at compiling nested functions, and this one was in one
of the hottest code paths in all of mypy. The nested function wasn't
even used that often, but mypyc would still construct a closure object
every time.

This adds some code duplication, but it's well worth it. Amazingly, this
speeds up self-check by about 10%, if my measurements are to be trusted!

This addresses some of the slowdown introduced in #13303. #14324
addresses another related slowdown.

Avoid the use of a nested function, which are a bit slow when compiled
with mypyc. Also avoid a callable value and instead call a function
directly, which allows using faster native calls.

Based on a quick experiment, this speeds up self check by about 3%.

This addresses some of the slowdown introduced in #13303.

Only perform type variable freshening if it's needed, i.e. there is a
nested generic callable, since it's fairly expensive. Make the check for
generic callables fast by creating a specialized type query visitor base
class for queries with bool results. The visitor tries hard to avoid
memory allocation in typical cases, since allocation is slow.

This addresses at least some of the performance regression in #14095.
This improved self-check performance by about 3% when compiled with
mypyc (-O2).

The new visitor class can potentially help with other type queries as
well. I'll explore it in follow-up PRs.

The implementation copied lots of callable types even when not using
the new feature, which was expensive. Now we only generate a copy if a
callable actually uses TypedDict types for **kwds.

This made self check 7-8% faster (when compiled with -O0).

The original implementation was in
#13471.

Removals from `stubinfo.py`:
- `atomicwrites` is archived and deprecated at runtime; stubs were
removed from typeshed in python/typeshed#8925
- `attrs` has had inline types for a very long time now
- `chardet` recently cut a release with inline types; typeshed's stubs
were marked obsolete in python/typeshed#9318
- `cryptography` has had inline types for a very long time now; the only
reason why it's still in typeshed is because other typeshed packages
need `types-cryptography` as a dependency, and our testing
infrastructure therefore can't currently cope with it being removed from
typeshed.
- `emoji` recently cut a release bundling stubs with the runtime
package; typeshed's stubs were marked obsolete in
python/typeshed#9051
- `termcolor` recently cut a release with inline types; typeshed's stubs
were marked obsolete in python/typeshed#8746
- `prettytable` recently cut a release with inline types; typeshed's
stubs were marked obsolete in
python/typeshed#9023

Additions:
- Stubs for `Xlib` were added in
python/typeshed#9279
- Stubs for `consolemenu` were added in
python/typeshed#8820
- Stubs for `dockerfile_parse` were added in
python/typeshed#9305
- Stubs for `flask_migrate` were added in
python/typeshed#8967
- Stubs for `paho.mqtt` were added in
python/typeshed#8853
- Stubs for `pycocotools` were added in
python/typeshed#9086
- Stubs for many `pywin32` modules were added in
python/typeshed#8825, and multiple follow-up PRs
- Stubs for `pyscreeze` were added in
python/typeshed#8823

Enable the `--debug-serialize` option to help catch issues during
serialization which would normally be skipped by mypy_primer.

Followup to #14155

Currently, `mypy_primer` sets `--cache-dir=/dev/null` which disables
cache generation. This can result in errors being missed which would
normally come up during `tree.serialize()`. Removing
`--cache-dir=/dev/null` isn't practical.

This PR adds a new debug / test option `--debug-serialize` which runs
`tree.serialize()` even if cache generation is disabled to help detect
serialize errors earlier.

**Refs**
* #14137
*
hauntsaninja/mypy_primer#54 (review)

cc: @hauntsaninja

This PR only adds the existing `TypeVar` support for the
`typing_extensions` variant.
I.e. it does not include support for `default` or `infer_variance`.

Fixes #14312

In particular, ran into an issue with an `if TYPE_CHECKING` case, so I
added a test for that.

A similar case was addressed in #14176 but I missed the part where it
doesn't need to be defined in a different branch. This makes the fix
more complete.

This adds support for try/except/finally/else check.

The implementation ended up pretty complicated because it had to handle
jumps different for finally. It took me a few iterations to get to this
solution and that's the cleanest one I could come up with.

Closes #13928.

Now mypy can figure out the values of final variables even if the
initializer has some operations on constant values:
```
A: Final = 2  # This has always worked
A: Final = -(1 << 2)  # This is now supported
B: Final = 'x' + 'y'  # This also now works
```
Currently we support integer arithmetic and bitwise operations, and
string concatenation.

This can be useful with literal types, but my main goal was to improve
constant folding in mypyc. In particular, this helps constant folding
with native ints in cases like these:
```
FLAG1: Final = 1 << 4
FLAG2: Final = 1 << 5

def f() -> i64:
    return FLAG1 | FLAG2  # Can now be constant folded
```

We still have another constant folding pass in mypyc, since it does some
things more aggressively (e.g. it constant folds some member expression
references).

Work on mypyc/mypyc#772.

Also helps with mypyc/mypyc#862.