creating new threads appears to leak memory rapidly in Python 3.11 / linux #99205

zzzeek · 2022-11-07T15:11:48Z

The following test program gains about 10M per second under top, on Python 3.11 only (confirmed for all development stages: 3.11.0a4, 3.11.0b1, 3.11.0rc1, 3.11.0) . 3.10 shows no memory growth.

from threading import Thread

while True:
    t1 = Thread()
    t1.start()

For direct results, here's the same program using the resource module that I just saw used over at #98467:

from threading import Thread
import resource

print("Before", resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024, "MB")

i = 0

while True:
    i += 1
    t1 = Thread()
    t1.start()
    if i % 10000 == 0:
        print(
            f"memory after {i} iterations:",
            resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024,
            "MB",
        )

The above program under Py 3.11.0 prints:

$ .venv-3.11/bin/python test5.py 
Before 8.14453125 MB
memory after 10000 iterations: 11.6875 MB
memory after 20000 iterations: 15.30078125 MB
memory after 30000 iterations: 18.65234375 MB
memory after 40000 iterations: 22.26171875 MB
memory after 50000 iterations: 25.87109375 MB
memory after 60000 iterations: 29.22265625 MB
memory after 70000 iterations: 32.83203125 MB
memory after 80000 iterations: 36.44140625 MB
memory after 90000 iterations: 39.78125 MB
memory after 100000 iterations: 43.390625 MB
...

under Python 3.10 it prints:

$ .venv-3.10/bin/python test5.py 
Before 9.25 MB
memory after 10000 iterations: 9.46484375 MB
memory after 20000 iterations: 9.47265625 MB
memory after 30000 iterations: 9.47265625 MB
memory after 40000 iterations: 9.47265625 MB
memory after 50000 iterations: 9.47265625 MB
memory after 60000 iterations: 9.47265625 MB
memory after 70000 iterations: 9.47265625 MB
memory after 80000 iterations: 9.47265625 MB

the issue looks extremely similar to another one we just fixed in greenlet, over at python-greenlet/greenlet#328, although this one is much more surprising. Issue #98467 might also be related.

Environment:

$ uname -a
Linux photon3 5.17.14-200.fc35.x86_64 #1 SMP PREEMPT Thu Jun 9 14:02:42 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

$ python3
Python 3.11.0 (main, Nov  5 2022, 23:27:22) [GCC 11.3.1 20220421 (Red Hat 11.3.1-3)] on linux

The text was updated successfully, but these errors were encountered:

kumaraditya303 · 2022-11-07T17:28:09Z

Seems related to the 3.11 new frame handling.

cc @markshannon @brandtbucher

zzzeek added the type-bug An unexpected behavior, bug, or error label Nov 7, 2022

zzzeek mentioned this issue Nov 7, 2022

Memory leak with asyncpg and Python 3.11 sqlalchemy/sqlalchemy#8763

Closed

kumaraditya303 added 3.11 3.12 interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) labels Nov 7, 2022

creating new threads appears to leak memory rapidly in Python 3.11 / linux #99205

creating new threads appears to leak memory rapidly in Python 3.11 / linux #99205

zzzeek commented Nov 7, 2022 •

edited

kumaraditya303 commented Nov 7, 2022

creating new threads appears to leak memory rapidly in Python 3.11 / linux #99205

creating new threads appears to leak memory rapidly in Python 3.11 / linux #99205

Comments

zzzeek commented Nov 7, 2022 • edited

kumaraditya303 commented Nov 7, 2022

zzzeek commented Nov 7, 2022 •

edited