Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

_abc._abc_subclasscheck has very poor performance and (I think) a memory leak #92810

Open
samuelcolvin opened this issue May 14, 2022 · 0 comments
Open
Labels
performance type-bug

Comments

@samuelcolvin
Copy link
Sponsor Contributor

@samuelcolvin samuelcolvin commented May 14, 2022

Bug report

I'm been hunting a memory leak in pydantic and I keep seeing _abc_subclasscheck popping up in the memray flamegraph.

Locally I've seen memray report that _abc_subclasscheck is using 2.8GB of memory in some cases!

I can't get anything that bad in a minimal example, but I have the following:

from abc import ABCMeta
from datetime import datetime


class MyMetaclass(ABCMeta):
    pass


class MyClass(metaclass=MyMetaclass):
    pass


def main():
    class Foobar(MyClass):
        pass

    assert issubclass(Foobar, MyClass)
    assert not issubclass(int, MyClass)
    assert not issubclass(str, MyClass)
    assert not issubclass(datetime, MyClass)
    t = type('A', (), {})
    assert not issubclass(t, MyClass)


if __name__ == '__main__':
    import os, psutil
    process = psutil.Process(os.getpid())
    mb = 1024 * 1024
    last = 0
    for i in range(5_000):
        main()
        # mem = process.memory_info().rss
        # print(f'{i + 1:>4d} {mem / mb:8.2f}MB {(mem - last) / mb:+8.2f}MB | {"━" * int(mem / 8_000_000)}')
        # last = mem

A few things to note:

  • the commented out last few lines print current memory and change in memory over time
  • I'm not sure exactly which issubclass calls are necessary, certainly I don't see a rise in memory with just the last one
  • some memory drops (presumably related to clearing the abc caches?) is happening, but overall memory is increasing
  • The performance of issubclass on an abc is very poor indeed - this script takes 14seconds, if i switch ABCMeta to type it takes 121ms!

Your environment

  • Python 3.10.0 installed with venv
  • Ubuntu 21.10
@samuelcolvin samuelcolvin added the type-bug label May 14, 2022
@AlexWaygood AlexWaygood added the performance label May 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance type-bug
Projects
None yet
Development

No branches or pull requests

2 participants