ABC causing massive memory swell #94284

zrothberg · 2022-06-26T06:04:21Z

Bug report

from abc import ABC, ABCMeta
from datetime import datetime


abcclasses = set()

normalclasses = set()

for i in range(10000):
    abcclasses.add(ABCMeta("abc_"+str(i), (ABC, ), {}))
    normalclasses.add(type("normal_"+str(i), (object,), {}))


if __name__ == '__main__':

    starttime = datetime.now()
    import os, psutil
    process = psutil.Process(os.getpid())
    mb = 1024 * 1024
    mem = last = process.memory_info().rss
    print(f'{i + 1:>4d} {mem / mb:8.2f}MB {(mem - last) / mb:+8.2f}MB | {"━" * int(mem / 8_000_000)}')
    for item in normalclasses:
        issubclass(item, ABC)

    mem = process.memory_info().rss
    print(f'{i + 1:>4d} {mem / mb:8.2f}MB {(mem - last) / mb:+8.2f}MB | {"━" * int(mem / 8_000_000)}')
    print(f"This took {datetime.now()-starttime}")

Your environment

Running this takes several minutes and consumes over 20 gbs of memory.

I've checked it against python 3.8-3.10. The issue is just that ABC caches every single issubclass evaluation combined with it searching its entire subclass tree recursively causes it to grind to a halt if you have any significant number of subclass 10k sub classes of ABC and 10k normal classes being compared to the base ABC climbs to over 20 GBs. I am not really sure it is necessary to check the entire subclass tree. I presume it is because of concerns about register calls happening low in the class tree.

It would likely be far more efficient to chase upward the calls to register and place those in the parent classes then to search the entire class hierarchy. This can get really aggressive performance issues if you have any multiple inheritance structures going on as well as those will get checked multiple times.

See these issues.
#92810
pydantic/pydantic#3829

zrothberg · 2022-09-21T15:43:36Z

@iritkatriel Does that mean you are working on something for this for 3.12? Or was that just an automated thing?

I have a few bits and pieces of this solved out but nothing is read and working fully yet.

I think it's doable to make large gains here by modifying register function to chase up the object hierarchy without having to crawl doward during isinstance. That would also resolve some really subtle logical bugs the current setup causes with object lifecycles redefining isinstance return values and the cached results outliving the current values.

I have small and terribly written proof of concept function for abstract methods via a decorator and custom init_subclass hook. That would alleviate a large number of cases where people need to create mixins with ABCs when they really just need the abstract method behavior without all the registration and isinstance stuff.

pydantic/pydantic#3829 (comment)

iritkatriel · 2022-09-21T15:46:32Z

No it means that the earliest version it can go into is 3.12. I'm not working on it.

iritkatriel · 2023-04-03T17:20:44Z

ABC causing massive memory swell #94284

ABC causing massive memory swell #94284

zrothberg commented Jun 26, 2022

zrothberg commented Sep 21, 2022

iritkatriel commented Sep 21, 2022

iritkatriel commented Apr 3, 2023

ABC causing massive memory swell #94284

ABC causing massive memory swell #94284

Comments

zrothberg commented Jun 26, 2022

zrothberg commented Sep 21, 2022

iritkatriel commented Sep 21, 2022

iritkatriel commented Apr 3, 2023