bpo-33587: inspect.getsource: reorder stat on file in linecache #6805

pankajp · 2018-05-14T15:27:24Z

The check for os.path.exists() on source file is postponed in
inspect.getsourcefile() until needed avoiding an expensive filesystem stat call.

https://bugs.python.org/issue33587

pankajp · 2018-05-14T15:34:05Z

This looks like a simple change to me. If I were to create a bug on bpo, what should it mention? Would something like "inspect.getsource() does unnecessary filesystem stat when file is already in linecache" suffice?

matrixise · 2018-05-14T15:42:55Z

Hi @pankajp

Thank you for your Pull Requests

but we need an issue from bpo and an explanation about your change.

Thank you,

serhiy-storchaka · 2018-05-14T22:20:58Z

getmodule() is a complex function, it can perform multiple calls of os.path.realpath() which can perform multiple expensive filesystem stat calls.

Please open an issue on the bug tracker for discussion and provide your benchmark results.

pankajp · 2018-05-20T17:01:16Z

Thanks for your advise @matrixise and @serhiy-storchaka . I have added a bug https://bugs.python.org/issue33587 and posted a cProfile performance comparison there. Which is the canonical place to discuss the patch (performance numbers and other things), over here at github or at bpo?

Here's the patch performance difference before and after the patch:

test_code.py:

import inspect


class A():
    def __init__(self):
        self.stack = inspect.stack()
        self.children = []

    def add_child(self, cls):
        child = cls()
        self.children.append(child)
        return child

class B(A):
    pass

class C(A):
    pass

node_classes = [A, B, C]

def load_nodes(N):
    root = A()
    for i in range(N):
        child_i = root.add_child(node_classes[i%len(node_classes)])
        for j in range(N):
            child_j = child_i.add_child(node_classes[(i*j)%len(node_classes)])

load_nodes(20)

Before:

Sun May 20 21:42:32 2018    prof1.stat

         1188991 function calls (1188851 primitive calls) in 4.821 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     18/1    0.006    0.000    4.824    4.824 {built-in method builtins.exec}
        1    0.000    0.000    4.824    4.824 inspect_stack_perf.py:1(<module>)
        1    0.001    0.001    4.790    4.790 inspect_stack_perf.py:22(load_nodes)
      421    0.001    0.000    4.787    0.011 inspect_stack_perf.py:5(__init__)
      421    0.001    0.000    4.786    0.011 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:1492(stack)
      421    0.011    0.000    4.785    0.011 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:1464(getouterframes)
     4630    0.031    0.000    4.770    0.001 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:1425(getframeinfo)
      420    0.001    0.000    4.739    0.011 inspect_stack_perf.py:9(add_child)
    13994    4.159    0.000    4.159    0.000 {built-in method nt.stat}
     4630    0.042    0.000    3.223    0.001 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:760(findsource)
     9322    0.043    0.000    2.960    0.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:680(getsourcefile)
     9322    0.011    0.000    2.832    0.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\genericpath.py:16(exists)
     4630    0.016    0.000    1.339    0.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\linecache.py:53(checkcache)
     4630    0.097    0.000    0.364    0.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:714(getmodule)
     4692    0.008    0.000    0.135    0.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:702(getabsfile)
     4754    0.008    0.000    0.091    0.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\ntpath.py:538(abspath)
     4754    0.036    0.000    0.074    0.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\ntpath.py:471(normpath)
   162600    0.071    0.000    0.071    0.000 {built-in method builtins.hasattr}
...

After the patch:

Sun May 20 21:39:44 2018    prof2.stat

         2639991 function calls (2639727 primitive calls) in 2.841 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     18/1    0.005    0.000    2.844    2.844 {built-in method builtins.exec}
        1    0.000    0.000    2.844    2.844 inspect_stack_perf.py:1(<module>)
        1    0.001    0.001    2.814    2.814 inspect_stack_perf.py:22(load_nodes)
      421    0.001    0.000    2.812    0.007 inspect_stack_perf.py:5(__init__)
      421    0.001    0.000    2.811    0.007 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:1492(stack)
      421    0.010    0.000    2.810    0.007 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:1464(getouterframes)
      420    0.001    0.000    2.802    0.007 inspect_stack_perf.py:9(add_child)
     4630    0.029    0.000    2.795    0.001 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:1425(getframeinfo)
     4630    0.040    0.000    2.380    0.001 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:760(findsource)
     4674    1.631    0.000    1.631    0.000 {built-in method nt.stat}
     4630    0.014    0.000    1.630    0.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\linecache.py:53(checkcache)
13952/13890    0.281    0.000    0.907    0.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:714(getmodule)
9322/9260    0.038    0.000    0.703    0.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:680(getsourcefile)
    13952    0.018    0.000    0.259    0.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\inspect.py:702(getabsfile)
    14014    0.017    0.000    0.215    0.000 C:\Users\ppandey\AppData\Local\Continuum\Anaconda3\lib\ntpath.py:538(abspath)
   478166    0.177    0.000    0.177    0.000 {built-in method builtins.hasattr}
...

Total runtime reduced from 4.8 s to 2.8 s, and the major difference can be seen in the nt.stat call. This is on a machine with SSD. In workplace where python is installed in cluster on nfs, I'm sure the performance is worse.

taleinat · 2018-06-10T11:56:14Z

@pankajp, the place to discuss proposed changes and bugs is indeed the issue tracker.

The check for os.path.exists() on source file is postponed in inspect.getsourcefile() until needed avoiding an expensive filesystem stat call and PEP 302 module loader check is moved last for performance since it is an uncommon case.

taleinat

This LGTM.

iritkatriel · 2021-05-21T14:44:00Z

LGTM, but the merge conflicts need to be resolved.

Lib/inspect.py

mdboom · 2022-08-05T20:08:46Z

@pankajp: While I understand it's been a long time, it looks still relevant. Are you interested in rebasing this and re-running the benchmarks to get this merged?

the-knights-who-say-ni added the CLA signed label May 14, 2018

bedevere-bot added the awaiting review label May 14, 2018

pankajp changed the title ~~inspect.getsource: reorder stat on file in linecache~~ bpo-33587: inspect.getsource: reorder stat on file in linecache May 20, 2018

inspect.getsource: avoid stat on file in linecache

7d823c1

The check for os.path.exists() on source file is postponed in inspect.getsourcefile() until needed avoiding an expensive filesystem stat call and PEP 302 module loader check is moved last for performance since it is an uncommon case.

pankajp force-pushed the patch-1 branch from 6190c7a to 7d823c1 Compare Jun 12, 2018

tweak code comment

af56911

taleinat approved these changes Sep 19, 2020

View changes

bedevere-bot added awaiting merge and removed awaiting review labels Sep 19, 2020

taleinat added skip news performance Performance or resource usage needs backport to 3.8 needs backport to 3.9 labels Sep 19, 2020

ambv removed the needs backport to 3.8 label May 4, 2021

JelleZijlstra added the needs backport to 3.10 label Jan 24, 2022

methane removed needs backport to 3.9 needs backport to 3.10 labels Jan 31, 2022

Merge branch 'main' into patch-1

8f36e26

methane reviewed Jan 31, 2022

View changes

Lib/inspect.py Outdated Show resolved Hide resolved

ezio-melotti removed the CLA signed label Jul 13, 2022

Merge branch 'main' into patch-1

c4af3c1

iritkatriel merged commit c1581a9 into python:main Aug 26, 2022
12 checks passed

bedevere-bot removed the awaiting merge label Aug 26, 2022

pankajp mannequin mentioned this pull request Aug 26, 2022

inspect.getsource performs unnecessary filesystem stat call #77768

Closed

pankajp deleted the patch-1 branch Sep 1, 2022

bpo-33587: inspect.getsource: reorder stat on file in linecache #6805

bpo-33587: inspect.getsource: reorder stat on file in linecache #6805

pankajp commented May 14, 2018 •

edited by bedevere-bot

pankajp commented May 14, 2018

matrixise commented May 14, 2018

serhiy-storchaka commented May 14, 2018

pankajp commented May 20, 2018 •

edited

taleinat commented Jun 10, 2018

taleinat left a comment

iritkatriel commented May 21, 2021

mdboom commented Aug 5, 2022

bpo-33587: inspect.getsource: reorder stat on file in linecache #6805

bpo-33587: inspect.getsource: reorder stat on file in linecache #6805

Conversation

pankajp commented May 14, 2018 • edited by bedevere-bot

pankajp commented May 14, 2018

matrixise commented May 14, 2018

serhiy-storchaka commented May 14, 2018

pankajp commented May 20, 2018 • edited

taleinat commented Jun 10, 2018

taleinat left a comment

iritkatriel commented May 21, 2021

mdboom commented Aug 5, 2022

pankajp commented May 14, 2018 •

edited by bedevere-bot

pankajp commented May 20, 2018 •

edited