Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance of attribute lookup for type objects #92216

Open
eendebakpt opened this issue May 3, 2022 · 4 comments
Open

Performance of attribute lookup for type objects #92216

eendebakpt opened this issue May 3, 2022 · 4 comments
Labels
3.12 performance Performance or resource usage type-bug An unexpected behavior, bug, or error

Comments

@eendebakpt
Copy link
Contributor

eendebakpt commented May 3, 2022 •
edited by bedevere-bot

Bug report

The performance of attribute lookup for type objects is worse than for other objects. A benchmark

import pyperf
runner=pyperf.Runner()

setup="""
class Class:
    def all(self):
        pass

x=Class()
"""

runner.timeit('hasattr x.all', "hasattr(x, 'all')", setup=setup)
runner.timeit('hasattr x.__array_ufunc__', "hasattr(x, '__array_ufunc__')", setup=setup)
runner.timeit('hasattr Class.all', "hasattr(Class, 'all')", setup=setup)
runner.timeit('hasattr Class.__array_ufunc__', "hasattr(Class, '__array_ufunc__')", setup=setup) # worse performance

Results:

hasattr x.all: Mean +- std dev: 68.1 ns +- 1.1 ns
hasattr x.__array_ufunc__: Mean +- std dev: 40.4 ns +- 0.3 ns
hasattr Class.all: Mean +- std dev: 38.1 ns +- 0.6 ns
hasattr Class.__array_ufunc__: Mean +- std dev: 255 ns +- 2 ns

The reason seems to be that the type_getattro always executes PyErr_Format, wheras for the "normal" attribute lookup this is avoided (see here and here)

Notes:

Your environment

  • CPython versions tested on: Python 3.11.0a7+
  • Operating system and architecture: Linux Ubuntu

Linked PRs
@eendebakpt eendebakpt added the type-bug An unexpected behavior, bug, or error label May 3, 2022
@hugovk hugovk added the performance Performance or resource usage label May 3, 2022
@seberg
Copy link
Contributor

seberg commented May 3, 2022 •
edited

Related to this issue, I am wondering if we can consider _PyObject_LookupAttr either as stable (removing the _) or as "semi stable"?
Of course that is only faster currently for the default tp_getattro, but I suspect that is a huge amount of calls.

EDIT: Ah, I half thought it was not fully internal API right now, but "semi" exposed.

@eendebakpt
Copy link
Contributor Author

eendebakpt commented May 3, 2022

Also see: #76752 (PRs where _PyObject_LookupAttr is introduced)

@serhiy-storchaka
Copy link
Member

serhiy-storchaka commented May 4, 2022

_PyObject_LookupAttr is an internal API.

I'll look if it is possible to speed up lookup of the absent attribute in a class object.

This was referenced May 5, 2022
Closed
Merged
seberg pushed a commit to numpy/numpy that referenced this issue May 11, 2022
@eendebakpt
PERF: Skip probing __array_ufunc__ for NumPy builtin scalars (#21470)
7987557 
The method `PyUFuncOverride_GetNonDefaultArrayUfunc` is expensive on numpy scalars because these objects do not have a `__array_ufunc__` set and for a missing attribute lookup cpython generates an exception that is later cleared by numpy. This is a performance bottleneck, see #21455.

An issue has been submitted to cpython (python/cpython#92216). But even if this is addressed in cpython, it will take untill python 3.12+ before this will be useable by numpy.

As an alternative solution, this PR adds a fast path to `PyUFuncOverride_GetNonDefaultArrayUfunc` to determine whether an object is a numpy scalar.
This was referenced Dec 3, 2022
Closed
Merged
Fidget-Spinner pushed a commit that referenced this issue Dec 23, 2022
@eendebakpt
gh-92216: improve performance of hasattr for type objects (GH-99979)
7fc7909
@Fidget-Spinner
Copy link
Member

Fidget-Spinner commented Dec 23, 2022

@eendebakpt thanks for the improvement. Can I close this issue now or do you plan to submit more PRs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.12 performance Performance or resource usage type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@seberg @eendebakpt @iritkatriel @hugovk @serhiy-storchaka @Fidget-Spinner