python / peps

PEP: 562 Title: Module __getattr__ and __dir__ Author: Ivan Levkivskyi <levkivskyi@gmail.com> Status: Final Type: Standards Track Content-Type: text/x-rst Created: 09-Sep-2017 Python-Version: 3.7 Post-History: 09-Sep-2017 Resolution: https://mail.python.org/pipermail/python-dev/2017-December/151033.html

Abstract

It is proposed to support __getattr__ and __dir__ function defined on modules to provide basic customization of module attribute access.

Rationale

It is sometimes convenient to customize or otherwise have control over access to module attributes. A typical example is managing deprecation warnings. Typical workarounds are assigning __class__ of a module object to a custom subclass of types.ModuleType or replacing the sys.modules item with a custom wrapper instance. It would be convenient to simplify this procedure by recognizing __getattr__ defined directly in a module that would act like a normal __getattr__ method, except that it will be defined on module instances. For example:

# lib.py

from warnings import warn

deprecated_names = ["old_function", ...]

def _deprecated_old_function(arg, other):
    ...

def __getattr__(name):
    if name in deprecated_names:
        warn(f"{name} is deprecated", DeprecationWarning)
        return globals()[f"_deprecated_{name}"]
    raise AttributeError(f"module {__name__} has no attribute {name}")

# main.py

from lib import old_function  # Works, but emits the warning

Another widespread use case for __getattr__ would be lazy submodule imports. Consider a simple example:

# lib/__init__.py

import importlib

__all__ = ['submod', ...]

def __getattr__(name):
    if name in __all__:
        return importlib.import_module("." + name, __name__)
    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

# lib/submod.py

print("Submodule loaded")
class HeavyClass:
    ...

# main.py

import lib
lib.submod.HeavyClass  # prints "Submodule loaded"

There is a related proposal PEP 549 that proposes to support instance properties for a similar functionality. The difference is this PEP proposes a faster and simpler mechanism, but provides more basic customization. An additional motivation for this proposal is that PEP 484 already defines the use of module __getattr__ for this purpose in Python stub files, see [1].

In addition, to allow modifying result of a dir() call on a module to show deprecated and other dynamically generated attributes, it is proposed to support module level __dir__ function. For example:

# lib.py

deprecated_names = ["old_function", ...]
__all__ = ["new_function_one", "new_function_two", ...]

def new_function_one(arg, other):
   ...
def new_function_two(arg, other):
    ...

def __dir__():
    return sorted(__all__ + deprecated_names)

# main.py

import lib

dir(lib)  # prints ["new_function_one", "new_function_two", "old_function", ...]

Specification

The __getattr__ function at the module level should accept one argument which is the name of an attribute and return the computed value or raise an AttributeError:

def __getattr__(name: str) -> Any: ...

If an attribute is not found on a module object through the normal lookup (i.e. object.__getattribute__), then __getattr__ is searched in the module __dict__ before raising an AttributeError. If found, it is called with the attribute name and the result is returned. Looking up a name as a module global will bypass module __getattr__. This is intentional, otherwise calling __getattr__ for builtins will significantly harm performance.

The __dir__ function should accept no arguments, and return a list of strings that represents the names accessible on module:

def __dir__() -> List[str]: ...

If present, this function overrides the standard dir() search on a module.

The reference implementation for this PEP can be found in [2].

Backwards compatibility and impact on performance

This PEP may break code that uses module level (global) names __getattr__ and __dir__. (But the language reference explicitly reserves all undocumented dunder names, and allows "breakage without warning"; see [3].) The performance implications of this PEP are minimal, since __getattr__ is called only for missing attributes.

Some tools that perform module attributes discovery might not expect __getattr__. This problem is not new however, since it is already possible to replace a module with a module subclass with overridden __getattr__ and __dir__, but with this PEP such problems can occur more often.

Discussion

Note that the use of module __getattr__ requires care to keep the referred objects pickleable. For example, the __name__ attribute of a function should correspond to the name with which it is accessible via __getattr__:

def keep_pickleable(func):
    func.__name__ = func.__name__.replace('_deprecated_', '')
    func.__qualname__ = func.__qualname__.replace('_deprecated_', '')
    return func

@keep_pickleable
def _deprecated_old_function(arg, other):
    ...

One should be also careful to avoid recursion as one would do with a class level __getattr__.

To use a module global with triggering __getattr__ (for example if one wants to use a lazy loaded submodule) one can access it as:

sys.modules[__name__].some_global

or as:

from . import some_global

Note that the latter sets the module attribute, thus __getattr__ will be called only once.

References

Copyright

This document has been placed in the public domain.