Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH allow for specifying CPU features to enable via NPY_ENABLE_CPU_FEATURES environment variable #22137

Open
wants to merge 28 commits into
base: main
Choose a base branch
from

Conversation

Micky774
Copy link
Contributor

@Micky774 Micky774 commented Aug 15, 2022

Related issues/PRs

Fixes #22058

What does this PR add?

This PR enables the usage of a NPY_ENABLE_CPU_FEATURES environment variable whose function is described in #22058.
Note that the enabling function errors when asked to enable features not supported by the CPU, or that numpy was not built with. This breaks symmetry with the disabling semantics, but I think this makes sense since it would be a drastic expectation violation otherwise.

Other comments

@Micky774 Micky774 changed the title Initial implementation ENH allow for specifying CPU features to enable via NPY_ENABLE_CPU_FEATURES environment variable Aug 15, 2022
@Micky774
Copy link
Contributor Author

Micky774 commented Aug 15, 2022

@mattip @seiko2plus Do you know where I should add the tests for this?

@seiko2plus
Copy link
Member

seiko2plus commented Aug 17, 2022

Do you know where I should add the tests for this?

maybe within [numpy/core/tests/test_cpu_dispatcher.py] (https://github.com/numpy/numpy/blob/main/numpy/core/tests/test_cpu_dispatcher.py) but it will require loading the module in a separate process.

@magsol
Copy link

magsol commented Aug 17, 2022

@seiko2plus Is there a preference in NumPy for warnings vs errors? My question here is partially because of the fact that enabling a feature that doesn't exist is not equivalent to disabling a feature that doesn't exist, and so may require different ways of handling the error--a warning for the latter seems appropriate, but in the case of the former, it may be better to throw an error and exit. Thoughts?

@Micky774
Copy link
Contributor Author

Micky774 commented Aug 18, 2022

@seiko2plus Is there a preference in NumPy for warnings vs errors? My question here is partially because of the fact that enabling a feature that doesn't exist is not equivalent to disabling a feature that doesn't exist, and so may require different ways of handling the error--a warning for the latter seems appropriate, but in the case of the former, it may be better to throw an error and exit. Thoughts?

For reviewers and interested parties, the current PR's behavior regarding errors/warnings:

* It will set a RuntimeError when
* - CPU baseline features from the build are not supported at runtime
* - 'NPY_DISABLE_CPU_FEATURES' tries to disable a baseline feature
* - 'NPY_DISABLE_CPU_FEATURES' and 'NPY_ENABLE_CPU_FEATURES' are
* simultaneously set
* - 'NPY_ENABLE_CPU_FEATURES' tries to enable a feature that is not supported
* by the machine or build
* - 'NPY_ENABLE_CPU_FEATURES' tries to enable a feature when the project was
* not built with any feature optimization support
*
* It will set an ImportWarning when:
* - 'NPY_DISABLE_CPU_FEATURES' tries to disable a feature that is not supported
* by the machine or build
* - 'NPY_DISABLE_CPU_FEATURES' or 'NPY_ENABLE_CPU_FEATURES' tries to
* disable/enable a feature when the project was not built with any feature
* optimization support

@seiko2plus
Copy link
Member

seiko2plus commented Aug 18, 2022

@magsol,

As long as the error does not affect the workflow of the following procedures, then choosing to raise runtime-warning would make sense. according to Python's warning-doc RuntimeWarning is reserved for warnings about dubious runtime features. maybe ImportWarning would be a better choice since it's ignored by default.
however, the end-user has the upper hand here, since this behavior can be changed during the runtime through module warning, for example, warnings can be treated as errors via:

import os, warnings
os.environ['NPY_DISABLE_CPU_FEATURES'] = 'XXXX'
warnings.simplefilter("error", RuntimeWarning)
import numpy as np

@magsol
Copy link

magsol commented Sep 6, 2022

@seiko2plus That's perfect. Thank you!

@mattip
Copy link
Member

mattip commented Sep 7, 2022

Tests are missing. For an example of tests that change the environment variables and then run python in a subprocess, see env manipulation in numpy/core/tests/test_cpu_feature.py

@Micky774
Copy link
Contributor Author

Micky774 commented Sep 13, 2022

I can't reproduce these errors locally. Even when setting up a machine with --cpu-baseline=native correctly, all the tests pass on my end. Wondering if you may have any insights @mattip @seiko2plus

Co-authored-by: Sebastian Berg <sebastian@sipsolutions.net>
@seiko2plus
Copy link
Member

seiko2plus commented Sep 19, 2022

@Micky774, to reproduce this error, you have to disable the dispatch-able features similar to what the smoke test does, for example:

python setup.py build --cpu-baseline=native --cpu-dispatch=none install --user
python runtests.py -n

or just

python runtests.py --cpu-dispatch=none

In general, you have to escape the dispatch tests if __cpu_dispatch__ is equal to None or if it's an empty list.

@seiko2plus
Copy link
Member

seiko2plus commented Sep 19, 2022

@Micky774, please let me know when it's ready for review.

@Micky774
Copy link
Contributor Author

Micky774 commented Sep 20, 2022

@Micky774, please let me know when it's ready for review.

@seiko2plus Should be good now! Thanks for the help :)

@mattip
Copy link
Member

mattip commented Sep 29, 2022

@seiko2plus could you take a look? I think the aarch64 failure is systematic and not connected to this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants