Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-85267: Improvements to inspect.signature __text_signature__ handling #98796

Merged
merged 7 commits into from Dec 21, 2022

Conversation

hauntsaninja
Copy link
Contributor

@hauntsaninja hauntsaninja commented Oct 28, 2022

This makes a couple related changes to inspect.signature's behaviour when parsing a signature from __text_signature__.

First, inspect.signature is documented as only raising ValueError or TypeError. However, in some cases, we could raise RuntimeError. This PR changes that, thereby fixing #83685.

(Note that the new ValueErrors in RewriteSymbolics are caught and then reraised with a message)

Second, inspect.signature could randomly drop parameters that it didn't understand (corresponding to return None in the p function). This is the core issue in #85267. I think this is very surprising behaviour and it seems better to fail outright.

Third, adding this new failure broke a couple tests. To fix them (and to e.g. allow inspect.signature(select.epoll.register) as in #85267), I add constant folding of a couple binary operations to RewriteSymbolics.

(There's some discussion of making signature expression evaluation arbitrary powerful in #68155. I think that's out of scope. The additional constant folding here is pretty straightforward, useful, and not much of a slippery slope)

Fourth, while #85267 is incorrect about the cause of the issue, it turns out if you had consecutive newlines in __text_signature__, you'd get tokenize.TokenError.

Finally, the if name is invalid: code path was dead, since parse_name never returned invalid.

…handling

This makes a couple related changes to inspect.signature's behaviour
when parsing a signature from `__text_signature__`.

First, `inspect.signature` is documented as only raising ValueError or
TypeError. However, in some cases, we could raise RuntimeError.  This PR
changes that, thereby fixing python#83685.

(Note that the new ValueErrors in RewriteSymbolics are caught and then
reraised with a message)

Second, `inspect.signature` could randomly drop parameters that it
didn't understand (corresponding to `return None` in the `p` function).
This is the core issue in python#85267. I think this is very surprising
behaviour and it seems better to fail outright.

Third, adding this new failure broke a couple tests. To fix them (and to
e.g. allow `inspect.signature(select.epoll.register)` as in python#85267), I
add constant folding of a couple binary operations to RewriteSymbolics.

(There's some discussion of making signature expression evaluation
arbitrary powerful in python#68155. I think that's out of scope. The
additional constant folding here is pretty straightforward, useful, and
not much of a slippery slope)

Fourth, while python#85267 is incorrect about the cause of the issue, it turns
out if you had consecutive newlines in __text_signature__, you'd get
`tokenize.TokenError`.

Finally, the `if name is invalid:` code path was dead, since
`parse_name` never returned `invalid`.
Lib/inspect.py Show resolved Hide resolved
@hauntsaninja
Copy link
Contributor Author

hauntsaninja commented Oct 28, 2022

The tokenize.TokenError fix could be made its own PR, but the other changes are somewhat linked.

@@ -2209,11 +2208,11 @@ def wrap_value(s):
try:
value = eval(s, sys_module_dict)
except NameError:
raise RuntimeError()
raise ValueError

Copy link
Contributor Author

@hauntsaninja hauntsaninja Oct 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These now get caught on L2254 and acquire an error message there

@JelleZijlstra JelleZijlstra self-requested a review Nov 3, 2022
Copy link
Member

@JelleZijlstra JelleZijlstra left a comment

Looks good, just a few comments.

Can we backport this to the bugfix branches? All of the changes here seem like they fix buggy behavior, so I'm leaning to yes.

Lib/inspect.py Show resolved Hide resolved
Lib/test/test_inspect.py Show resolved Hide resolved
Lib/test/test_inspect.py Outdated Show resolved Hide resolved
Lib/inspect.py Show resolved Hide resolved
hauntsaninja and others added 4 commits Nov 9, 2022
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
@hauntsaninja
Copy link
Contributor Author

hauntsaninja commented Nov 9, 2022

Thanks! Made the requested changes. I agree that this seems backport-able

@netlify
Copy link

netlify bot commented Dec 12, 2022

Deploy Preview for python-cpython-preview canceled.

Name Link
🔨 Latest commit fef9787
🔍 Latest deploy log https://app.netlify.com/sites/python-cpython-preview/deploys/6396ac051af5c300087493ef

@JelleZijlstra
Copy link
Member

JelleZijlstra commented Dec 12, 2022

#21104 did a subset of what this PR is changing, fixed the merge conflict.

@JelleZijlstra JelleZijlstra self-assigned this Dec 12, 2022
@JelleZijlstra JelleZijlstra merged commit 79311cb into python:main Dec 21, 2022
20 checks passed
@miss-islington
Copy link
Contributor

miss-islington commented Dec 21, 2022

Thanks @hauntsaninja for the PR, and @JelleZijlstra for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒🤖 I'm not a witch! I'm not a witch!

@miss-islington
Copy link
Contributor

miss-islington commented Dec 21, 2022

Sorry, @hauntsaninja and @JelleZijlstra, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 79311cbfe718f17c89bab67d7f89da3931bfa2ac 3.11

@miss-islington
Copy link
Contributor

miss-islington commented Dec 21, 2022

Sorry @hauntsaninja and @JelleZijlstra, I had trouble checking out the 3.10 backport branch.
Please retry by removing and re-adding the "needs backport to 3.10" label.
Alternatively, you can backport using cherry_picker on the command line.
cherry_picker 79311cbfe718f17c89bab67d7f89da3931bfa2ac 3.10

@JelleZijlstra
Copy link
Member

JelleZijlstra commented Dec 21, 2022

@hauntsaninja do you want to do the manual backports?

@bedevere-bot
Copy link

bedevere-bot commented Dec 21, 2022

GH-100392 is a backport of this pull request to the 3.11 branch.

@hauntsaninja hauntsaninja deleted the inspect-signature branch Dec 21, 2022
hauntsaninja added a commit to hauntsaninja/cpython that referenced this pull request Dec 21, 2022
…ture__ handling (pythonGH-98796)

This makes a couple related changes to inspect.signature's behaviour
when parsing a signature from `__text_signature__`.

First, `inspect.signature` is documented as only raising ValueError or
TypeError. However, in some cases, we could raise RuntimeError.  This PR
changes that, thereby fixing pythonGH-83685.

(Note that the new ValueErrors in RewriteSymbolics are caught and then
reraised with a message)

Second, `inspect.signature` could randomly drop parameters that it
didn't understand (corresponding to `return None` in the `p` function).
This is the core issue in pythonGH-85267. I think this is very surprising
behaviour and it seems better to fail outright.

Third, adding this new failure broke a couple tests. To fix them (and to
e.g. allow `inspect.signature(select.epoll.register)` as in pythonGH-85267), I
add constant folding of a couple binary operations to RewriteSymbolics.

(There's some discussion of making signature expression evaluation
arbitrary powerful in pythonGH-68155. I think that's out of scope. The
additional constant folding here is pretty straightforward, useful, and
not much of a slippery slope)

Fourth, while pythonGH-85267 is incorrect about the cause of the issue, it turns
out if you had consecutive newlines in __text_signature__, you'd get
`tokenize.TokenError`.

Finally, the `if name is invalid:` code path was dead, since
`parse_name` never returned `invalid`..
(cherry picked from commit 79311cb)

Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
@bedevere-bot
Copy link

bedevere-bot commented Dec 21, 2022

GH-100393 is a backport of this pull request to the 3.10 branch.

hauntsaninja added a commit to hauntsaninja/cpython that referenced this pull request Dec 21, 2022
…ture__ handling (pythonGH-98796)

This makes a couple related changes to inspect.signature's behaviour
when parsing a signature from `__text_signature__`.

First, `inspect.signature` is documented as only raising ValueError or
TypeError. However, in some cases, we could raise RuntimeError.  This PR
changes that, thereby fixing pythonGH-83685.

(Note that the new ValueErrors in RewriteSymbolics are caught and then
reraised with a message)

Second, `inspect.signature` could randomly drop parameters that it
didn't understand (corresponding to `return None` in the `p` function).
This is the core issue in pythonGH-85267. I think this is very surprising
behaviour and it seems better to fail outright.

Third, adding this new failure broke a couple tests. To fix them (and to
e.g. allow `inspect.signature(select.epoll.register)` as in pythonGH-85267), I
add constant folding of a couple binary operations to RewriteSymbolics.

(There's some discussion of making signature expression evaluation
arbitrary powerful in pythonGH-68155. I think that's out of scope. The
additional constant folding here is pretty straightforward, useful, and
not much of a slippery slope)

Fourth, while pythonGH-85267 is incorrect about the cause of the issue, it turns
out if you had consecutive newlines in __text_signature__, you'd get
`tokenize.TokenError`.

Finally, the `if name is invalid:` code path was dead, since
`parse_name` never returned `invalid`..
(cherry picked from commit 79311cb)

Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
@JelleZijlstra
Copy link
Member

JelleZijlstra commented Dec 21, 2022

Thank you!

JelleZijlstra pushed a commit that referenced this pull request Dec 21, 2022
… handling (GH-98796) (#100392)

This makes a couple related changes to inspect.signature's behaviour
when parsing a signature from `__text_signature__`.

First, `inspect.signature` is documented as only raising ValueError or
TypeError. However, in some cases, we could raise RuntimeError.  This PR
changes that, thereby fixing GH-83685.

(Note that the new ValueErrors in RewriteSymbolics are caught and then
reraised with a message)

Second, `inspect.signature` could randomly drop parameters that it
didn't understand (corresponding to `return None` in the `p` function).
This is the core issue in GH-85267. I think this is very surprising
behaviour and it seems better to fail outright.

Third, adding this new failure broke a couple tests. To fix them (and to
e.g. allow `inspect.signature(select.epoll.register)` as in GH-85267), I
add constant folding of a couple binary operations to RewriteSymbolics.

(There's some discussion of making signature expression evaluation
arbitrary powerful in GH-68155. I think that's out of scope. The
additional constant folding here is pretty straightforward, useful, and
not much of a slippery slope)

Fourth, while GH-85267 is incorrect about the cause of the issue, it turns
out if you had consecutive newlines in __text_signature__, you'd get
`tokenize.TokenError`.

Finally, the `if name is invalid:` code path was dead, since
`parse_name` never returned `invalid`..
(cherry picked from commit 79311cb)

Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
JelleZijlstra pushed a commit that referenced this pull request Dec 21, 2022
… handling (GH-98796) (#100393)

This makes a couple related changes to inspect.signature's behaviour
when parsing a signature from `__text_signature__`.

First, `inspect.signature` is documented as only raising ValueError or
TypeError. However, in some cases, we could raise RuntimeError.  This PR
changes that, thereby fixing GH-83685.

(Note that the new ValueErrors in RewriteSymbolics are caught and then
reraised with a message)

Second, `inspect.signature` could randomly drop parameters that it
didn't understand (corresponding to `return None` in the `p` function).
This is the core issue in GH-85267. I think this is very surprising
behaviour and it seems better to fail outright.

Third, adding this new failure broke a couple tests. To fix them (and to
e.g. allow `inspect.signature(select.epoll.register)` as in GH-85267), I
add constant folding of a couple binary operations to RewriteSymbolics.

(There's some discussion of making signature expression evaluation
arbitrary powerful in GH-68155. I think that's out of scope. The
additional constant folding here is pretty straightforward, useful, and
not much of a slippery slope)

Fourth, while GH-85267 is incorrect about the cause of the issue, it turns
out if you had consecutive newlines in __text_signature__, you'd get
`tokenize.TokenError`.

Finally, the `if name is invalid:` code path was dead, since
`parse_name` never returned `invalid`..
(cherry picked from commit 79311cb)

Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants