inspect.signature() doesn't parse __text_signature__ containing a newline character #85267

vstinner · 2020-06-23T22:26:29Z

BPO	41095
Nosy	@vstinner, @serhiy-storchaka, @1st1, @asottile
PRs	bpo-31938: Fix default-value signatures of several functions in the select module #21066

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-06-23.22:26:29.252>
labels = ['library', '3.10']
title = "inspect.signature() doesn't parse __text_signature__ containing a newline character"
updated_at = <Date 2020-06-24.01:12:14.804>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-06-24.01:12:14.804>
actor = 'Anthony Sottile'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-06-23.22:26:29.252>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 41095
keywords = ['patch']
message_count = 2.0
messages = ['372213', '372222']
nosy_count = 4.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'yselivanov', 'Anthony Sottile']
pr_nums = ['21066']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue41095'
versions = ['Python 3.10']

PR: gh-85267: Improvements to inspect.signature __text_signature__ handling #98796

PR: [3.11] gh-85267: Improvements to inspect.signature __text_signature__ handling (GH-98796) #100392

PR: [3.10] gh-85267: Improvements to inspect.signature __text_signature__ handling (GH-98796) #100393

vstinner · 2020-06-23T22:26:29Z

$ ./python
Python 3.10.0a0 (heads/unicode_latin1:40855c7064, Jun 24 2020, 00:20:07) 
>>> import select
>>> select.epoll.register.__text_signature__
'($self, /, fd,\n         eventmask=select.EPOLLIN | select.EPOLLPRI | select.EPOLLOUT)'

>>> import inspect
>>> inspect.signature(select.epoll.register)
<Signature (self, /, fd)>

=> eventmask parameter is gone!

Either signature() must raise an exception, or it must handle a __text_signature__ containing a newline character.

Issue spotted on bpo-31938 when fixing "./python -m pydoc select".

By the way, as expected, pydoc shows:

Help on method_descriptor in select.epoll:
---

$ ./python -m pydoc select.epoll.register

select.epoll.register = register(self, /, fd)
    Registers a new fd or raises an OSError if the fd is already registered.
(...)

asottile · 2020-06-24T01:12:15Z

Looking into this, it appears to be due to the default value and not due to the newline

I've stumbled upon two simplifications to the routines in inspect but not a fix for this

bpo-20684: Remove unused inspect._signature_get_bound_param #21100
gh-99815: remove unused 'invalid' sentinel value and code that checks for it in inspect.signature parsing #21104

the following code is hit because ast.literal_eval(...) fails for the |d expression:

cpython/Lib/inspect.py

Lines 2069 to 2070 in 2f9ada9

    
           if o is invalid: 
        
               return None

this causes the parameter to be skipped entirely

…handling This makes a couple related changes to inspect.signature's behaviour when parsing a signature from `__text_signature__`. First, `inspect.signature` is documented as only raising ValueError or TypeError. However, in some cases, we could raise RuntimeError. This PR changes that, thereby fixing python#83685. (Note that the new ValueErrors in RewriteSymbolics are caught and then reraised with a message) Second, `inspect.signature` could randomly drop parameters that it didn't understand (corresponding to `return None` in the `p` function). This is the core issue in python#85267. I think this is very surprising behaviour and it seems better to fail outright. Third, adding this new failure broke a couple tests. To fix them (and to e.g. allow `inspect.signature(select.epoll.register)` as in python#85267), I add constant folding of a couple binary operations to RewriteSymbolics. (There's some discussion of making signature expression evaluation arbitrary powerful in python#68155. I think that's out of scope. The additional constant folding here is pretty straightforward, useful, and not much of a slippery slope) Fourth, while python#85267 is incorrect about the cause of the issue, it turns out if you had consecutive newlines in __text_signature__, you'd get `tokenize.TokenError`. Finally, the `if name is invalid:` code path was dead, since `parse_name` never returned `invalid`.

vstinner · 2022-11-03T18:53:34Z

Sadly, I don't have the bandwidth to work on this issue, so I just close it.

JelleZijlstra · 2022-11-08T04:00:55Z

Reopening as there's an active PR fixing this.

vstinner · 2022-11-08T07:20:15Z

What is the active PR?

JelleZijlstra · 2022-11-08T14:31:56Z

#98796

…ng (#98796) This makes a couple related changes to inspect.signature's behaviour when parsing a signature from `__text_signature__`. First, `inspect.signature` is documented as only raising ValueError or TypeError. However, in some cases, we could raise RuntimeError. This PR changes that, thereby fixing #83685. (Note that the new ValueErrors in RewriteSymbolics are caught and then reraised with a message) Second, `inspect.signature` could randomly drop parameters that it didn't understand (corresponding to `return None` in the `p` function). This is the core issue in #85267. I think this is very surprising behaviour and it seems better to fail outright. Third, adding this new failure broke a couple tests. To fix them (and to e.g. allow `inspect.signature(select.epoll.register)` as in #85267), I add constant folding of a couple binary operations to RewriteSymbolics. (There's some discussion of making signature expression evaluation arbitrary powerful in #68155. I think that's out of scope. The additional constant folding here is pretty straightforward, useful, and not much of a slippery slope) Fourth, while #85267 is incorrect about the cause of the issue, it turns out if you had consecutive newlines in __text_signature__, you'd get `tokenize.TokenError`. Finally, the `if name is invalid:` code path was dead, since `parse_name` never returned `invalid`.

…ture__ handling (pythonGH-98796) This makes a couple related changes to inspect.signature's behaviour when parsing a signature from `__text_signature__`. First, `inspect.signature` is documented as only raising ValueError or TypeError. However, in some cases, we could raise RuntimeError. This PR changes that, thereby fixing pythonGH-83685. (Note that the new ValueErrors in RewriteSymbolics are caught and then reraised with a message) Second, `inspect.signature` could randomly drop parameters that it didn't understand (corresponding to `return None` in the `p` function). This is the core issue in pythonGH-85267. I think this is very surprising behaviour and it seems better to fail outright. Third, adding this new failure broke a couple tests. To fix them (and to e.g. allow `inspect.signature(select.epoll.register)` as in pythonGH-85267), I add constant folding of a couple binary operations to RewriteSymbolics. (There's some discussion of making signature expression evaluation arbitrary powerful in pythonGH-68155. I think that's out of scope. The additional constant folding here is pretty straightforward, useful, and not much of a slippery slope) Fourth, while pythonGH-85267 is incorrect about the cause of the issue, it turns out if you had consecutive newlines in __text_signature__, you'd get `tokenize.TokenError`. Finally, the `if name is invalid:` code path was dead, since `parse_name` never returned `invalid`.. (cherry picked from commit 79311cb) Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>

… handling (GH-98796) (#100392) This makes a couple related changes to inspect.signature's behaviour when parsing a signature from `__text_signature__`. First, `inspect.signature` is documented as only raising ValueError or TypeError. However, in some cases, we could raise RuntimeError. This PR changes that, thereby fixing GH-83685. (Note that the new ValueErrors in RewriteSymbolics are caught and then reraised with a message) Second, `inspect.signature` could randomly drop parameters that it didn't understand (corresponding to `return None` in the `p` function). This is the core issue in GH-85267. I think this is very surprising behaviour and it seems better to fail outright. Third, adding this new failure broke a couple tests. To fix them (and to e.g. allow `inspect.signature(select.epoll.register)` as in GH-85267), I add constant folding of a couple binary operations to RewriteSymbolics. (There's some discussion of making signature expression evaluation arbitrary powerful in GH-68155. I think that's out of scope. The additional constant folding here is pretty straightforward, useful, and not much of a slippery slope) Fourth, while GH-85267 is incorrect about the cause of the issue, it turns out if you had consecutive newlines in __text_signature__, you'd get `tokenize.TokenError`. Finally, the `if name is invalid:` code path was dead, since `parse_name` never returned `invalid`.. (cherry picked from commit 79311cb) Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>

… handling (GH-98796) (#100393) This makes a couple related changes to inspect.signature's behaviour when parsing a signature from `__text_signature__`. First, `inspect.signature` is documented as only raising ValueError or TypeError. However, in some cases, we could raise RuntimeError. This PR changes that, thereby fixing GH-83685. (Note that the new ValueErrors in RewriteSymbolics are caught and then reraised with a message) Second, `inspect.signature` could randomly drop parameters that it didn't understand (corresponding to `return None` in the `p` function). This is the core issue in GH-85267. I think this is very surprising behaviour and it seems better to fail outright. Third, adding this new failure broke a couple tests. To fix them (and to e.g. allow `inspect.signature(select.epoll.register)` as in GH-85267), I add constant folding of a couple binary operations to RewriteSymbolics. (There's some discussion of making signature expression evaluation arbitrary powerful in GH-68155. I think that's out of scope. The additional constant folding here is pretty straightforward, useful, and not much of a slippery slope) Fourth, while GH-85267 is incorrect about the cause of the issue, it turns out if you had consecutive newlines in __text_signature__, you'd get `tokenize.TokenError`. Finally, the `if name is invalid:` code path was dead, since `parse_name` never returned `invalid`.. (cherry picked from commit 79311cb) Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>

vstinner added 3.10 stdlib Python modules in the Lib dir labels Jun 23, 2020

ezio-melotti transferred this issue from another repository Apr 10, 2022

hauntsaninja mentioned this issue Oct 28, 2022

gh-85267: Improvements to inspect.signature __text_signature__ handling #98796

Merged

vstinner closed this as completed Nov 3, 2022

JelleZijlstra reopened this Nov 8, 2022

hauntsaninja mentioned this issue Dec 21, 2022

[3.11] gh-85267: Improvements to inspect.signature __text_signature__ handling (GH-98796) #100392

Merged

hauntsaninja mentioned this issue Dec 21, 2022

[3.10] gh-85267: Improvements to inspect.signature __text_signature__ handling (GH-98796) #100393

Merged

hauntsaninja closed this as completed Dec 21, 2022

hauntsaninja reopened this Dec 21, 2022

hauntsaninja closed this as completed Dec 21, 2022

inspect.signature() doesn't parse __text_signature__ containing a newline character #85267

inspect.signature() doesn't parse __text_signature__ containing a newline character #85267

vstinner commented Jun 23, 2020 •

edited by bedevere-bot

vstinner commented Jun 23, 2020

asottile mannequin commented Jun 24, 2020

vstinner commented Nov 3, 2022

JelleZijlstra commented Nov 8, 2022

vstinner commented Nov 8, 2022

JelleZijlstra commented Nov 8, 2022

inspect.signature() doesn't parse __text_signature__ containing a newline character #85267

inspect.signature() doesn't parse __text_signature__ containing a newline character #85267

Comments

vstinner commented Jun 23, 2020 • edited by bedevere-bot

vstinner commented Jun 23, 2020

asottile mannequin commented Jun 24, 2020

vstinner commented Nov 3, 2022

JelleZijlstra commented Nov 8, 2022

vstinner commented Nov 8, 2022

JelleZijlstra commented Nov 8, 2022

vstinner commented Jun 23, 2020 •

edited by bedevere-bot