Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-42005: Fix CLI of cProfile and profile to catch BrokenPipeError #22643

Merged
merged 1 commit into from Jan 20, 2021

Conversation

zmwangx
Copy link
Contributor

@zmwangx zmwangx commented Oct 11, 2020

Catch BrokenPipeError in the CLI of cProfile and profile to reduce noise when piping through head.

Prior art of catching BrokenPipeError in a PSL CLI: bpo-39828 #18779 targeting json.tool.

A test can be added but I've held off for now since it would be rather unwieldy and of pretty limited value.

https://bugs.python.org/issue42005

@zmwangx zmwangx force-pushed the profile-handle-brokenpipeerror branch 2 times, most recently from a12e713 to ce36ff1 Compare Oct 11, 2020
@vstinner
Copy link
Member

vstinner commented Oct 11, 2020

Please open a new issue: https://bugs.python.org/issue39828 is now closed.

@vstinner
Copy link
Member

vstinner commented Oct 11, 2020

Please open a new issue: https://bugs.python.org/issue39828 is now closed.

Ignore my comment. This PR was added to https://bugs.python.org/issue39828 because you used " bpo-39828" in the commit message. But you already used a new issue: https://bugs.python.org/issue42005

@zmwangx
Copy link
Contributor Author

zmwangx commented Oct 11, 2020

Actually it's not in the commit message, I only referenced bpo-39828 for context in the PR body. Sorry it accidentally confused the bot.

Lib/cProfile.py Outdated
runctx(code, globs, None, options.outfile, options.sort)
except BrokenPipeError as exc:
# Prevent "Exception ignored" during interpreter shutdown.
sys.stdout = sys.stderr = io.StringIO()
Copy link
Member

@vstinner vstinner Oct 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting stderr to a new stringIO can hide other errors written into stderr at exit.

BrokenPipeError is more likely related to stdout than stderr. I understand that your intent is to ignore the warning displayed on stderr when sys.stdout file is closed.

There is now sys.unraisablehook which can be overidden temporarily to not log the exception on stderr when stdout is closed.

Since json.tool already has a similar code pattern, maybe it would be worth it to provide a helper function somewhere. I would prefer a private function.

By the way, it might be better to delete sys.stdout or set it to None, rather than creating a StringIO. So if the code after that tries to write tons of lines into sys.stdout, the writes are just ignored/lost, rather than growing memory (which will never be consumed anyway).

@zmwangx zmwangx force-pushed the profile-handle-brokenpipeerror branch from ce36ff1 to 03387c4 Compare Oct 11, 2020
@zmwangx
Copy link
Contributor Author

zmwangx commented Oct 11, 2020

@vstinner Thanks for the review.

BrokenPipeError is more likely related to stdout than stderr.

Absolutely.

I understand that your intent is to ignore the warning displayed on stderr when sys.stdout file is closed.

In fact setting sys.stdout was enough to suppress the unraisable exception. I threw sys.stderr in there without much thought just in case someone has stderr closed too (say if stderr is redirected to stdout). Now I realize it's more trouble than it's worth.

Since json.tool already has a similar code pattern, maybe it would be worth it to provide a helper function somewhere. I would prefer a private function.

Are you suggesting something like

def run_with_broken_pipe_awareness(func, *args, **kwargs):
    try:
        return func(*args, **kwargs)
    except BrokenPipeError as exc:
        sys.stdout = None
        sys.exit(exc.errno)

? If I were to add such a private helper, where should I place it? I can't seem to find a place in the code base where cross-module helpers live.

By the way, it might be better to delete sys.stdout or set it to None

I erroneously thought it needs to be set to something that still supports write() or it would again cause an exception. Apparently it's not the case, corrected.

@github-actions
Copy link

github-actions bot commented Dec 17, 2020

This PR is stale because it has been open for 30 days with no activity.

@github-actions github-actions bot added the stale Stale PR or inactive for long period of time. label Dec 17, 2020
@zmwangx zmwangx force-pushed the profile-handle-brokenpipeerror branch from 03387c4 to fa36b03 Compare Dec 17, 2020
@zmwangx
Copy link
Contributor Author

zmwangx commented Dec 17, 2020

I rebased on top of master. Other than that I think I'm just waiting for reviewer(s).

@github-actions github-actions bot removed the stale Stale PR or inactive for long period of time. label Dec 18, 2020
@vstinner vstinner merged commit 3554fa4 into python:master Jan 20, 2021
3 checks passed
@miss-islington
Copy link
Contributor

miss-islington commented Jan 20, 2021

Thanks @zmwangx for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒🤖 I'm not a witch! I'm not a witch!

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jan 20, 2021
(cherry picked from commit 3554fa4)

Co-authored-by: Zhiming Wang <i@zhimingwang.org>
@bedevere-bot
Copy link

bedevere-bot commented Jan 20, 2021

GH-24262 is a backport of this pull request to the 3.9 branch.

@miss-islington
Copy link
Contributor

miss-islington commented Jan 20, 2021

Thanks @zmwangx for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jan 20, 2021
(cherry picked from commit 3554fa4)

Co-authored-by: Zhiming Wang <i@zhimingwang.org>
@bedevere-bot
Copy link

bedevere-bot commented Jan 20, 2021

GH-24263 is a backport of this pull request to the 3.8 branch.

@vstinner
Copy link
Member

vstinner commented Jan 20, 2021

Thanks @zmwangx: I tested your PR and it works as expected, I merged it and backported the fix to 3.8 and 3.9.

$ ./python -m profile -m http.server -h | head 
usage: http.server [-h] [--cgi] [--bind ADDRESS] [--directory DIRECTORY]
                   [port]

positional arguments:
  port                  Specify alternate port [default: 8000]

options:
  -h, --help            show this help message and exit
  --cgi                 Run as CGI Server
  --bind ADDRESS, -b ADDRESS

Previously, it wrote a long traceback:

$ python3.9 -m profile -m http.server -h | head 
usage: http.server [-h] [--cgi] [--bind ADDRESS] [--directory DIRECTORY]
                   [port]

positional arguments:
  port                  Specify alternate port [default: 8000]

optional arguments:
  -h, --help            show this help message and exit
  --cgi                 Run as CGI Server
  --bind ADDRESS, -b ADDRESS
Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/lib64/python3.9/profile.py", line 605, in <module>
    main()
  File "/usr/lib64/python3.9/profile.py", line 598, in main
    runctx(code, globs, None, options.outfile, options.sort)
  File "/usr/lib64/python3.9/profile.py", line 99, in runctx
    return _Utils(Profile).runctx(statement, globals, locals, filename, sort)
  File "/usr/lib64/python3.9/profile.py", line 66, in runctx
    self._show(prof, filename, sort)
  File "/usr/lib64/python3.9/profile.py", line 72, in _show
    prof.print_stats(sort)
  File "/usr/lib64/python3.9/profile.py", line 388, in print_stats
    pstats.Stats(self).strip_dirs().sort_stats(sort). \
  File "/usr/lib64/python3.9/pstats.py", line 431, in print_stats
    self.print_line(func)
  File "/usr/lib64/python3.9/pstats.py", line 519, in print_line
    print(func_std_string(func), file=self.stream)
BrokenPipeError: [Errno 32] Broken pipe

@zmwangx
Copy link
Contributor Author

zmwangx commented Jan 20, 2021

Thanks!

miss-islington added a commit that referenced this pull request Jan 20, 2021
(cherry picked from commit 3554fa4)

Co-authored-by: Zhiming Wang <i@zhimingwang.org>
miss-islington added a commit that referenced this pull request Jan 20, 2021
(cherry picked from commit 3554fa4)

Co-authored-by: Zhiming Wang <i@zhimingwang.org>
@zmwangx zmwangx deleted the profile-handle-brokenpipeerror branch Jan 20, 2021
adorilson pushed a commit to adorilson/cpython that referenced this pull request Mar 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants