Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Emscripten test worker sometimes fails witth "BlockingIOError: write could not complete without blocking" #94026

Closed
tiran opened this issue Jun 20, 2022 · 4 comments
Closed

Emscripten test worker sometimes fails witth "BlockingIOError: write could not complete without blocking" #94026

tiran opened this issue Jun 20, 2022 · 4 comments
Labels
3.11 3.12 tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@tiran
Copy link
Member

tiran commented Jun 20, 2022

Bug report

Sometimes test worker processes on wasm32-emscripten build bot fails with BlockingIOError: write could not complete without blocking. Example https://buildbot.python.org/all/#/builders/1044/builds/27/steps/10/logs/stdio

0:37:30 load avg: 4.42 [422/436/3] test_pathlib crashed (Exit code 1)
... 
result=Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node/build/Lib/test/regrtest.py", line 47, in <module>
    _main()
    ^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node/build/Lib/test/regrtest.py", line 43, in _main
    main()
    ^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node/build/Lib/test/libregrtest/main.py", line 754, in main
    Regrtest().main(tests=tests, **kwargs)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node/build/Lib/test/libregrtest/main.py", line 692, in main
    self._main(tests, kwargs)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node/build/Lib/test/libregrtest/main.py", line 713, in _main
    run_tests_worker(self.ns, self.worker_test_name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node/build/Lib/test/libregrtest/runtest_mp.py", line 102, in run_tests_worker
    print(json.dumps(result, cls=EncodeTestResult), flush=True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
BlockingIOError: [Errno 6] write could not complete without blocking

The error seems to occur at the end of a test job when the worker process returns its status to the control process. The control process runs CPython natively (x86_64-pc-linux-gnu in case of the build bot). The worker processes is wasm32-emscripten CPython with NodeJS as runner. I guess that every now and then, the test worker process produces output faster than the control process can consume from the pipe that connects stdout of the worker and read pipe of the control process. The runtime does not like blocking IO and returns EAGAIN.

>>> import sys, errno
>>> sys.platform
'emscripten'
>>> errno.errorcode[6]
'EAGAIN'
@tiran tiran added type-bug An unexpected behavior, bug, or error tests Tests in the Lib/test dir 3.11 3.12 labels Jun 20, 2022
@tiran
Copy link
Member Author

tiran commented Jun 20, 2022

The fcntl module is currently not available on Emscripten. I hacked the module manually and was able to get flags from the stdout fd. Emscripten does not report that the fd is set to non-blocking mode.

>>> import os, fcntl
>>> fcntl.fcntl(1, fcntl.F_GETFL)
577
>>> fcntl.fcntl(1, fcntl.F_GETFL) & os.O_NONBLOCK
0

@tiran
Copy link
Member Author

tiran commented Jun 21, 2022

I applied a hack on the buildbot worker host. Buildbot now starts NodeJS with nice -n5. That seems to be good enough to make the tests pass.

@bedevere-bot bedevere-bot mentioned this issue Jun 23, 2022
Closed
tiran added a commit to tiran/cpython that referenced this issue Jun 23, 2022
@tiran
pythongh-94026: Workaround for BlockingIOError in runtest_mp on Emscr…
a937ec9 
…ipten

`runtest_mp` test worker now handles :exc:`BlockingIOError` to work around
non-blocking pipes on Emscripten.
@serhiy-storchaka
Copy link
Member

serhiy-storchaka commented Jun 23, 2022

Why is it set to non-blocking mode? Can it be set to blocking mode? Can it be replaced with an fd set to blocking mode? Can TextIOWrapper and BufferedWriter be made to handle this internally? Can we use an alternate channel for communication between processes?

@tiran
Copy link
Member Author

tiran commented Jun 23, 2022 •
edited

It is set to non-blocking mode because different rules apply to browser-like environments. It is not possible to change the property. In fact there is no way to even detect that the pipe fd is in non-blocking mode. I can only deduce that it is in non-blocking mode because tests sometimes fail when the system is under high load and the other side cannot read from the pipe fast enough.

Emscripten tries hard to mimic a POSIX-like OS on top of a JavaScript and WebAssembly engine. Ultimately it is bound by restriction of the runtime environment.

To handle the special case in Python's io module we would first have to detect the special case. That is not possible on Emscripten yet. The fcntl module is not available because I disabled ioctl as it leads to crashes. The fcntl(F_GETFD) syscall does not return O_NONBLOCK and fstat does not work on pipes and standard streams.

We could only use a temporary file for communication. Socket's don't work as expected. IPC and socketpair are not available.

This was referenced Jun 24, 2022
Closed
Merged
tiran added a commit to tiran/cpython that referenced this issue Jun 29, 2022
@tiran @vstinner
pythongh-94026: Buffer regrtest worker stdout in temporary file
198027a 
Co-authored-by: Victor Stinner <vstinner@python.org>
tiran added a commit that referenced this issue Jun 29, 2022
@tiran @vstinner
gh-94026: Buffer regrtest worker stdout in temporary file (GH-94253)
199ba23 
Co-authored-by: Victor Stinner <vstinner@python.org>
tiran added a commit to tiran/cpython that referenced this issue Jun 29, 2022
@tiran
[3.11] pythongh-94026: Buffer regrtest worker stdout in temporary file (
5dea1bb 
pythonGH-94253)

Co-authored-by: Victor Stinner <vstinner@python.org>.
(cherry picked from commit 199ba23)

Co-authored-by: Christian Heimes <christian@python.org>
@bedevere-bot bedevere-bot mentioned this issue Jun 29, 2022
Merged
@tiran tiran closed this as completed Jun 29, 2022
tiran added a commit that referenced this issue Jun 29, 2022
@tiran
[3.11] gh-94026: Buffer regrtest worker stdout in temporary file (GH-…
0122ab2 
…94253) (GH-94408)

Co-authored-by: Victor Stinner <vstinner@python.org>.
Co-authored-by: Christian Heimes <christian@python.org>
gvanrossum pushed a commit to gvanrossum/cpython that referenced this issue Jun 30, 2022
@tiran @vstinner @gvanrossum
pythongh-94026: Buffer regrtest worker stdout in temporary file (pyth…
2e919ba 
…onGH-94253)

Co-authored-by: Victor Stinner <vstinner@python.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.11 3.12 tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tiran @serhiy-storchaka