Skip to content

gh-113440: Windows CI: Raise exception with stderr if ver call fails #113392

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from

Conversation

hugovk
Copy link
Member

@hugovk hugovk commented Dec 22, 2023

Fixes gh-113440.

We've had build failures on "Windows / build and test (x86)", after some unknown problem calling the ver command.

For example:

Run .\python.bat -m test.pythoninfo
Running Debug|Win32 interpreter...
ERROR: collect_windows() failed
Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\pythoninfo.py", line 1022, in collect_info
    collect_func(info_add)
    ~~~~~~~~~~~~^^^^^^^^^^
  File "D:\a\cpython\cpython\Lib\test\pythoninfo.py", line [9](https://github.com/python/cpython/actions/runs/7286334537/job/19854864684#step:4:10)34, in collect_windows
    line = output.splitlines()[0]
           ~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Collection failed: exit with error

https://github.com/python/cpython/actions/runs/7286334537/job/19854864684

More examples:

Sometimes it fails on re-run, sometimes it passes.

Let's output the exit code and error message when it happens, and fail fast, to help debug.

@hugovk
Copy link
Member Author

hugovk commented Dec 22, 2023

Ah interesting:

I ran this about 25 times on my fork, they passed every time.

First time with this PR it failed :)

Running Debug|Win32 interpreter...
ERROR: collect_windows() failed
Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\pythoninfo.py", line 1025, in collect_info
    collect_func(info_add)
    ~~~~~~~~~~~~^^^^^^^^^^
  File "D:\a\cpython\cpython\Lib\test\pythoninfo.py", line 929, in collect_windows
    raise ValueError(
    ...<2 lines>...
    )
ValueError: Command 'ver' failed with exit code 3221225794: stdout='' stderr=''

https://github.com/python/cpython/actions/runs/7298627675/job/19889972821?pr=113392#step:4:17

https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-diagnostics-code-package-errors says:

Exit code Hexadecimal value Short description Root cause Potential fix
3221225794 0xc0000142 STATUS_DLL_INIT_FAILED This error sometimes means that the machine has run out of desktop heap space. This cause is especially likely if you have numerous processes that belong to your application running on the node. If your program wasn't built to respond to Ctrl+C signals, you can enable the EnableActivateNoWindow setting in the Cluster manifest. Enabling this setting means your code package will run without a GUI window and won't receive Ctrl+C signals. This action also reduces the amount of desktop heap space each process consumes. If your code package needs to receive Ctrl+C signals, you can increase the size of your node's desktop heap.

@hugovk hugovk added needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes labels Dec 22, 2023
hugovk and others added 2 commits December 22, 2023 13:32
Co-authored-by: Steve Dower <steve.dower@microsoft.com>
@zooba
Copy link
Member

zooba commented Dec 22, 2023

Looks like it's actually a problem with PATH then. I don't have a problem with leaving that env code in there, though it's probably worth a comment.

@hugovk
Copy link
Member Author

hugovk commented Dec 22, 2023

Would you like to suggest a comment?

@zooba
Copy link
Member

zooba commented Dec 22, 2023

Avoid interference from CI systems with messy environments (gh-113392)?

@zooba
Copy link
Member

zooba commented Dec 22, 2023

Looks like it's gotten stuck this time, which is very interesting. Perhaps the previous run was on the successful side of the 50/50?

@hugovk
Copy link
Member Author

hugovk commented Dec 22, 2023

I ran about 20 times on my fork, but it passed each time.

I'll restart it. But getting stuck is worse than stopping with an error...

@hugovk
Copy link
Member Author

hugovk commented Dec 22, 2023

I won't restart it as it's already been restarted twice.

ERROR: collect_windows() failed
Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\pythoninfo.py", line 1027, in collect_info
    collect_func(info_add)
    ~~~~~~~~~~~~^^^^^^^^^^
  File "D:\a\cpython\cpython\Lib\test\pythoninfo.py", line 931, in collect_windows
    raise ValueError(
    ...<2 lines>...
    )
ValueError: Command 'ver' failed with exit code 3221225794: stdout='' stderr=''

@zooba
Copy link
Member

zooba commented Dec 22, 2023

Okay, guess we can assume that PATH isn't the culprit (and revert that change).

I've been able to find some (private) discussion about a possible issue that looks like this. It suggests that STARTUPINFOW.lpDesktop needs to be initialised to Winsta0\Default rather than being left empty for cases where the process is being launched into a new window (which shell=True should do) from a service account (which we shouldn't be, but maybe we are?).

Currently it's _winapi_CreateProcess_impl in Modules\_winapi.c that needs to set that value (in si.StartupInfo). It's probably not a real behaviour change to set it unconditionally, but ideally we'd support it on the startup info object. Might be worth making the change just for testing. It might also be possible to repro locally using psexec to run python Lib\test\pythoninfo.py as NETWORK SERVICE.

A simpler fix might be to pass CREATE_NO_WINDOW in the creation flags, which ought to bypass the entire code path that leads to the issue above. But I see no suggestion of that being a fix/workaround (because in the general case where you need a window, it's not).

This reverts commit 033142a.
This reverts commit 0cb76cb.
@serhiy-storchaka
Copy link
Member

I compared successful and failing logs, and they are identical to this point, ignoring time, commit hash, git pull progress indicator and the order of compiling files.

@serhiy-storchaka
Copy link
Member

Is there an open issue? I'm trying to ignore this error, if it can help, because it blocks our work. See #113435.

@hugovk
Copy link
Member Author

hugovk commented Dec 23, 2023

I don't think there's an issue.

Your PR looks good to unblock, perhaps combine with this to log the error rather than raising an exception?

@serhiy-storchaka
Copy link
Member

@hugovk Then could you please open an issue and copy your description to it? We will link both PRs to this issue, unblock the workflow, and then continue investigation of the problem.

@hugovk
Copy link
Member Author

hugovk commented Dec 23, 2023

Certainly, I've opened issue #113440.

@hugovk hugovk changed the title Windows CI: Raise exception with stderr if ver call fails gh-113440: Windows CI: Raise exception with stderr if ver call fails Dec 23, 2023
@hugovk
Copy link
Member Author

hugovk commented Mar 25, 2024

Do we still need this or can we close it (and #113440)?

@serhiy-storchaka
Copy link
Member

Do you still need this?

The ver error no longer blocks CI, but the cause of the failure is still unclear. It would be nice to find and fix the cause, but for now we can just ignore it.

@hugovk hugovk removed the needs backport to 3.11 only security fixes label Apr 2, 2024
@serhiy-storchaka serhiy-storchaka added the needs backport to 3.13 bugs and security fixes label May 9, 2024
@hugovk hugovk closed this Jun 16, 2024
@hugovk hugovk deleted the ver-stderr branch June 18, 2024 13:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Windows 32-bit CI: failures calling ver command
3 participants