Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-46720: Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems #31279

Merged
merged 18 commits into from Apr 22, 2022

Conversation

maggyero
Copy link
Contributor

@maggyero maggyero commented Feb 11, 2022

On Unix-like systems, the function multiprocessing.set_executable accepts any path-like objects (str, bytes, and os.PathLike) as its executable argument for setting up the path to the Python interpreter multiprocessing.spawn._python_exe.

For instance these work (tested on MacOS with all start methods: ‘spawn’, ‘fork’, and ‘forkserver’):

  • multiprocessing.set_executable(sys.executable) (str);
  • multiprocessing.set_executable(sys.executable.encode()) (bytes);
  • multiprocessing.set_executable(pathlib.Path(sys.executable)) (os.PathLike).

This is because

  • the ‘fork’ start method does not execute any program in the child process;

  • the ‘spawn’ start method converts multiprocessing.spawn._python_exe to bytes with os.fsencode before passing it to the function _posixsubprocess.fork_exec to create a child process, and bytes is the expected argv argument type:

            return _posixsubprocess.fork_exec(
                args, [os.fsencode(path)], True, passfds, None, None,
                -1, -1, -1, -1, -1, -1, errpipe_read, errpipe_write,
                False, False, None, None, None, -1, None)
    
  • the ‘forkserver’ start method spawns a server process (like with the ‘spawn’ start method) which then forks itself at each request (like the ‘fork’ start method).

But on Windows, the ‘spawn’ start method (the only start method available on this O.S.) does not convert multiprocessing.spawn._python_exe to str with os.fsdecode before passing it to the function _winapi.CreateProcess to create a child process, whereas str is the expected application_name argument type:

                hp, ht, pid, tid = _winapi.CreateProcess(
                    python_exe, cmd,
                    None, None, False, 0, env, None, None)

So on Windows the function multiprocessing.set_executable accepts only str objects as its executable argument. This pull request fixes this to be on a par with Unix-like systems for which the function accepts any path-like objects.

https://bugs.python.org/issue46720

Lib/multiprocessing/spawn.py Outdated Show resolved Hide resolved
@maggyero maggyero changed the title bpo-46720: Add support of path-like objects to multiprocessing.set_executable for Windows to match Unix-like systems bpo-46720: Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems Feb 11, 2022
@maggyero
Copy link
Contributor Author

maggyero commented Feb 12, 2022

Thanks for the review @eryksun, I have also updated the documentation and added a news entry.

@maggyero
Copy link
Contributor Author

maggyero commented Feb 12, 2022

@brettcannon, since you coauthored PEP 519, can I request your review of this PR?

@brettcannon brettcannon self-requested a review Feb 14, 2022
@brettcannon
Copy link
Member

brettcannon commented Mar 11, 2022

Do we need an explicit test for this?

@maggyero
Copy link
Contributor Author

maggyero commented Mar 12, 2022

Do we need an explicit test for this?

@brettcannon Yes, I think so. I have just added one.

Lib/test/_test_multiprocessing.py Outdated Show resolved Hide resolved
@maggyero
Copy link
Contributor Author

maggyero commented Apr 26, 2022

Thanks for the review @brettcannon!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants