Since asm.js came on the scene, and now Web Assembly people have created CPython patches to support building CPython with emscripten. See:

• https://github.com/PeachPy/EmCPython -- Python 2.7
• https://github.com/dgym/cpython-emscripten/tree/master/3.5.2/patches -- Python 3.5.2
• https://github.com/iodide-project/pyodide/tree/master/cpython/patches -- Python 3.7.4

To ease the compiling of CPython with emscripten it would be helpful if patches that achieved these ends for the compiling to Web Assembly with emscripten were built into the upstream source repository itself.

If web assembly were to became a supported compilation target of the upstream CPython repository this would significantly reduce the friction of allowing CPython, and the latest CPython, to become a language readily usable within the browser.

Cheers,
Simon

Do you want to provide a pull request?

you can add

• https://github.com/pmp-p/pydk/tree/master/sources.em/Python-3.8.0b4.patchset -- Python 3.8.x

(wasm not asm.js, clang-10+ required)

demo https://pmp-p.github.io/python-next/test.html

CPython can already run in the browser with very little patching, but major issues are :

• asyncify'ing the whole wasm VM to have pre-emption over cPython's one to prevent blocking I/O slows down things *a lot* (10x)
  => (very?) bad user experience.

• the size of vm + stdlib ~ 30 MiB and wasm compilation time.
  => bad user experience on first load or slow connexion.

• the lack of threading in wasm MinimumViableProduct specification (but this is the browser standard for now), that leads to rewrite bits of stdlib ( like eg asyncio module )
  => adding more maintenance burden on stdlib (!)

i tested them all and my personnal opinion is : I can see no use case that would favour "stock" cPython wasm versus a blazing fast MicroPytho (or pycopy) wasm flavour or supercharged full stack pyodide.

Hi pmp-p and Serhiy,

I'd be more than happy to attempt a pull request, but I imagine a change
such as this needs to be discussed first, trying not to "rush to make a
patch" (https://www.youtube.com/watch?v=voXVTjwnn-U&feature=youtu.be&t=2546).
Also, I doubt I will do a good job of it... but I am more than happy to try.

A note regarding "supercharged full stack pyodide", potentially without
efforts such as upstreaming into CPython and emscripten the relevant
patches, that supercharged full stack may just unfortunately stagnate. See
pyodide/pyodide#635 (comment)

With respect to blocking when running Python as WASM, I have found running
the WebAssembly CPython within a webworker and signalling data back and
forth causes there to be no UI issues. It ends up being quite a neat set
up. Main down side right now however is the set up is currently going
stale, hence me believing reaching out like this is in the best interests
of Python going forward.

Cheers,
Simon

On Tue, 14 Apr 2020 at 18:58, pmp-p <report@bugs.python.org> wrote:

pmp-p <pmpp.pub@gmail.com> added the comment:

you can add
*
https://github.com/pmp-p/pydk/tree/master/sources.em/Python-3.8.0b4.patchset
-- Python 3.8.x

(wasm not asm.js, clang-10+ required)

demo https://pmp-p.github.io/python-next/test.html

CPython can already run in the browser with very little patching, but
major issues are :

• asyncify'ing the whole wasm VM to have pre-emption over cPython's one
  to prevent blocking I/O slows down things *a lot* (10x)
  => (very?) bad user experience.

• the size of vm + stdlib ~ 30 MiB and wasm compilation time.
  => bad user experience on first load or slow connexion.

• the lack of threading in wasm MinimumViableProduct specification (but
  this is the browser standard for now), that leads to rewrite bits of stdlib
  ( like eg asyncio module )
  => adding more maintenance burden on stdlib (!)

i tested them all and my personnal opinion is : I can see no use case that
would favour "stock" cPython wasm versus a blazing fast MicroPytho (or
pycopy) wasm flavour or supercharged full stack pyodide.

----------
nosy: +pmpp

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue40280\>

---

I've been maintaining a Python Emscripten build for the Ren'Py (game engine) web port:
https://github.com/python-emscripten/python
https://renpy.beuc.net/

I recently tackled Python3 with a minimal/embeddable approach and checking the other ports already pointed in the discussion:
https://github.com/python-emscripten/python/tree/trunk/3.8/
(2 patches, and a short pyconfig.h fix-up)

There is also a Cython module to use the Emscripten C API.

Here's a demo at:
https://www.beuc.net/python-emscripten/demo/3/
(hello-world size: 3MB, with a few common modules: 4MB)

I can provide a pull request with a first few core changes.
Should this be done on github?

cross-compilation handling appears to follow an incorrect logic, in particular by querying 'dpkg' or parsing compiler output to detect include paths -- it is the (cross-)compiler's responsibility to provide the system paths, and detecting them manually causes conflicts. I had to patch setup.py. Let me know if I missed something.

Usually cross-compiling is triggered by non-matching build-type/host-type. Here cross-compilation logic is apparently triggered when exporting _PYTHON_HOST_PLATFORM=xxx manually (it's in the Makefile but not exported, and breaks normal build when exported). Is this the way it's meant to be used?

I think the first thing we should do is figure out whether we want to support Emscripten or WASI (or both).

Emscripten uses Javascript polyfills for some syscalls, while WASI makes direct calls the VM it is running in. They both can use WebAssembly for executing the code. This means Emscripten has wider API support, but WASI is lighter weight in many ways.

I think starting with patches to support Emscripten would be best, as it is easier to target, then add support for WASI later.

I think supporting WASI has a lot of value, because it can be run deterministically, which would be great for data science (Imagine a jupyter notebook that runs the same everywhere!)

One issue with WASI, and may be an issue with Emscripten, is threads. In 3.8 (or 3.9?) threadless builds were removed. However, WebAssembly's threading API is not really meant to emulate pthread, and SharedArrayBuffer, the primitive it is built on, is disabled in several browsers due to Spectre concerns.

Would patches to re-add a threadless build mode be accepted?

I have added wasm32/wasm64 architectures with emscripten/wasi operating system as cross-build targets. The values are based on Rust targets:

$ rustc --print target-list | grep wasm
wasm32-unknown-emscripten
wasm32-unknown-unknown
wasm32-wasi
wasm64-unknown-unknown

wasm (WebAssembly) is "native instruction set" for the JavaScript VM while wasi or emscripten provide operating system facilities like memory management and I/O.

Our config.sub is recent enough and has support for wasm32, wasm64, wasi, and emscripten:

$ grep was[mi] config.sub 
                        | wasm32 | wasm64 \
             | midnightbsd* | amdhsa* | unleashed* | emscripten* | wasi* \

LLVM considers was32-wasi an alias for wasm32-unknown-wasi. Verified on the WebAssembly Discover server at https://discord.com/channels/453584038356058112/596492540388179976/898618010221310062.

My last message had a couple of typos; should have been wasm32-wasi and "Discord", not "Discovery".

Do we need to care about our config.guess being updated as well? This is a totally ignorant question based on https://github.com/WebAssembly/wasi-sdk#notes-for-autoconf mentioning config.guess.

To help keep links up-to-date, Pyodide now lives at:

https://github.com/pyodide/pyodide/tree/main/cpython

Our config.sub agrees with LLVM:

$ ./config.sub wasm32-wasi
wasm32-unknown-wasi

The config.sub and config.guess scripts in main are recent enough for wasm. Just to be sure I created #29781 and plan to backport the changeset to 3.10 and 3.9. It's generally safe to update the files to latest version.

New changeset 1052a39 by Christian Heimes in branch 'main':
bpo-40280: Add wasm cross build targets (GH-29771)
1052a39

I have uploaded my config.site override to https://gist.github.com/tiran/5ccffa28723d3e4739db848451bd9efa . It contains overrides based on pyodide and overrides for new features.

I'm also getting this error with emscripten 2.0.13. _sys_shutdown is the syscall for shutdown(2) used by the socket module.

error: undefined symbol: __sys_shutdown (referenced by top-level compiled C/C++ code)
warning: Link with -s LLD_REPORT_UNDEFINED to get more information on undefined symbols
warning: To disable errors for undefined symbols use -s ERROR_ON_UNDEFINED_SYMBOLS=0
warning: ___sys_shutdown may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
Error: Aborting compilation due to previous errors

emscripten-core/emscripten#13393

New changeset ee1e2c6 by Christian Heimes in branch 'main':
bpo-40280: Use Setup.stdlib static for wasm builds (GH-29784)
ee1e2c6

Thanks a lot for working on this!

_sys_shutdown is the syscall for shutdown(2) used by the socket module.

Yes, the issue with Emscripten is that a number of system calls are either not implemented or implemented but not tested. See a list we are using in https://github.com/pyodide/pyodide/blob/main/cpython/pyconfig.undefs.h (though things might have improved since it was created).

FYI, with Emscripten, the list of CPython unit tests that are currently skipped (as of Python 3.9.5) is in https://github.com/pyodide/pyodide/blob/main/src/tests/python_tests.txt some of those are due to browser VM limitations (e.g. virtual filestem by Emscripten that's not fully POSIX compliant, no processes, no sockets, async only via the browser event loop etc), others because we are not yet using threading since not all browsers support it, and some failures probably need more investigation.

Also opened pyodide/pyodide#2000 . Let us know if there is anything we can do help with this effort.

New changeset d224e76 by Christian Heimes in branch 'main':
bpo-40280: clean and ignore .wasm files (GH-29794)
d224e76

Thanks Roman, I replied on the pyodide issue tracker.

Seems that commit d1de107 has introduced a regression when building with --enable-optimizations. As we made the libinstall target depend on all install of build_all not running altinstall tries to build with PGO at that point.

when building with PGO, out of tree builds fail with:

$make -C ../ altinstall DESTDIR=$PWD -j
...
error: [Errno 2] No such file or directory: 'build/temp.macosx-12.4-arm64-3.11/multiarch'

I am marking this as release-blocker

@ethanhs Please, take a look as soon as possible

@ethanhs

Steps to reproduce.

# On the cpython root folder
$ mkdir somedir
$ cd somedir
$ ../configure --enable-optimizations
$ make -j 
$ make altinstall DESTDIR=$PWD -j 
...
error: [Errno 2] No such file or directory: 'build/temp.macosx-12.4-arm64-3.11/multiarch'
make[1]: *** [sharedmods] Error 1
make: *** [profile-opt] Error 2
make: *** Waiting for unfinished jobs....
...
$ echo $?
2

Doing the same on 3.10 suceeds.

@pablogsal Does the error also occur when you serialize make with -j1? Could you please upload Makefile, pybuilddir.txt and the full output?

Does the error also occur when you serialize make with -j1?

Yes Actually, I tried again and -j1 fixes the problem. It seems that there is some race

Could you please upload Makefile, pybuilddir.txt and the full output?

Will do when I am back at my computer, but you can reproduce with the steps that I pasted.

The requested files:

files.zip

Also, I noticed another potential bug: the flags in the PGO build sometimes don't include the PKGCONFIG stuff.

I don't see where [Errno 2] No such file or directory: 'build/temp.macosx-12.4-arm64-3.11/multiarch' is coming from. Is it coming from setup.py?

I don't see where [Errno 2] No such file or directory: 'build/temp.macosx-12.4-arm64-3.11/multiarch' is coming from. Is it coming from setup.py?

I think so, but with the parallelism is difficult to tell. Check the attached files for reference.

I think is coming from running build_ext

Hi, taking a look at this now, sorry to cause the issue!

@ethanhs are you in the core Dev discord chat? There is some context also there

@pablogsal I am not a core dev yet, so I presume I don't have access? But if it is public then I will take a look.

I also was not able to repro the failure with my first attempt, but it sounds like it may be something to do with parallelism, which is why I didn't run into at first.

I also was not able to repro the failure with my first attempt, but it sounds like it may be something to do with parallelism, which is why I didn't run into at first.

It requires parallel builds and lots of CPU cores. I'm unable to reproduce the issue on my 8 core laptop.

Ah OK, I have a 32 core machine I can try to repro on.

@pablogsal I am not a core dev yet, so I presume I don't have access? But if it is public then I will take a look.

You don't need to be a core Dev to access most channels, only a triagger :)

I also was not able to repro the failure with my first attempt, but it sounds like it may be something to do with parallelism, which is why I didn't run into at first.

It requires parallel builds and lots of CPU cores. I'm unable to reproduce the issue on my 8 core laptop.

You also may need to run it multiple times to see it fail. Seems that reproducing on macOS is easier (at least in my case).

I am not a triager either so I guess I am out of luck :/

I am trying to run it until it fails on my 32 core machine. I don't have any macOS machines unfortunately, but at least its quite fast to build on 32 cores 😅

gh-93584 has more context on the issue. tl;dr Make rules for PGO builds have a bug and run two ./python setup.py build instances in parallel.