Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No zlib in WASI #93819

Open
DaAwesomeP opened this issue Jun 14, 2022 · 7 comments
Open

No zlib in WASI #93819

DaAwesomeP opened this issue Jun 14, 2022 · 7 comments
Assignees
@brettcannon @tiran
Labels
expert-WebAssembly type-bug An unexpected behavior, bug, or error

Comments

@DaAwesomeP
Copy link

Bug report
When compiling the WASI version using the docker image, it does not appear to include zlib in the build, or maybe the run-python-wasi.sh is not configured to allow zlib to work: ModuleNotFoundError: No module named 'zlib'

If this is already a known issue, then I can open a pull to update the docs.

Your environment

  • CPython versions tested on: main 30610d2
  • Operating system and architecture: quay.io/tiran/cpythonbuild:emsdk3 x86_64
@DaAwesomeP DaAwesomeP added the type-bug An unexpected behavior, bug, or error label Jun 14, 2022
@DaAwesomeP
Copy link
Author

Reading more docs, this may be really because --enable-wasm-dynamic-linking is not supported on WASI?

@tiran
Copy link
Member

tiran commented Jun 14, 2022

WASI-SDK does not come with zlib out of the box. My container image does not contain any extra libraries for WASI yet. I'll look into external dependencies once we have buildbots working.

@robbertvanginkel robbertvanginkel mentioned this issue Dec 7, 2022
Open
@robbertvanginkel
Copy link

I ran into this as well when trying to use the precompiled stdlib from make wasm_stdlib in a wasi build. This would fail as the script that produces the stdlib compresses with zlib:

cpython/Tools/wasm/wasm_assets.py

Lines 221 to 224 in 1de4395

# bpo-17004: zipimport supports only zlib compression.
# Emscripten ZIP_STORED + -sLZ4=1 linker flags results in larger file.
args.compression = zipfile.ZIP_DEFLATED
args.compresslevel = 9

I managed to build my own wasi python with zlib with the following dockerfile:

# syntax=docker/dockerfile:1.4

# must be linux/amd64 as wasi-sdk only has amd64 binaries
FROM --platform=linux/amd64 debian:bullseye
RUN apt-get update \
 && apt-get install -y xz-utils git libxml2 build-essential pkg-config zlib1g-dev curl python3 \
 && rm -rf /var/lib/apt/lists/*
ADD https://github.com/python/cpython/archive/refs/tags/v3.11.1.tar.gz /src/python-v3.11.1.tar.gz
ADD https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-16/wasi-sdk-16.0-linux.tar.gz /src/
ADD https://www.zlib.net/zlib-1.2.13.tar.xz /src/
ADD https://github.com/bytecodealliance/wasmtime/releases/download/v4.0.0/wasmtime-v4.0.0-x86_64-linux.tar.xz /tmp/wasmtime.tar.xz
WORKDIR /build
RUN mkdir /opt/wasi-sdk && tar -xf /src/wasi-sdk-16.0-linux.tar.gz --strip-components=1 -C /opt/wasi-sdk/ \
    	&& tar -xf /tmp/wasmtime.tar.xz --strip-components=1 -C /usr/local/bin \
	&& tar -xf /src/python-v3.11.1.tar.gz \
	&& tar -xf /src/zlib-1.2.13.tar.xz
RUN <<EOF
set -e
export PATH="/opt/wasi-sdk/bin:$PATH"
cd zlib-1.2.13/
CC=/opt/wasi-sdk/bin/clang RANLIB=/opt/wasi-sdk/bin/ranlib ./configure --prefix=
make install \
	prefix=/opt/wasi-sdk/share/wasi-sysroot \
	libdir=/opt/wasi-sdk/share/wasi-sysroot/lib/wasm32-wasi \
	pkgconfigdir=/opt/wasi-sdk/share/wasi-sysroot/lib/pkgconfig
cd ../cpython-3.11.1
./Tools/wasm/wasm_build.py wasi
/opt/wasi-sdk/bin/llvm-strip /build/cpython-3.11.1/builddir/wasi/python.wasm
cd builddir/wasi
make wasm_stdlib
EOF

With that, I can run an interpreter with the compressed stdlib:

$ docker run -it <ref to build docker file above>
root@4b26fe2dbb1b:/build# cd cpython-3.11.1/builddir/wasi/
root@4b26fe2dbb1b:/build/cpython-3.11.1/builddir/wasi# wasmtime run --mapdir /usr/local/lib::usr/local/lib python.wasm
Python 3.11.1 (main, Jan 16 2023, 20:30:30) [Clang 14.0.4 (https://github.com/llvm/llvm-project 29f1039a7285a5c3a9c353d05414 on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>>

The build is a bit akward since it copies zlib into the wasi-sdk sysroot, which I did because the wasi-env script seems to override any other place it could be discovered from:

cpython/Tools/wasm/wasi-env

Lines 63 to 64 in 1de4395

PKG_CONFIG_PATH=""
PKG_CONFIG_LIBDIR="${WASI_SYSROOT}/lib/pkgconfig:${WASI_SYSROOT}/share/pkgconfig"

I'd be open to contributing something to get this working by default for the wasi build, but couldn't find where the buildbot containers are defined and wanted to get some feedback on approaches before starting to modify things in the Tools/Wasm dir.

@brettcannon
Copy link
Member

I believe the current buildbots are using https://github.com/tiran/ansible-wasm-buildbot to set them up.

@gpshead
Copy link
Member

gpshead commented Jan 24, 2023

FYI this is probably the only issue blocking me making zlib required per https://discuss.python.org/t/lets-make-zlib-required-rather-than-optional-to-build-cpython/23062 and #91246

@brettcannon brettcannon self-assigned this Jan 24, 2023
@kushaldas
Copy link
Member

I can verify that static linking (of zlib) works as mentioned by @robbertvanginkel in #93819 (comment).

@juntao juntao mentioned this issue Jan 31, 2023
Open
@brettcannon
Copy link
Member

https://github.com/singlestore-labs/python-wasi also builds zlib, so there might be some inspiration there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brettcannon brettcannon

@tiran tiran

Labels
expert-WebAssembly type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@brettcannon @gpshead @kushaldas @tiran @robbertvanginkel @DaAwesomeP