Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for WebAssembly System Interface (wasm32-wasi) #90473

Open
tiran opened this issue Jan 9, 2022 · 4 comments
Open

Add support for WebAssembly System Interface (wasm32-wasi) #90473

tiran opened this issue Jan 9, 2022 · 4 comments
Assignees
@tiran
Labels
3.11 build type-feature

Comments

@tiran
Copy link
Member

@tiran tiran commented Jan 9, 2022

BPO 46315
Nosy @tiran, @pmp-p, @kumaraditya303
PRs
  • #30507
  • #32033
  • #32266
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2022-01-09.14:47:43.508>
labels = ['type-feature', 'build', '3.11']
title = 'Add support for WebAssembly System Interface (wasm32-wasi)'
updated_at = <Date 2022-04-02.21:11:46.918>
user = 'https://github.com/tiran'

    bugs.python.org fields:

    activity = <Date 2022-04-02.21:11:46.918>
actor = 'christian.heimes'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2022-01-09.14:47:43.508>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 46315
keywords = ['patch']
message_count = 4.0
messages = ['410153', '410198', '410470', '416588']
nosy_count = 3.0
nosy_names = ['christian.heimes', 'pmpp', 'kumaraditya']
pr_nums = ['30507', '32033', '32266']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue46315'
versions = ['Python 3.11']
    @tiran
    Copy link
    Member Author

    @tiran tiran commented Jan 9, 2022

    WASI is another WebAssembly platform similar to Emscripten (bpo-40280). Simply speaking Emscripten binaries (wasm32-emscripten) run inside a browser while WASI binaries target standalone runtimes like wasmtime [2][3] on the host. The lines are a bit blurry, as it is possible to run WASI binaries in the browser with help of JS-polyfills. WASI provides compile once, run anyway with JIT/AOT and sandboxing.

    WASI is still under development and is lacking several core features:

    • threading support and pthread API
    • sockets
    • signals are emulated (_WASI_EMULATED_SIGNAL and -lwasi-emulated-signal)
    • DAC APIs like chmod, umask
    • user-related APIs like pwd, grp
    • dup(), dup2(), F_DUPFD

    For 3.11 I plan to fix our use of #ifdef HAVE_FEATURE to make it easier to experiment with WASI. The pthread APIs need stubs, which I won't commit to 3.11 upstream yet.

    [1] https://wasi.dev/
    [2] https://github.com/bytecodealliance/wasmtime
    [3] https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/

    @tiran tiran added the 3.11 label Jan 9, 2022
    @tiran tiran self-assigned this Jan 9, 2022
    @tiran tiran added build type-feature 3.11 labels Jan 9, 2022
    @tiran tiran self-assigned this Jan 9, 2022
    @tiran tiran added build type-feature labels Jan 9, 2022
    @tiran
    Copy link
    Member Author

    @tiran tiran commented Jan 10, 2022

    dup() is required by _PyTokenizer_FindEncodingFilename(). I came up with this hack:

    // from wasi-libc libc-top-half/musl/src/internal/stdio_impl.h
struct _IO_FILE {
    unsigned flags;
    unsigned char *rpos, *rend;
    int (*close)(FILE *);
    unsigned char *wend, *wpos;
    // incomplete
};

static int
dummy_close(FILE *fp) {
    return 0;
};

static FILE *
_Py_fdopen_borrow(int fd, const char *mode) {
    FILE *fp = fdopen(fd, mode);
    ((struct _IO_FILE*)fp)->close = dummy_close;
    return fp;
}

    keithw on #wasi pointed out that fopencookie() can archive the same outcome without resorting to ABI-specific hack. A trivial implementation is straight forward:

    typedef union {
    void *cookie;
    int fd;
} borrowed;

    static ssize_t
    borrow_read(void *cookie, char *buf, size_t size)
    {
    borrowed b;
    b.cookie = cookie;
    return read(b.fd, (void *)buf, size);
    }

    static ssize_t
borrow_write(void *cookie, const char *buf, size_t size)
{
    errno = ENOTSUP;
    return -1;
}

static int
borrow_seek(void *cookie, off_t *off, int whence)
{
    borrowed b;
    b.cookie = cookie;
    off_t pos;
    pos = lseek(b.fd, *off, whence);
    if (pos == (off_t)-1) {
        return -1;
    } else {
        *off = pos;
        return 0;    
    }
}

static int
borrow_close(void *cookie)
{
    // does not close(fd)
    return 0;
}

FILE *
_Py_fdopen_borrow(int fd, const char *mode) {
    // only support read for now
    if (strcmp(mode, "r") != 0) {
        return NULL;
    }
    cookie_io_functions_t cookie_io = {
        borrow_read, borrow_write, borrow_seek, borrow_close
    };
  	// cookie is just the fd
    borrowed b;
    b.fd = fd;
    return fopencookie(b.cookie, "r", cookie_io);
}

    @tiran
    Copy link
    Member Author

    @tiran tiran commented Jan 13, 2022

    New changeset a6ca8ee by Christian Heimes in branch 'main':
    bpo-46315: Add ifdef HAVE_ feature checks for WASI compatibility (GH-30507)
    a6ca8ee

    @tiran
    Copy link
    Member Author

    @tiran tiran commented Apr 2, 2022

    New changeset 3df0e63 by Christian Heimes in branch 'main':
    bpo-46315: Use fopencookie only on Emscripten 3.x and newer (GH-32266)
    3df0e63

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    tiran added a commit to tiran/cpython that referenced this issue May 12, 2022
    @tiran
    pythongh-90473: Increase stack size, disable obmalloc on WASI
    ca57f6e
    tiran added a commit to tiran/cpython that referenced this issue May 12, 2022
    @tiran
    pythongh-90473: Increase stack size, disable obmalloc on WASI
    85b5db7
    tiran added a commit that referenced this issue May 13, 2022
    @tiran
    gh-90473: Increase stack size, disable obmalloc on WASI (GH-92732)
    d81d57e
    miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 13, 2022
    @tiran @miss-islington
    pythongh-90473: Increase stack size, disable obmalloc on WASI (python…
    1aedaa3 
    …GH-92732)

(cherry picked from commit d81d57e)

Co-authored-by: Christian Heimes <christian@python.org>
    miss-islington added a commit that referenced this issue May 13, 2022
    @miss-islington @tiran
    gh-90473: Increase stack size, disable obmalloc on WASI (GH-92732)
    c41667e 
    (cherry picked from commit d81d57e)

Co-authored-by: Christian Heimes <christian@python.org>
    tiran added a commit to tiran/cpython that referenced this issue May 14, 2022
    @tiran
    pythongh-90473: Fail subprocess early on Emscripten/WASI
    b62e78f
    tiran added a commit to tiran/cpython that referenced this issue May 14, 2022
    @tiran
    pythongh-90473: Decrease recursion limit and skip tests on WASI
    6fd22ec 
    Reduce recursion limit to 750. WASI has limited call stack.

Mark tests that require mmap, os.pipe, or fail on musl libc.
    tiran added a commit to tiran/cpython that referenced this issue May 14, 2022
    @tiran
    pythongh-90473: Decrease recursion limit and skip tests on WASI
    881f75a 
    Reduce recursion limit to 750. WASI has limited call stack.

Mark tests that require mmap, os.pipe, or fail on musl libc.
    tiran added a commit that referenced this issue May 14, 2022
    @tiran
    gh-90473: Fail subprocess early on Emscripten/WASI (GH-92802)
    db0b455
    miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 14, 2022
    @tiran @miss-islington
    pythongh-90473: Fail subprocess early on Emscripten/WASI (pythonGH-92802
    f08baed 
    )

(cherry picked from commit db0b455)

Co-authored-by: Christian Heimes <christian@python.org>
    tiran added a commit to tiran/cpython that referenced this issue May 14, 2022
    @tiran
    pythongh-90473: Decrease recursion limit and skip tests on WASI
    e7e42b6 
    Reduce recursion limit to 750. WASI has limited call stack.

Mark tests that require mmap, os.pipe, or fail on musl libc.
    miss-islington added a commit that referenced this issue May 14, 2022
    @miss-islington @tiran
    gh-90473: Fail subprocess early on Emscripten/WASI (GH-92802)
    2bcae11 
    (cherry picked from commit db0b455)

Co-authored-by: Christian Heimes <christian@python.org>
    @stonebig stonebig mentioned this issue May 15, 2022
    Open
    tiran added a commit to tiran/cpython that referenced this issue May 15, 2022
    @tiran
    pythongh-90473: Decrease recursion limit and skip tests on WASI
    0d3f889 
    Reduce recursion limit to 750. WASI has limited call stack.

Mark tests that require mmap, os.pipe, or fail on musl libc.
    tiran added a commit to tiran/cpython that referenced this issue May 15, 2022
    @tiran
    pythongh-90473: Decrease recursion limit and skip tests on WASI
    24ff31d 
    Reduce recursion limit to 750. WASI has limited call stack.

Mark tests that require mmap, os.pipe, or fail on musl libc.
    tiran added a commit to tiran/cpython that referenced this issue May 16, 2022
    @tiran
    pythongh-90473: WASI: Mark tests that require os.pipe()
    f661987
    tiran added a commit that referenced this issue May 16, 2022
    @tiran
    gh-90473: WASI: Mark tests that require os.pipe() (GH-92837)
    730902c
    tiran added a commit to tiran/cpython that referenced this issue Jun 6, 2022
    @tiran
    pythongh-90473: WASI requires proper open(2) flags
    1c40a4b
    tiran added a commit that referenced this issue Jun 6, 2022
    @tiran
    gh-90473: WASI requires proper open(2) flags (GH-93529)
    4c71d22
    miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 6, 2022
    @tiran @miss-islington
    pythongh-90473: WASI requires proper open(2) flags (pythonGH-93529)
    f87ab92 
    (cherry picked from commit 4c71d22)

Co-authored-by: Christian Heimes <christian@python.org>
    miss-islington added a commit that referenced this issue Jun 6, 2022
    @miss-islington @tiran
    gh-90473: WASI requires proper open(2) flags (GH-93529)
    20be4a1 
    (cherry picked from commit 4c71d22)

Co-authored-by: Christian Heimes <christian@python.org>
    tiran added a commit to tiran/cpython that referenced this issue Jun 6, 2022
    @tiran
    pythongh-90473: Make chmod a dummy on WASI, skip chmod tests
    eae6042 
    WASI does not have ``chmod(2)`` syscall yet.
    tiran added a commit to tiran/cpython that referenced this issue Jun 6, 2022
    @tiran
    pythongh-90473: Skip POSIX tests that don't apply to WASI
    b5a6725
    tiran added a commit that referenced this issue Jun 6, 2022
    @tiran
    gh-90473: Skip POSIX tests that don't apply to WASI (GH-93536)
    80a39da
    tiran added a commit to tiran/cpython that referenced this issue Jun 6, 2022
    @tiran
    [3.11] pythongh-90473: Skip POSIX tests that don't apply to WASI (pyt…
    26d7646 
    …honGH-93536)

(cherry picked from commit 80a39da)

Co-authored-by: Christian Heimes <christian@python.org>
    ambv pushed a commit that referenced this issue Jun 6, 2022
    @tiran
    gh-90473: Make chmod a dummy on WASI, skip chmod tests (GH-93534)
    22fed60 
    WASI does not have the ``chmod(2)`` syscall yet.
    miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 6, 2022
    @tiran @miss-islington
    pythongh-90473: Make chmod a dummy on WASI, skip chmod tests (pythonG…
    ca7b0d0 
    …H-93534)

WASI does not have the ``chmod(2)`` syscall yet.
(cherry picked from commit 22fed60)

Co-authored-by: Christian Heimes <christian@python.org>
    ambv pushed a commit that referenced this issue Jun 6, 2022
    @tiran
    [3.11] gh-90473: Skip POSIX tests that don't apply to WASI (GH-93536) (…
    986ce4e 
    …GH-93540)

(cherry picked from commit 80a39da)

Co-authored-by: Christian Heimes <christian@python.org>
    ambv pushed a commit that referenced this issue Jun 6, 2022
    @miss-islington @tiran
    gh-90473: Make chmod a dummy on WASI, skip chmod tests (GH-93534) (GH…
    6787a8f 
    …-93550)

WASI does not have the ``chmod(2)`` syscall yet.
(cherry picked from commit 22fed60)

Co-authored-by: Christian Heimes <christian@python.org>
    tiran added a commit to tiran/cpython that referenced this issue Jun 8, 2022
    @tiran
    pythongh-90473: Define HOSTRUNNER for WASI
    8213b7a
    tiran added a commit that referenced this issue Jun 8, 2022
    @tiran
    gh-90473: Define HOSTRUNNER for WASI (GH-93606)
    22df2e0
    tiran added a commit to tiran/cpython that referenced this issue Jun 8, 2022
    @tiran
    [3.11] pythongh-90473: Define HOSTRUNNER for WASI (pythonGH-93606)
    87235d5 
    (cherry picked from commit 22df2e0)

Co-authored-by: Christian Heimes <christian@python.org>
    tiran added a commit that referenced this issue Jun 8, 2022
    @tiran
    [3.11] gh-90473: Define HOSTRUNNER for WASI (GH-93606) (GH-93612)
    1b7942a 
    (cherry picked from commit 22df2e0)

Co-authored-by: Christian Heimes <christian@python.org>
    tiran added a commit to tiran/cpython that referenced this issue Jun 9, 2022
    @tiran
    pythongh-90473: disable user site packages on WASI/Emscripten
    62aee79
    tiran added a commit to tiran/cpython that referenced this issue Jun 9, 2022
    @tiran
    pythongh-90473: Skip get_config_h() tests on WASI
    2c21590 
    The test cases are incompatible with WASI mapdir and OOT builds.
wasmtime starts the process with ``/`` as CWD, but the files are in a
subdirectory.
    tiran added a commit that referenced this issue Jun 9, 2022
    @tiran
    gh-90473: disable user site packages on WASI/Emscripten (GH-93633)
    5a4af3a
    miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 9, 2022
    @tiran @miss-islington
    pythongh-90473: disable user site packages on WASI/Emscripten (python…
    f81c8e8 
    …GH-93633)

(cherry picked from commit 5a4af3a)

Co-authored-by: Christian Heimes <christian@python.org>
    tiran added a commit that referenced this issue Jun 9, 2022
    @tiran
    gh-90473: Skip get_config_h() tests on WASI (GH-93645)
    6099611
    miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 9, 2022
    @tiran @miss-islington
    pythongh-90473: Skip get_config_h() tests on WASI (pythonGH-93645)
    46b13ef 
    (cherry picked from commit 6099611)

Co-authored-by: Christian Heimes <christian@python.org>
    miss-islington added a commit that referenced this issue Jun 9, 2022
    @miss-islington @tiran
    gh-90473: disable user site packages on WASI/Emscripten (GH-93633)
    4c41f21 
    (cherry picked from commit 5a4af3a)

Co-authored-by: Christian Heimes <christian@python.org>
    miss-islington added a commit that referenced this issue Jun 9, 2022
    @miss-islington @tiran
    gh-90473: Skip get_config_h() tests on WASI (GH-93645)
    06340f6 
    (cherry picked from commit 6099611)

Co-authored-by: Christian Heimes <christian@python.org>
    tiran added a commit to tiran/cpython that referenced this issue Jun 11, 2022
    @tiran
    pythongh-90473: Skip test_queue when threading is not available
    0c3793f 
    Almost all tests were already skipped anyways. The remaining tests are
not useful without threading primitives like locks and conditions.
    miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 11, 2022
    @tiran @miss-islington
    pythongh-90473: Skip test_queue when threading is not available (pyth…
    43239ff 
    …onGH-93712)

(cherry picked from commit f0b7aa7)

Co-authored-by: Christian Heimes <christian@python.org>
    tiran added a commit that referenced this issue Jun 11, 2022
    @tiran
    gh-90473: Skip test_queue when threading is not available (GH-93712)
    f0b7aa7
    miss-islington added a commit that referenced this issue Jun 11, 2022
    @miss-islington @tiran
    gh-90473: Skip test_queue when threading is not available (GH-93712)
    34ef12a 
    (cherry picked from commit f0b7aa7)

Co-authored-by: Christian Heimes <christian@python.org>
    tiran added a commit to tiran/cpython that referenced this issue Jun 14, 2022
    @tiran
    pythongh-90473: Include stdlib dir in wasmtime PYTHONPATH
    cbdadbc
    tiran added a commit that referenced this issue Jun 14, 2022
    @tiran
    gh-90473: Include stdlib dir in wasmtime PYTHONPATH (GH-93797)
    cd543d0
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @tiran tiran

    Labels
    3.11 build type-feature
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @tiran