Compatibility with Emscriptenforge, Pyodide, Pyscript?

[westurner]

• Does cowasm intend to invent a new WASM packaging format?
• Will it be possible to use pyodide packages with cowasm?
• Will it be possible to use cowasm packages with pyodide (pyscript,)?
• Cocalc is composed of conda packages (from conda-forge?) now
• Conda-forge noarch packages can be installed with emscriptenforge without modification
• Emscriptenforge WASM packages are built with empack, hosted on Quetz, and installed with picomamba
• Will cowasm be able to use emscriptenforge packages?

.

• Better integration with conda/conda-forge for building packages pyodide/pyodide#795 (comment) :

  • EmscriptenForge

    • https://github.com/emscripten-forge/empack
    • https://github.com/emscripten-forge/recipes
      • https://beta.mamba.pm/channels/emscripten-forge?tab=packages
      • https://repo.mamba.pm/emscripten-forge
    • https://github.com/emscripten-forge/recipes/blob/main/.github/workflows/build_recipes.yaml
    • https://github.com/emscripten-forge/recipes/blob/main/recipes/recipes_emscripten/python/recipe.yaml
    • https://github.com/mamba-org/picomamba/
    • https://github.com/mamba-org/picomamba/blob/d15420d2a2cf1cabf19da42e4c7a1734a5c4c46b/python/module/picomamba/picomamba.py#L176-L189

  • EmscriptenForge, CondaForge, Pyodide package SECurity and PERFormance

    • [...] CORS headers, [...]

    • FWIU packages are persisted w/ SQLite in WASM, per-request?

    • https://www.anaconda.com/blog/conda-signature-verification (2021)
      https://github.com/conda/conda-content-trust

    • Signed HTTP Exchanges (SXG) & Web Bundles weren't yet finalized (could be used to secure WASM packages); from https://web.dev/web-bundles/ :

      HTTP resources in a Web Bundle are indexed by request URLs, and can optionally come with signatures that vouch for the resources. Signatures allow browsers to understand and verify where each resource came from, and treats each as coming from its true origin. This is similar to how Signed HTTP Exchanges, a feature for signing a single HTTP resource, are handled.

    • HTTP Subresource Integrity (SRI):

          <script src="https://cdn.example.com/app.js"
           integrity="sha384-+/M6kredJcxdsqkczBUjMLvqyHb1K/JThDXWsBVxMEeZHEaMKEOEct339VItX1zB"
           crossorigin="anonymous"></script>

[williamstein]

Great questions. Thanks for taking an interest in CoWasm!

Does cowasm intend to invent a new WASM packaging format?

In cases that don't necessarily have much to do with Python, I'm probably just using npm. It's a relatively mature package format at this point. Example: https://www.npmjs.com/package/dash-wasm

For Python modules, I tried everything I could think of, and I'm currently happiest with "a .tar.xz file that when extracted can be imported by Python". It's a pretty simple format. I tried a dozen or so popular Python packages (e.g., pandas, sympy, numpy, etc.) and compressed it's usually about 5% the size of what pip install produces (uncompressed), so more suitable for the web. It's also easy to extract and decompress just using code in the Python standard library.

Will it be possible to use pyodide packages with cowasm? Will it be possible to use cowasm packages with pyodide (pyscript,)?

CoWasm currently uses the same dynamic linking WASM format as PyOdide/Esmscripten (i.e., clang's --target wasm32-emscripten with -fPIC), so there might be hope. One thing is that I've only tested pyodide so libraries that I found publicly, which are built for Python 3.10, whereas CoWasm uses Python 3.11, so I really don't know yet if they are compatible.

Cocalc is composed of conda packages (from conda-forge?) now

CoCalc doesn't have anything to do with CoWasm yet. The public website CoCalc.com does have a large number of conda packages preinstalled, which people find very useful. See https://cocalc.com/software/python for a list of the ones from Anaconda.

CoWasm itself doesn't technically currently have any relationship with CoCalc or Conda. CoWasm is a distinct BSD-licensed attempt to make something similar in spirit to PyOdide, but starting with Zig/Clang/Node.js, and trying to use lots of existing popular open source libraries like https://www.npmjs.com/package/memfs, which obviously emscripten couldn't use since it was such a pioneering project. (That said, I've had to fix a number of bugs in memfs, etc.)

One thing is that CoWasm directly uses pip to install pure python packages now. I spent the last two weeks implementing enough of the TCP networking stack in WASM (via posix-node). This is kind of like https://github.com/emscripten-core/emscripten/tree/main/tools/websocket_to_posix_proxy, except implemented using node.js instead. I haven't quite got extension modules to build via pip yet, but that's on my roadmap.

I guess my plan is more to install via pip, then post-process the results in some settings to be more suitable for distribution, since what pip produces or uses is not optimized for size.

Will cowasm be able to use emscriptenforge packages?

I don't know. Thanks for the additional information about these projects.