Support importing host functions

[cohix]

Hello... EXCELLENT project you've got here. I'm the founder of Suborbital, and we're currently investigating different ways to support Python in our E2 Core project and our SE2 platform. This project seems like it would be a very good method.

In order to use this, I'd love to know if it's currently possible to import functions from the Wasm host? We use Wasmtime, and have a set of host functions that we make available to the runtime (such as making HTTP requests, etc), and in other languages like Rust, you can import these host-provided functions to call them. (example from our SDK)

If this is possible, I'd love to learn how, and we'd love to build a prototype of Python support using zython.

If it's not possible, we'd love to contribute to the project and help out any way we can (code, testing, sponsorship, anything!)

Please let me know :)

[williamstein]

Cool! I'm glad you're interested!

I haven't yet implemented a way to import functions into Python from the Wasm host yet, though I'm certainly planning to do so.

Note that I'll be renaming this overall project to CoWasm in the next few days, and part of that will be a major revamp and improvement in how everything works, which makes much more extensive use of dynamic linking. The new architecture (in the dev branch now) involves a "kernel" (less than 0.5MB compressed) that provides the c library. Then there are many executables that are all dynamically linked and can be run by this kernel, and they can in turn run subprocesses. It's a powerful and very efficient architecture, since the executables are extremely small. One executables is CPython itself, which can then load dynamically linked extension modules (e.g., numpy), which is all working now. I'm building this so in addition to Python you can also run other programs in the same way, e.g., a traditional UNIX shell, coreutils, maybe a latex engine (?), etc., all in one place with no redundant code.

I'm building this all as a free BSD licensed extremely open source business-friendly project, since it will be useful for making versions of https://cocalc.com that run in a wide range of places (e.g., phones, tablets, laptops) in a uniform and easy to install way, without involving Docker or anything heavyweight like that. Incidentally, I'm the founder/CEO of CoCalc.com (which is very much a commercial product). I'm also being careful in the whole stack to avoid GPL licensed components to keep things commercial friendly.

I think this should also be useful for many other projects like yours, and will hopefully be a lot more lightweight, transparent, and modern than emscripten / pyodide.

I don't know how much work it would be for what I'm building to integrate with E2 Core. A key component of my project is https://www.npmjs.com/package/dylink, which is written in Javascript, and I imagine you guys would have to implement it again from scratch in Go. If you do, make sure to start with what's in my dev branch (or wait until I merge it into master), since I've made some improvements. In any case, it'll be MUCH easier to rewrite dylink based on what I've written, than it was for me to figure out how to load dynamically libraries by reading the spec and trying to untable the emscripten code.

Anyway, we could setup a video chat sometime if you want to compare notes that way and see if there is any opportunity for our companies to collaborate.

[cohix]

That sounds really friggin' cool 😄 I'd love to chat. Could you email connor at suborbital.dev so we can find time?

[williamstein]

I created an associated issue here: https://github.com/sagemathinc/zython/issues/35