npm packaging for node and web

[elijahmorg]

Add web support to npm package.

Still a WIP need to do some cleanup still

I assumed it is better to keep the server code and web code together in the same package (bigger download number). It took quite a bit of experimentation but the ultimate experience is

node - commonjs

const { Database } = require("limbo-wasm/node");

web - module

const worker = new Worker(new URL('limbo-wasm/web/limbo-worker.js', import.meta.url), { type: 'module' });

Like I said this took a lot of experimentation on my part as this is my first time trying to create an npm package let alone that mixes commonjs and modules.

The structure is an npm workspace with two sub packages (web and node).

node
├── dist
│   ├── README.md
│   ├── index.d.ts
│   ├── index.js
│   ├── index_bg.wasm
│   ├── index_bg.wasm.d.ts
│   └── snippets
│       └── limbo-wasm-d1562e55b90f5289
│           └── node
│               └── src
│                   └── vfs.js
├── package.json
└── src
    └── vfs.js
web
├── dist
│   ├── README.md
│   ├── index.d.ts
│   ├── index.js
│   ├── index_bg.wasm
│   ├── index_bg.wasm.d.ts
│   └── snippets
│       └── limbo-wasm-d1562e55b90f5289
│           └── web
│               └── src
│                   └── web-vfs.js
├── html
│   ├── index.html
│   ├── limbo-opfs-test.html
│   └── limbo-test.html
├── node_modules
├── package.json
├── playwright.config.js
├── src
│   ├── limbo-worker.js
│   ├── opfs-interface.js
│   ├── opfs-sync-proxy.js
│   ├── opfs-worker.js
│   ├── opfs.js
│   └── web-vfs.js
├── test
│   ├── helpers.js
│   ├── limbo.test.js
│   ├── opfs.test.js
│   └── setup.js
└── vite.config.js

The output of wasm-pack gets put in <web/node>dist
JS code moves into <web/node>src/

Tests move under web/test

The npm package looks like (you can see I need to cleanup some of the stuff that gets included).

-rw-r--r--  0 0      0         195 Oct 26  1985 package/web/html/index.html
-rw-r--r--  0 0      0        2733 Oct 26  1985 package/web/html/limbo-opfs-test.html
-rw-r--r--  0 0      0         162 Oct 26  1985 package/web/html/limbo-test.html
-rw-r--r--  0 0      0         632 Oct 26  1985 package/web/test/helpers.js
-rw-r--r--  0 0      0       18128 Oct 26  1985 package/node/dist/index.js
-rw-r--r--  0 0      0       21732 Oct 26  1985 package/web/dist/index.js
-rw-r--r--  0 0      0        1836 Oct 26  1985 package/web/src/limbo-worker.js
-rw-r--r--  0 0      0        2108 Oct 26  1985 package/web/test/limbo.test.js
-rw-r--r--  0 0      0        1764 Oct 26  1985 package/web/src/opfs-interface.js
-rw-r--r--  0 0      0        3409 Oct 26  1985 package/web/src/opfs-sync-proxy.js
-rw-r--r--  0 0      0        1430 Oct 26  1985 package/web/src/opfs-worker.js
-rw-r--r--  0 0      0        3976 Oct 26  1985 package/web/src/opfs.js
-rw-r--r--  0 0      0        4502 Oct 26  1985 package/web/test/opfs.test.js
-rw-r--r--  0 0      0         269 Oct 26  1985 package/web/playwright.config.js
-rw-r--r--  0 0      0           0 Oct 26  1985 package/web/test/setup.js
-rw-r--r--  0 0      0         519 Oct 26  1985 package/node/dist/snippets/limbo-wasm-d1562e55b90f5289/node/src/vfs.js
-rw-r--r--  0 0      0         519 Oct 26  1985 package/node/src/vfs.js
-rw-r--r--  0 0      0         608 Oct 26  1985 package/web/vite.config.js
-rw-r--r--  0 0      0         435 Oct 26  1985 package/web/dist/snippets/limbo-wasm-d1562e55b90f5289/web/src/web-vfs.js
-rw-r--r--  0 0      0         435 Oct 26  1985 package/web/src/web-vfs.js
-rw-r--r--  0 0      0         146 Oct 26  1985 package/web/node_modules/.vite/deps/_metadata.json
-rw-r--r--  0 0      0         309 Oct 26  1985 package/node/package.json
-rw-r--r--  0 0      0         671 Oct 26  1985 package/package.json
-rw-r--r--  0 0      0          23 Oct 26  1985 package/web/node_modules/.vite/deps/package.json
-rw-r--r--  0 0      0         602 Oct 26  1985 package/web/package.json
-rw-r--r--  0 0      0         153 Oct 26  1985 package/web/node_modules/.vite/vitest/results.json
-rw-r--r--  0 0      0        1296 Oct 26  1985 package/node/dist/README.md
-rw-r--r--  0 0      0        1296 Oct 26  1985 package/README.md
-rw-r--r--  0 0      0        1296 Oct 26  1985 package/web/dist/README.md
-rw-r--r--  0 0      0        1217 Oct 26  1985 package/node/dist/index_bg.wasm.d.ts
-rw-r--r--  0 0      0        1217 Oct 26  1985 package/web/dist/index_bg.wasm.d.ts
-rw-r--r--  0 0      0         449 Oct 26  1985 package/node/dist/index.d.ts
-rw-r--r--  0 0      0        2554 Oct 26  1985 package/web/dist/index.d.ts
-rw-r--r--  0 0      0     2215065 Oct 26  1985 package/node/dist/index_bg.wasm
-rw-r--r--  0 0      0     2213889 Oct 26  1985 package/web/dist/index_bg.wasm

resolves #624

[elijahmorg]

I need to add these back - but did the limbo version of these tests even work? I could not get them to run even before I made changes. Easily could've been a skill issue.

[elijahmorg]

This is here to test the npm package as its packaged.

[jussisaurio]

Thanks a lot! I have a few questions/issues:

---

Single vs multiple package

Doesn't this in web/package.json:

"name": "@limbo-wasm/web",

now mean that web becomes a different npm package (a package called web under the @limbo-wasm namespace), rather than the same as you said in the PR description? You can try this out by running

cd web
npm link

cd test-limbo-pkg
npm link @limbo-wasm/web

then adding in test-limbo-pkg/package.json:

  "dependencies": {
    "@limbo-wasm/web": "0.0.11"
  },

and running:

npm run dev

---

Issues with the package itself

Also you'll notice that this doesn't quite work since the web package json has this configuration:

  "main": "./dist/index.mjs",
  "types": "./dist/index.d.ts",
  "files": ["dist"],
  "exports": {
    ".": {
      "import": "./dist/index.mjs",
      "types": "./dist/index.d.ts"
    }
  },

1. During the build process, copies of all the OPFS related files, limbo-worker.js etc should end up in dist, otherwise they won't be found by imports

# scripts/build
# Copy web interface files
cp web/src/*.js $WEB_DIR/dist/

There's also vite-plugin-wasm-pack that could obviate the need for a separate script and you could just build the entire web package using vite possibly.

EDIT: I quickly tested this and it should work, but it requires some changes to the vite config. e.g.

  build: {
    lib: {
      entry: "./src/limbo-worker.js",
      formats: ["es"],
    },
  },

We should aim so that it creates a single js bundle for the main thread and a single bundle for the worker thread, I think. (or two bundles if the worker requires another worker, but generally the module should import each other hierarchically)

2. index.mjs doesn't exist, index.js does, so probably need to change that. But is that our main entrypoint, instead of limbo-worker.js?

By making limbo-worker.js the entrypoint, test-limbo-pkg can now import the package like this:

const worker = new Worker(new URL('@limbo-wasm/web', import.meta.url), { type: 'module' });

assuming that is going to be the main entrypoint of our program, and we don't want to have the main entrypoint be a higher level interface that doesn't require postMessage and so on.

---

Other notes

We should probably consider such a higher level interface, I think. We can start with this, but eventually we'd probably like to have something like:

import { Database } from '@limbo-wasm/web';
const worker = new Worker(new URL('@limbo-wasm/web/limbo-worker.js', import.meta.url), { type: 'module' });

const limbo = new Database({ worker, db: 'mydb.db' }); // or whatever

await limbo.ready(); // or whatever

await limbo.exec('CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT, email TEXT'); // or whatever

not meant as an exact api design, but just abstract the webworker communication out.

A couple of other side notes for the future:

• if we do go with multiple npm packages, we could just call it @limbo/web :)
• we should probably have the main public interface of the package be written in typescript.

[elijahmorg]

Ok! I'll take a look at at addressing those comments. FYI it will probably be Tuesday before I get back around to this.