Parametric packaging? (e.g., replicating @iconify-json/<icon-set> from Iconify's NPM packages)

[jmuchovej]

Description

Not quite an issue, more-so a feature request. I'm in the middle of porting https://github.com/iconify/icon-sets to Typst so we can have our pick of SVG icons (without needing them as fonts 🎉)!

Iconify has a really slick layout (mostly for tree-shaking, if I understand correctly) where users can install an icon-set's JSON to support it (rather than having to install all 150+ icon sets, which turns in to something like 200K+ icons).

I'm not sure that Typst has/needs tree-shaking the way that web-oriented projects do, but the full package I've built (https://github.com/jmuchovej/typst-iconify) is north of 1GB because it has almost all the icons included.

There's two packaging strategies I can think of for the typst-iconify project:

1. typst-iconify just packages up the entire Iconify library (200K+ icons, relevant JSON/SVGs is ~950MB as of 15 Oct 2024).
2. typst-iconify has some kind setup like... @preview/iconify (core library) and users can add @preview/iconify-{iconset} where {iconset} might be academicons, carbon, catppuccin, fa (font awesome), etc.

(1) is by far the most straightforward, but it introduces some critical problems:

1. Installation requires downloading ~950MB (or more, in the future) worth of SVGS, most of which will go unused.
2. Once typst-iconify is stable, I suspect that the icon sets will update far more frequently than the core package code – having some way to separate these concerns would be ideal – however I'm not sure how. (It appears that @iconify-json/{iconset} essentially installs the corresponding JSON file in a shared directory that other @iconify/... tools know how to access. (If Typst supports similarly, I'd be more than happy to explore that, I just don't know where/if that's possible.)

In the current packaging approach, where everything is namedspaced under @preview/... I don't think this avoids the patch-version explosion (though it could still cause significant version explosion because the icon sets appear to be upgraded pretty regularly).

So, this leads me to my questions:

1. Is there a way to support something like the following – where line 2 essentially adds the academicons JSON files that @preview/iconify knows how to parse to some shared location?

   1 | #import "@preview/iconify:0.1.0": icon
   2 | #import "@preview/iconify-academicons:1.2.8"
   3 | 
   4 | #icon("academicon:arxiv")

2. If ☝️ is possible, how would I achieve this? Particularly, this introduces a second set of problems:
   1. Can Typst, functionally, write temporary files (or similar)? (i.e., If @preview/iconify dynamically loads SVG data, the packaged code in @preview/iconify-academicons will be an SVG string, but image requires a file-path.
   2. Is it possible to essentially write to a cache in this manner, or would it be "more typst-ian" to recompute all this data upon initial compilation? (i.e., if someone closes a doc, then reopens it later.)
3. How concerned are Typst with "patch bloat" from packages? (e.g., the split-package route I've described/favored would introduce over 150 packages, each with their own versioning/etc. I'd have them configured to auto-update, but I don't want these to end up back-logging other package updates, since they're definitely low priority. 😅)

[elegaanz]

Hello! To answer your questions:

1. It is not possible for packages to share files by putting them in a well known location. It is maybe possible to do something similar to what you want with state (but after some quick testing I'm not even sure this would work). I think the proper way of doing what you want to do here would be to have the iconify package with its icon function that is used by each icon set package, but never directly by the user (if the icon function does something more than resolving icon names). In the icon set package (iconify-academicons in your example) you could then read the corresponding JSON file into a dictionary, where each key is the icon name, and the corresponding value is the content to display it. This way, people would be able to do something like that:

#import "@preview/iconify-academicons:1.2.8": academicons

#academicons.arxiv

i. You don't have to write to temporary files (which Typst can't do anyways), you can pass a SVG string to image.decode directly.
ii. As I just said, it is not possible to write to a cache, so data would have to be recomputed during the initial compilation.
3. I am actually concerned that this would spam the repository quite a lot, especially if package updates are automated. Having automated submissions for 150 packages, even if they are straightforward to review, would mean a lot more work for us and would mean other packages get less attention. There is also the fact that all these icons amount to almost 1GB and that our current infrastructure doesn't de-duplicate files between versions, which would mean that the repository would probably grow quite fast. What I would recommend for now is to only create packages for icon sets that you actually need, and manually update them every once in a while. We may have a better package infrastructure some day that will allow you to have all these packages published in an automated fashion, but for now I don't think it is really possible.

Also, we have some naming guidelines, so unless you are the owner of iconify, the package names you used in your example won't be accepted.

[jmuchovej]

Thanks for your quick reply! 🙂

Re: (1) – sweet. I hadn't considered turning @preview/iconify into a package in that way. Will definitely look into this a bit more.

Re: (2) – hmm... I'll need to look over how I was trying to use image.decode, then.

Re: (3) – definitely understood. 👍 I'll poke around and figure out which icon sets make the most sense to package and a less impactful update schedule.

Re: naming – I'm not the owner, but I wonder if obtaining permission from Iconify would allow for the naming to maintained? (i.e., Would written permission suffice? Does the package need to come from Iconify's GitHub org? etc.)

There might be another route, to avoid bloating the git-based package registry. Iconify has an API that can be queried for SVG code – though this would require that users have a constant network connection (at least when requesting "un-cached" icons). I don't know what Typst's view on constant network connectivity is, though. (Btw, for "un-cached" icons – I'm really referring to icons which are placed in state.)

However, to pull this API-route approach off, it seems like I'd need to write the extension in Rust then compile it to WASM? This seems like an undocumented capability (or perhaps even an unsupported one). I'm basing this conclusion off the implementation of the titleize package.

[elegaanz]

I would prefer to have the package(s) be part of the iconify GitHub org, because once the name is reserved they won't be able to claim it back if you can't personally maintain it anymore. Having it in the org makes it clear that other people own the package as well, and can submit new versions.

Using the API would not work, as Typst doesn't let you access network, not even through a WebAssembly plugin (they are quite limited in what they can do, there is no file system API either for instance).