Plugin system

[eliandoran]

This would be one of the single most important changes, if implemented.

The idea is simple: I believe the current system of "plugins" is not scalable. Plugins are added by creating notes into your own note tree and they are distributed in various ways and in different repos.

All reputable note-taking applications (including big-guys such as Obsidian) have a plugin system. And we need this.

There are multiple challenges here:

1. Defining the API: in order to have a mature system we need to know what APIs to expose. There is already a pretty good API that is used by the existing scripts, however we need to check if we need to expose more than what is already provided.
2. Extracting existing functionality into plugins? Honestly, I think some of the features of Trilium don't necessarily need to be in the main repo as they might not be used by the main majority of people and they increase the maintenance burden. Splitting into plugins might also simplify the codebase since we are grouping/modularizing features that would otherwise be split across the application, making it more difficult to find them (and increasing the overall size of the codebase).
3. Distribution system. Plugins come with a security risk and probably we shouldn't allow the regular user to install plugins from everywhere. Maybe having a central repo, probably another GitHub repository containing manifests or even actual code that is reviewed by the main devs is a good way forward.

Looking forward to your opinions.

[deajan]

Not enough of a frontend dev here, but one of the advantages of having a "note plugin system" is that every remote synced trilium instance automatically inherited said plugins, without any setup burden.
I am myself using a trilium server and 3 trilium clients, and I'd hate to have to regulary update plugins on each of my instances.

Perhaps the distribution system you're suggesting could come handy here, allowing a client to automatically pick up the same plugins as the server ?

[eliandoran]

Indeed, note-based plugins do support this use case.
I suppose we can always use hidden notes to store the plugins internally, which would be synchronized automatically.

Nevertheless, @deajan, using the existing process, how do you handle update? I could never bring myself to manually update plugins/themes in Trilium.

When developing https://github.com/eliandoran/breeze-trilium I've ended up using the REST API to automatically upload the theme just to bypass this chore.

[eliandoran]

@rauenzi , we think alike. :)

[zerebos]

When developing https://github.com/eliandoran/breeze-trilium I've ended up using the REST API to automatically upload the theme just to bypass this chore.

I did the same thing with all of my addons! So much that I even made trilium-etapi so I had a simple and consistent way to do it for all my projects!

[eliandoran]

@rauenzi , cool, if things go well with the org we can move https://github.com/rauenzi/trilium-etapi to the org as well with your permission.
I'm planning to do the same with breeze-trilium.

[zerebos]

Yeah I don't mind moving any of my Trilium related repos.

[zerebos]

As someone that's built multiple plugin systems before, I have a lot of thoughts when it comes to this. I'll try to gather them and update this comment, but I absolutely agree with everything you've said Elian. A proper plugin system is absolutely crucial to have.

[ProfDoof]

@rauenzi, I would love to hear these thoughts when you get time.

[meichthys]

Extracting existing functionality into plugins?

This is a VERY good point, and i like the idea very much 👍

[zerebos]

Yes I glossed over it when I was reading initially, but this is a really great idea. The more customization to the end-user the better! And from a development perspective it makes things more modularized and manageable rather than having things overly tied together like spaghetti.

[shutaozhenzhen]

For API, seems trilium is open-source, I think currently, we can just export trilium instance in backend API can allow any function we need.

[deajan]

As long as the plugins can be "stored like notes" like @rauenzi suggested so they will be synced to clients without them having to manually follow plugins, that's a very good idea :)

[ProfDoof]

One thought I had while I was thinking about this is that if multi-user functionality is something desirable at some point, we need to be extra careful about where plugins are stored and where they are run (should plugins only ever run in the frontend and only interact with the backend through an API of some kind?)

Another thing I was considering, and y'all may have more thoughts than me since you've probably interacted with the API more than I have, is that it could be nice for the plugin system to be able to leverage functionalities in libraries available to the languages for the supported plugin languages (currently it's just JavaScript and I'm fine with it staying that since I can't quite figure out how to make typescript work smoothly from a code note, but it could be interesting to allow for other types of code notes to run and allow for possibly a simplified scripting experience for those who are not as adept with programming). But that does run into the issue of now having to install the dependencies for them (even if it just means putting those dependencies into their own code notes or something).

So, yeah, I'm probably scope creeping myself right now but they are cool ideas, just not sure about approaches.

[eliandoran]

@ProfDoof , I think JavaScript is a good distribution method for plug-ins. Personally I would keep it that way. That doesn't mean that developers for more advanced plugins can't use TypeScript and transpile it to JavaScript before generating the plugin.

Using notes for synchronizing the plugins does sound like a decent idea, similarly to how the settings are now managed as notes in the latest versions of Trilium.

[ProfDoof]

So, I think JavaScript is fine for now, but ideally, I'd like to support people being able to make their hot paths faster if they would like. Especially if we want to because to make certain built in functionalities just a plug-in. The only ways I can think to do that is to have compilers that are built into the backend or to have integration with outside sources. It could be something like allowing file watching for external notes, or having further integration with external tools that allow plugins to be conveniently updated.

[ProfDoof]

I was talking about a plugin system with someone else and someone mentioned this right here: https://extism.org/

This may actually be helpful since people can already use code notes and so theoretically they could use any code note and as long as we have compilers for that language (which I feel like we could figure out how to make that only download as necessary or through a docker variant or something) then they could compile to wasm and be run on the front-end or the back-end.

This more seems like an interesting concept but I will admit that I'm not sure how this will play with interacting with the dom which is rather important.

[Alumniminium]

I agree, we need a distribution system. But we should keep the plugins as notes, still. That's consistent across instances/clients and makes it easy for people to edit them. How we will deal with plugin versions and updates might be a more complex subject.

[ProfDoof]

So, I did have a thought on a distribution system.

What about a shared note with a label. Then, if we add support for including shared notes, a user could just include the shared note, and gain access to a plugin, or a registry of plugins. Then, we could have a single page that aggregates all the available plugins, gives access to a search bar of their plugins, and they can toggle the plugins on or off, or import the plugins into an active plugins note or something. Then, settings could even be just another note or attributes on the plugin. Something like namespaced attributes could work #plugin:setting:blee="blah blah blah".

You could have something like

stateDiagram-v2
    source: Plugin & Theme Registry
    source: #registry #shared
    p1: Plugin 1
    p1: #plugin
    p1s: Plugin 1 Settings
    p1s: #pluginSettings
    p2: Plugin 2
    p2: #plugin
    p3: Plugin 3
    p3: #plugin
    t1: Theme 1
    t1: #themeCss 
    source --> p1
    p1 --> p1s
    source --> p2
    source --> p3
    source --> t1

Loading

or

stateDiagram-v2
    source: Plugin & Theme Registry
    source: #registry #shared
    p1: Plugin 1
    state "#plugin #plugin:setting:blee" as p1
    p2: Plugin 2
    p2: #plugin
    p3: Plugin 3
    p3: #plugin
    t1: Theme 1
    t1: #themeCss 
    source --> p1
    source --> p2
    source --> p3
    source --> t1

Loading

which would then allow you to just include a registry and voila, you have access to plugins. Then the distribution could just be through the shared notes.

As for updates, notes already allow for syncing, so we could have included notes support updating the imported notes. You could also just allow for tagged note revisions and then tagged note revisions could be used to allow people to set the version of the plugin.

Ok, now that I've overengineered this, I'm going to go work on my research XD.

[mm21]

I like it. This is similar to an option discussed here: zadam#4012 (reply in thread)

[ProfDoof]

Looking at the conversation, one thought I have is that perhaps the external notes can really just be synced notes and we make the sync tool integrated in such a way that it's easy to extend it via backend plugins so you can define a new method to sync in notes from various sources including git, filesystem, base sqlite DB, etc.

Then, one option would be to allow it to sync in both directions so you could write your code in whatever, then push it and sync it to Trilium via git. Or you could write it and sync it to GitHub via git.

Then, you can just say, we have three types of sync

1. ASAP sync - basic realish time sync
2. Negotiated Sync - has a pop-up or update page or something that users can update as they want.
3. Request-only sync - Synchronized only on request by the user. Not sure necessarily an exact use case here but it feels like the logical conclusion. Perhaps for sharing information that should only be updated as needed. Something like data lists that are good for a while and only need updates when they have gone bad.

[shutaozhenzhen]

This idea is better discussed in a separate thread

[TheOneValen]

I use many modifications in trilium. I place them in an "Extension" subnote. It has all the sync stuff etc. taken care of. The only painpoint is updating (external) modifications and I cannot publish my work (easily) because there is data in there that is specific for my instance.

I propose creating some kind of metadata or leverage the existing metadata system to mark stuff as "variable". I.e. it is "emtied" or "defaulted" by using an "export as plugin" checkmark. And an extra checkmark for updating a tree so that all "variables" are kept as they are and not overwritten.

This I think would keep the current spirit and make it easier to publish and update extensions.

Then one could go further as to maybe import from a github repo or something like that. I propose to keep it as simple as possible to not add another layer of complexity and or more types of modification.

[CobriMediaJulien]

The PlugIn note could be also some standard not type or folder you could hide like archived notes.

[ProfDoof]

https://github.com/orgs/TriliumNext/discussions/6#discussioncomment-8935425 I think that I was discussing something similar to that in this reply here. Then, the plugins themselves would have you manipulate settings through the main top-level plugins note (cuz everything is a note) or they would have the settings exposed through the plugin's respective note and you would just change the setting there and the default would be stored in the history or could be fetched by hitting the shared endpoint to allow people to reset defaults on a setting.

[meichthys]

For reference, this is being tracked as a Notes issue here: #107