Open Provider/Connector Protocol

[arendjr]

At Fiberplane, we use Providers (https://fiberplane.dev/blog/fiberplane-providers/) for connecting to arbitrary data-sources. The fp-bindgen protocol we define for this is pretty basic at the moment: It allows providers to perform HTTP requests, log messages, access the current time and generate random numbers. But that's it.

We know we will expand this protocol over time as we come across use-cases that may require it. Some things we may anticipate are support for non-HTTP sockets, persistent connections and generally more data types (currently we mainly focus on logs and metrics).

@joepio mentioned here's working on a service for open government access (please correct me if I didn't understand right), for which he also needs to develop connectors that can pull data from various sources. Obviously the types of data that are interesting for such a platform are different than what we currently support in Fiberplane Studio, but it could be interesting to see if there's overlap in concerns here...

• Are there data types that are valuable to support for the both of us?
• Do we want to share a specification for the capabilities our hosts environments offer to providers/connectors?
• Should we create an open standard for this protocol definition?

Please let us hear your thoughts, and let's see where this leads :)

[arendjr]

In the context of this discussion, it's also worth mentioning that we're looking at OpenTelemetry standards for our Providers to support logs, metrics and traces.

[joepio]

Thanks for posting this! A small correction: I'm currently working on a plug-in system for Atomic-Server, which aims to be a personal data store / knowledge base. The goal is to have importers / connectors for various data sources. This might include things like iCal synchronization, fetching a list of Tweets, importing an album from Google photos, synchronizing your Spotify playlists...

An open provider protocol / crate may increase compatibility between plug-in ecosytems. It would be pretty cool if Atomic-Server plugins could easily be used (or converted) to work with Fiberplane and vice-versa. Of course many hosts will have highly custom interfaces, though.

I think there are some functionalities that may be very commonly used in a plugin-runtime:

• HTTP Access (i.e. a fetch equivalent), you already have one in the examples.
• HTTP middleware (i.e. receive an HTTP request, do something with it, output a modified HTTP request)
• Network Access (TCP/IP), for other types of network API's
• File system access
• EDIT: Key/Value store, similar to BtreeMap / sled to provide scoped / safe persistence to a plugin

Take these with a good amount of salt, as of now it's very unverified that these will be common requirements :)

[mies]

Thanks for the feedback @joepio! I can definitely see the HTTP use-cases being relevant for Fiberplane as we expand beyond just observability tooling. Think GitHub or CI/CD integrations.

[emschwartz]

It's interesting to think about some kind of standard for the plugin runtime. If the plugins have full network and filesystem access, I'd imagine that WASI might be the right type of abstraction.

Short of that, it feels like HTTP access + just a bit more might be the right level for a sandboxed plugin. It might be worth looking at the runtime APIs of some serverless platforms like Cloudflare Workers to see what else they've added.

[joepio]

Cloudflare Workers

Ooh yeah, that reminds me: maybe also a Key/value abstraction, similar to a BtreeMap provided by the host, maybe scoped to the plugin. It's just a simple, yet powerful, way to do persistence, and it can be implemented with many databases (sled, redis, etc)

[arendjr]

Through Reddit, I just stumbled on this: https://lib.rs/crates/assemblage_kv
Definitely a demand for a Key/Value storage in these use cases.