Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build a collections adaptor #758

Open
josephjclark opened this issue Sep 25, 2024 · 4 comments · May be fixed by #789
Open

Build a collections adaptor #758

josephjclark opened this issue Sep 25, 2024 · 4 comments · May be fixed by #789

Comments

@josephjclark
Copy link
Collaborator

josephjclark commented Sep 25, 2024

Overview

Create a Collections adaptor that speaks to the Lightning Collections API.

It should probably be suffixed by the backend data store, like collections-postgres or collections-lightning or something, so that later it's easier to introduce new collection types, like collections-redis.

The collections API is a bit unusual in that it will be loaded as a second adaptor to jobs. It's a second class citizen in the job code. So everything needs to be namespaced - collections and operations and stuff. So collections.get rather than get. Otherwise there's a risk for clashing with the main adaptor namespace.

API

/**
 * Get one more values from a collection. For large collections, use each.
 * options can be a string key name (including wildcards), or an object
 * { key, created_after, created_before, inserted_before, inserted_after }
 * Throws if created AND inserted are specified
 * If no wild card or query included, the value will be written to state.data (or throw if not found)
 * If a wild card or query is included, an array of values will be written to state.data
 */
collections.get(name, query) // writes to state.data

/**
 * Upserts one or more values, as a { key, value } pair, to the named collection
 * If any errors are returned by the server, this will be thrown
 */
collections.set(name, data, options) // throws if errors

/**
 * Remove one or more values from the collection
 * Options can be a string key or a query object
 */
collections.remove(name, query, options) 

/**
 * Iterate over values in a collection which match the query
 * Query can be a wildcard string or object
 * The callback will be invoked with { key, value } on state.data
 * Or what if we do `(state, key, value)` ? because the first thing you need to do
 * is deconstruct anyway
 */
collections.each(name, query, (state) => {})

Configuration

{
  collections_key: /* a JWT */
  collections_endpoint: /* set by the worker  */
}

Note that config will be set by the worker automatically. Maybe later Lightning will take more control.

Later work

Maybe add APIs for administering a collection

@josephjclark
Copy link
Collaborator Author

Stu: we should go streaming first on this! The adaptor uses an async iterator with a stream under the hood (but passes full objects to the callback). It should also decode on the fly for get etc

@josephjclark
Copy link
Collaborator Author

josephjclark commented Sep 26, 2024

Query: should it just be time series? So you get a key by id, or you get keys between two dates (or before/after one date)

Maybe allow key scanning? Pass a pattern and we'll find keys which match that name

@josephjclark
Copy link
Collaborator Author

Probable API:

/**
 * Get one more items from a collection. For large collections, use each
 * options can be a string key name (including wildcards), or an object
 * { key, created_after, created_before, inserted_before, inserted_after }
 * Throws if created AND inserted are specified
 * If no wild card or query included, the item will be written to state.data (or throw if not found)
 * If a wild card or query is included, an array will be written to state.data
 */
collections.get(name, query) // writes to state.data

/**
 * Upserts one or more items, as a { key, record } pair, to the named collection
 * If any errors are returned by the server, this will be thrown
 */
collections.set(name, data, options) // throws if errors

/**
 * Remove one or more items from the collection
 * Options can be a string key or a query object
 */
collections.remove(name, query, options) 

/**
 * Iterate over items in a collection which match the query
 * Query can be a wildcard string or object
 * The callback will be invoke with { key, record } on state.data
 */
collections.each(name, query, (state) => state)

@josephjclark josephjclark linked a pull request Oct 16, 2024 that will close this issue
11 tasks
@josephjclark
Copy link
Collaborator Author

I've remembered that in the original doc, Taylor suggested a key generator function. I've been thinking about this all morning and I actually think it's a way better solution - so I'm going to deviate from the spec on set

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Icebox
Development

Successfully merging a pull request may close this issue.

1 participant