Realtimey features, aka change streams

[tetious]

I'm wondering if there are plans around or interest in implementing something like Mongo's Change Streams?

It seems like it might not be too bad to add, given it is essentially a special-case of replication. This might be an oversimplification based on optimistic naivety, though. 😄

Being able to subscribe to changes at the "Type" level, as well as database level, would be very interesting I think. It would also allow for fleshing out some of the more flashy bits of the Redis (pub/sub) and Mongo feature-sets.

Thoughts?

[lvca]

@tetious in OrientDB we had that: Live Queries: https://dzone.com/articles/reactive-orientdb-live-queries. @antarcticgiraffe how complex would be porting them to ArcadeDB?

On the other side, creating a proxy listener could be a quick start. Our replication log is page-based, so It can't be useful for this. The listener must be at CRUD level. We were thinking of providing such a listener mechanism anyway to hook up custom validation and plugins, so it could be a good reason to start the design of that.

[lvca]

Implemented in #150.

[lvca]

I've added database level listener and at Type level (vertices, edges, documents).

Example of before-record-create listener where vertices with "validated" field equal to false cannot be saved (callback returns false):

database.getEvents().registerListener((BeforeRecordCreateListener) record -> record instanceof Vertex && 
record.asVertex().getBoolean("validated"));

The same by only for vertex type "Client":

database.getSchema().getType("Client").getEvents()
   .registerListener((BeforeRecordCreateListener) record -> record.asVertex().getBoolean("validated"));

[lvca]

Added new paragraph about events: https://docs.arcadedb.com/#Java-Events.

[tetious]

I like this a lot. The validation use-case is compelling!

It's been ages since I wrote any Java, but I'm thinking of taking a stab at adding a web-socket based implementation of after-record-[create|update] for the http API.

My first thought for a design would be a single database-level listener that stuffs all updates/creates into a thread-safe queue. (so as not to tie up the synchronous save with network requests, or load it down with too many listeners, etc) Then a worker thread could pop and publish the record to any sockets subbed to that type/database, provided that user has read permissions for that type and/or the database as a whole. (might need to give permissions a bit more thought)

I can think of two potential designs for the API:

1. Something like ws://*/api/v1/listen/DATABASE/[TYPE], where each type gets its own connection/socket, or you leave off the type for the whole database.

If the app cares about a handful of types, but doesn't want the firehose of the whole database, that would lead to lotsa open connections, and potential connection starving, though.

2. We could use a single endpoint, and sub/unsub via json messages sent via the socket, kinda like this:

ws://*/api/v1/listen, SEND: {"action":"[subscribe|unsubscribe]", "type": "Client", "database": "database"}

I'm doing option 2 with a Mongo backed app right now, and it works quite well.

Does either of these designs seem reasonable? Anything I'm missing?

EDIT: It occurs to me that this will fall apart when dealing with a cluster without some kind of backplane...

[lvca]

I like this idea and I think it could be useful to many users. ArcadeDB Server integrates JBoss Undertow, a fast and lightweight HTTP server that supports WebSocket. Maybe we could provide this out of the box?

[tetious]

That would be ideal, yep. I think it will be a good opportunity for me to dust off my rusting Java skills. Not sure of the timeline, but I'll submit a PR soonish.

My initial implementation will only work when not clustered. A cluster-friendly solution will need a backplane, but it might be possible to use ArcadeDb itself for this. I'm planning to experiment with that. First things first, though!

[tetious]

Also! Once this is properly working with clusters, I'd like to wire in support via the Redis and Mongo drivers. Should be fun!

[lvca]

Sounds good. For any help, don't hesitate to ask. Maybe Discord is quicker for this.

[lvca]

Thanks, @tetious, your PR has been merged into main and will be released with v21.11.1

[lvca]

#163