Switching to NATS for worker-engine or engine-engine communication

[abelanger5]

The goal of Hatchet is to provide a scalable and distributed system for managing tasks. Eventually we would like to support geographic distribution of workers that connect to a set of brokers living close to the workers.

In Hatchet architecture terms, we would like the dispatchers to live close to the workers and write results to a nearby data store. We are still a long way from working out the consistency and replication model here.

The first version of Hatchet is using RabbitMQ to broker messages between different engine components. Many users and people giving feedback have suggested switching to NATS instead. I like this idea particularly for engine-engine communication, because it will be easier to isolate and scale our engine components via nats than amqp.

I am uncertain about worker-engine communication, because we get a lot out of the box with gRPC. Starting this thread to entertain some thoughts and ideas from the community and add my own input when we start mapping it out in more detail.

[gedw99]

well you can use Protobufs with NATS JetStream. So you can reuse the protobufs. there are many examples of this on GitHub.

also Choria is similar to Hatchet and built on NATS Jetstream. https://github.com/choria-io

[geekodour]

I think it'll be nice to have NATS both for engine-engine and worker-engine just for consistency.

Referring to #89, do we have a choice to go NATS or postgres for this now? or I am misunderstanding something?

[gedw99]

https://github.com/choria-io/asyncjobs

Nats can do all of it without PostgreSQL