Returning data on job completion

[TimNZ]

What is the good reason why neither you or Kue support returning data in done()?

A pain in the ass needing to persist a response through another mechanism.

[manast]

If you mean the done callback in the process function, who will be the receiver, and what will be the purpose? the only argument allowed now is an error and it is just to notify the system that the job has failed, I cannot imagine the use of sending other type of data...

[TimNZ]

1. I post a job on the queue - say I'm a web server
2. Job is processed by a remote worker that eventually does a done(data)
3. I monitor queue.on('completed')
4. I pull the return data out of job object and return something to the caller

Surely that is quite a common scenario where you want to offload potentially expensive tasks to workers and keep separate from a front end.

I can't even understand how you don't see a scenario where a worker doesn't return any output. And I don't want to have a separate mechanism for passing result/response data back.

[manast]

Ok, that was a good scenario, I guess I had a lack of imagination this morning :).

What I do in such a case is that I create a new queue just for returning results. The advantage is that if the server waiting for the response is dead when the job processor finishes, then it will get the result as soon as it is started again.

It seems quite easy to implement such a feature so I will see if I can add it in the next release.

[TimNZ]

Great thanks. The key thing is it must be easy for the job submitter to pull the response out with low overhead.

Can you clarify how you would architect the response queue solution today? I could have multiple FE servers and I need affinity to the original FE server that submitted the initial job.

It sounds like you are saying the worker posts a job on the response queue with result data, but then how do I ensure FE server #1 gets that job. Wouldn't I have to have a queue for each FE server?

Thanks for any guidance you can give how to do it today Manuel.

[manast]

ok. The way to do it is that you also pack in the job data a unique queue name where to put the result. Every FE knows what queue to listen for the response. Remember queues are very cheap, you can have as many as you want.

[TimNZ]

I guess I can do that with a random generated queue id for each FE that gets passed in the original job data.

Better do some performance testing before I commit though. I need fastest return time possible.

My other options include something more enterprise like RabbitMQ or Zero mq.

[manast]

If you do benchmarks please publish them, I am interested to know the results.
Btw, zeromq is really cool, but keep in mind that the queues are not persistent.

[LewisJEllis]

Any further thoughts on this? I see the potential workaround with a response queue for each frontend server, but it could be much simpler to have it built in.

[manast]

Yes, although it is feasible to return data in the onCompleted event when a job has been performed, I dont really think it is a good practice, simply because it is not guaranteed that you will be listening at the right time and you will end loosing data. Best is to use a different queue to transmit results from the worker to the process that needs the results, that way you will always get the end results and not loose any data.

[TylerBrock]

Hey did you ever wind up adding this feature? It would be killer to have bull support this use case.

[albertjan]

@TylerBrock see #178

[TylerBrock]

@albertjan thanks. Nice work sir!

[rinogo]

For those who land here from Google: Bull 3.0.0. RC 4 added support for the glorious await job.finished(). Enjoy! :)