Need "server mode" or plugin to run Nextflow from a server

[stevekm]

After a couple years of working with Nextflow and within the greater Bioinformatics space, I have seen several instances where a server is needed to in order to manage and run pipelines. This especially struck me when I was investigating Cromwell and discovered that it has a built-in server mode for managing and submitting pipelines.

The need for a server to manage and run pipelines is pretty fundamental once your use case scales up beyond running just a handful of pipelines, even Galaxy has been around for a long time to help with this.

I and others have found themselves in the situation of needing some kind of manager so that we can automate the process of setting up and running new Nextflow pipelines. We often end up having to build such server apps ourselves.

I think it would be great if Nextflow could offer some type of "server mode", so that we could have some persistent service which could be communicated with, perhaps via REST API, in order to create and run new pipelines, and manage old ones. One consideration for such a system we find is that the parent Nextflow pipeline process itself needs to be submitted to a compute instance (e.g. submit a job to the HPC which runs the Nextflow pipeline which submits its own jobs to the HPC as well).

This kind of feature would be really nice to round out the current stack of Nextflow + Nextflow Tower, and could help to 'complete' the ecosystem and reduce the need for Nextflow users to build their own persistent Nextflow-running API and service from scratch.

[abhi18av]

Hi @stevekm , this is a great suggestion and would probably need to be discussed.

I'd be converting this issue into a Github Discussion and hopefully the suggestion would gain more visibility there.