Create a framework for long running commands being run by Choria

[ripienaar]

We would like to be able to make running long running shell stuff a bit easier, this has a lot of hurdles since once running it will require management - status, kill, list etc.

While that in its own is not hard, it gets harder when agent foo#bar started the process based on RBAC for that agent and action, we would need to be sure that those managing the processes also have RBAC access to the agent that first created it.

So we'll make a basic supervisor that monitor the process - pid, stdout, stderr, exit code etc - and then send all above to Streams if desired for observation by a client - this way client side code can write programs that react to the finishing of these long running commands.

Lots to work out here, for now we start with the supervisor.

• Create basic framework that supervises a process (#2209) Adds a basic long running execution framework supervisor #2210
• Extract the authorisation from mcorpc so its callable from compiled-in go agents (#2209) Extract authorization into a external callable #2212
• Create a agent that can list, status, signal processes and CLI to talk to it, should do authorisation around signal so only those with access to the agent#action that created the process can kill it (#2209) Add basic agent #2211 (#2209) Add a signal API #2213
• Ensure this is easily usable from within agents - external and compiled in
• Create a framework that can view data from the stream and get like a digital twin of the process
• Create CLI that can observe running commands

Streaming Status

When running a command under the supervisor it will, using Submit, publish to JetStream updates:

• choria.executor.$requestid.$jobid.spec - the full Process as JSON
• choria.executor.$requestid.$jobid.pid - once started the pid that the program is running as, the pid number is published no json wrapper etc
• choria.executor.$requestid.$jobid.out.stdout - lines received on stdout, could be one line or multiple lines, should be published around every 500ms or whenever stderr arrives after stdout
• choria.executor.$requestid.$jobid.out.stderr - as above for stdout
• choria.executor.$requestid.$jobid.hb - a heartbeat sent at the configured interval indicating the process is still running and supervised
• choria.executor.$requestid.$jobid.exit - the exit code and any error found during wait call as a json document

The stdout and stderr is, as best possible given buffers, interleaved and published in the correct order but there's a small chance that some out of order stdout/stderr interleaving might happen

Authorization Model

Authorisation has always been the hurdle here, if Bob starts a process Jill might take over from Bob and might even need to kill it or see its output etc. But how do we know Jill is allowed to kill this process?

What I think will work is if the Process instance records Agent, Action and Caller and then any action that might have to do something like Kill the process would check if the new caller has access to the Agent and Action that initially started the action using normal authorisation.

We'd send output to the Stream - if opted into, off by default - but interactions like kill and seeing the command/args being run would need to match authorisation.

Normal status would not require that since we don't include stdout/stderr/command/arguments in the status - but should one ask for full status or any of those details then authorisation must pass.