Add groups in status output

[bdice]

Feature description

I submitted groups containing multiple operations, but the status output does not reflect this. It appears that groups are currently not integrated with the status output code, resulting in unknown [U] status codes even when the operations are contained in actively running groups. (See screenshot.)

Proposed solution

Perhaps we need another status code for operations like [GA] for "group active" or [GQ] for "group queued." We could also add a separate line in the status output for each group present in the project. I think it might even make sense to do both, so that it would look like this. Assume my_group contains my_operation1 and my_operation2. 10 jobs were submitted as my_group, 5 of which are queued and 5 of which are running.

operation        number of eligible jobs  submission status
-------------  -------------------------  -------------------------
my_operation1                         10  [U]: 10, [GQ]: 5, [GA]: 5
my_operation2                         10  [U]: 10, [GQ]: 5, [GA]: 5
my_group                              10  [U]: 10, [Q]: 5, [A]: 5

Additional context

Related to #469.

[csadorf]

Maybe the status output should only show groups. I've argued before that operations are in essence just groups of size 1. I think internally we still need to treat operations as first-class entities, but for the status view, everything could be a group?

[b-butler]

@csadorf I like the idea in principle, but I think that only showing groups would be confusing to users. Since every operation has its singleton group, the operations would seem to be there regardless (as they are properly groups). However, when submitting through a user created group, they would show as unsubmitted, but would not be eligible for submission with the only indication of their status being the status of the composing group. That may be fine since multi-operation groups are an opt-in feature, but we would just need to be very clear in our documentation.

I would also potentially want more distinction from singleton operations and user/multi-operation groups since in this case [GA], [GQ] could never apply to them. I do think having a separate status for user group submission makes sense as just because a group an operation belongs to is submitted or active does not mean that that operation will run/complete.

I am in favor of @bdice's solution with an additional heading/separation for user groups (even though semantically multi-operation groups are not different from single operation groups)

[csadorf]

@b-butler I see your point and I see why it might be necessary to differentiate individual operations regardless of their group associations. That said, I will keep pushing towards what I consider "conceptual elegance", but I agree that there simply is a subtle difference between operations and singleton groups and trying to ignore that might create more confusion after all.

[bdice]

@csadorf @b-butler Could you offer a proposal for your desired output? I think that having some concrete designs would help, and then perhaps we can discuss further at the next developer meeting.

[csadorf]

I've tried to come up with something, but had enormous trouble in doing so. Do you have a good example workflow (state) that we could all work off from?

[bdice]

@csadorf I copy-pasted a status output from another project and manually modified it in a text editor to have different operation names and statuses. I was thinking of this as more of a whiteboard sketch, just to understand what kinds of output you’d want to see and how it compares to the example I posted at the top of this issue. 😊

I made up a “state” that I described in the top post:

Assume my_group contains my_operation1 and my_operation2. 10 jobs were submitted as my_group, 5 of which are queued and 5 of which are running.

[b-butler]

@csadorf Would having a groups flag make sense? We could then delineate between singleton groups and operations, by only showing operations with the default output (with @bdice's new status categories [GA, GQ]), or show a group focused output with the current status options (as the singleton group isn't submitted when a group composing the same operation is).

So, something like,

operation        number of eligible jobs  submission status
-------------  -------------------------  -------------------------
my_operation1                         10  [U]: 10, [GQ]: 5, [GA]: 5
my_operation2                         10  [U]: 10, [GQ]: 5, [GA]: 5

by default, and

group            number of eligible jobs  submission status
-------------  -------------------------  -------------------------
my_operation1                         10  [U]: 10
my_operation2                         10  [U]: 10
my_group                              10  [Q]: 5, [A]: 5

with the groups flag.

I think this avoids conceptual ambiguity while also providing the user the information necessary.

[bdice]

Another option discussed with @atravitz and @b-butler during office hours. We like this because it gives more info by default.

operation/group     number of eligible jobs  submission status
----------------  -------------------------  -------------------------
my_operation1                            10  [U]: 10, [GQ]: 5, [GA]: 5
my_operation2                            10  [U]: 10, [GQ]: 5, [GA]: 5
my_group (2 ops)                         10  [Q]: 5, [A]: 5

[vyasr]

The current plan is to go with the proposal in the last comment. The first column will be renamed to operation/group (with groups indicating the number of operations in parenthesis). The status symbols in the submission status column will be prefixed with a G for group status.

[cbkerr]

Resolved by #593