Exclude SYSTEM sessions from resource allocation while keeping per-container resource limits

[achimnol]

Currently, SYSTEM session containers also occupy the resource allocation and sometimes this confuses our users and admins when they figure out the remaining resources available for computing.

We should make it intuitive to see the remaining "usable" resources, and SYSTEM session containers makes it difficult as they are hidden but still affects the scheduling process. It is hard to design the UI to display the remaining "usable" resources as the actual agent occupation differs from what the user see in the UI's resource indicators. The problem becomes more complicated if we consider hide_agents=false configuration.

Let's resolve this situation by making SYSTEM session containers to have only resource limits but not resource allocation:

• Assume that they DO NOT use accelerators and require relatively small amount of CPU cores and memory, allowing oversubscription.
• Just keep track of the maximum number of SYSTEM session containers per agent.
• Just use the minimum CPU/MEM resource requirements to set the resource limits.

Note

• Resource allocation: Assign an exclusive, dedicated portion of resources which the scheduler will treat as "consumed". (manager/agent-level scheduling)
• Resource limit: Ensure the container cannot use more resources exceeding the configured limit. (container's cgroup configuration, jail, hook, and some env-vars)
• Normal Backend.AI session containers have both.

This conceptual model change make it possible to merge #710 into the SFTP-only resource group and hidden agents running along with the storage proxy. Both SFTP sessions and filebrowser sessions could be treated in the same way.

Expected scope:

• We may need to add many branches in the scheduler to apply the different semantics to SYSTEM pending sessions. For instance, we should skip or override the normal scheduler and agent selection strategy for them.
  • By default, we could apply a simple load balancing of SYSTEM sessions by selecting the agent with the minimum number of running SYSTEM sessions within the target resource group.
• After implementation, we could exclude SYSTEM sessions from the normal session listing APIs, except for when explicitly requested to include it via a new option.