GPU support

[ksatzke]

Environment]: first on bare metal with NVIDIA GPUs
[Known affected releases]: master (includes all releases)

allow KNIX functions to use available GPU respources.

[iakkus]

How are the GPUs multiplexed to various applications? For example, if there are two applications/workflows that want to use a GPU and there is only one GPU on the host, can they share the same GPU?

Or do you keep track of available GPUs and exclusively assign them to applications?

[ksatzke]

Hi Ekin, if we specifically want to support CUDA and nvidia-docker. The NVIDIA engineers found a way to share GPU drivers from host to containers, without having them installed on each container individually. https://github.com/NVIDIA/nvidia-docker So one GPU can shared between different sandboxes using this framework. This would also happen with e.g. two workflows running in different containers. BR Klaus

[iakkus]

I see, thanks!

For more information, the following is from the wiki of the nvidia-docker repo.

Can I share a GPU between multiple containers?
Yes. This is no different than sharing a GPU between multiple processes outside of containers.
Scheduling and compute preemption vary from one GPU architecture to another (e.g. CTA-level, instruction-level).

Can I limit the GPU resources (e.g. bandwidth, memory, CUDA cores) taken by a container?
No. Your only option is to set the GPU clocks at a lower frequency before starting the container.

Can I enforce exclusive access for a GPU?
This is not currently supported but you can enforce it:

At the container orchestration layer (Kubernetes, Swarm, Mesos, Slurm…) since this is tied to resource allocation.
At the driver level by setting the compute mode of the GPU.

[ksatzke]

Yes but it seems as if GPU support for knative is still an open issue? knative/client#490

[iakkus]

I'd think that the GPU support would first be supported only on bare metal environment until that issue is resolved for knative.

[iakkus]

Implementation started on branch feature/GPU_support.

[iakkus]

This issue is subsumed by #87.