dask: spawn jobs only to selected cluster nodes

[tiborsimko]

Optionally, REANA spawns user jobs to cluster nodes labelled with certain node_label_runtimejobs value, see https://docs.reana.io/administration/deployment/deploying-at-scale/ This helps to separate "infrastructure pods" from user runtime workloads pods for better isolation and safety in case user pods eat a lot of memory.

This is implemented in the job controller by doing:

$ rg -C 3 REANA_RUNTIME_JOBS_KUBERNETES_NODE_LABEL
• reana-job-controller/reana_job_controller/kubernetes_job_manager.py
22-    KRB5_STATUS_FILE_LOCATION,
23-    REANA_JOB_HOSTPATH_MOUNTS,
24-    REANA_RUNTIME_KUBERNETES_NAMESPACE,
25:    REANA_RUNTIME_JOBS_KUBERNETES_NODE_LABEL,
26-    WORKFLOW_RUNTIME_USER_GID,
27-    WORKFLOW_RUNTIME_USER_UID,
28-)
--
234-        if self.rucio:
235-            self._add_rucio_init_container(secrets_volume_mount, secret_env_vars)
236-
237:        if REANA_RUNTIME_JOBS_KUBERNETES_NODE_LABEL:
238-            self.job["spec"]["template"]["spec"][
239-                "nodeSelector"
240:            ] = REANA_RUNTIME_JOBS_KUBERNETES_NODE_LABEL
241-
242-        backend_job_id = self._submit()
243-        return backend_job_id

We should make sure that reana-run-dask-... pods are also scheduled only onto cluster nodes labelled with "runtimejobs" label, if the deployment uses this feature.