Using cgroups to limit job resource usage

[chrisburr]

We recently failed to add support for cgroups in #7723 due to there being no session dbus instance started for the non-interactive jobs started by the batch system.

@fstagni pointed out to me that ALICE has made progress with this in:

https://indico.cern.ch/event/1338689/contributions/6010982/attachments/2952948/5191366/CHEP2024_Subdivision_GM.pdf

We should see if we can profit from it also.

cc @sfayer as you helped me understand this in the first place.

[fstagni]

Few additional minor bullets after talking with the author from ALICE:

• they patched HtCondor and SLURM
• the WN needs to be Rhel9
• they create a "slot cgroup" and inside it sub-cgroups
• they create first the cgroup and inside it they launch apptainer (IIUC we were try to do the opposite) and inside it the job
• check the backup slides for some implementation (and caveat) details

[sfayer]

Hi,

I had a quick look on our production HTCondor at IC and we do seem to have the patch already: A few of our cgroup control files for the per-slot cgroup tree are owned by the job user (rather than root), so it should be possible to create custom groups below that.

We're running the feature release of condor (to fix an unrelated cgroup bug!), so I'm not sure if it'll be available at most condor sites yet, but it certainly looks promising... I can't comment about the other batch systems, I felt they seemed less optimistic about getting the patches generally available for those in the talk.

Regards,
Simon