The trainer stopped and hasn't progressed in several minutes.
Usually an issue with the GPU's communicating with each other. See the NCCL doc
Exitcode -9
This usually happens when you run out of system RAM.
Exitcode -7 while using deepspeed
Try upgrading deepspeed w: pip install -U deepspeed