Stacked at evaluation at the end of training for multi-GPU and patience termination

[dbidoggia]

Dear all,
I observed that if I train on multiple GPUs and the trains ends with "patience" criteria the evaluation done after the training get stacked after loading the checkpoint (I think at this point

mace/mace/cli/run_train.py

Line 702 in 4081abd

distributed_model = DDP(model, device_ids=[local_rank])

).
I do not observe this problem if I use "distributed:false" or if the training ends after reaching max_num_epochs.

[ilyes319]

Hello,

Thank you for reporting that. Could you tell me a bit more in details what you mean by "stacked"?
Would you helpful if you could attach your log, or any unexpected output that you observe.

[dbidoggia]

Thanks for your reply.
I attached the log file. It gets stacked after loading the checkpoint, the calculation goes on until slurm killes the job for having reached the max required time apparently without doing anything or returning any error.
test_run-3242_debug.log

[alinelena]

I have noticed something similar, looks like a race condition, since I could not reproduce it consistently.