how to restart a training job that was killed by walltime

[GengSS]

Hi I wonder how to restart a training job (by nequip-train) that was killed because of walltime?

I just tried to resubmit the job at the original folder, but it fails immediately.

Thank you very much

Best
Geng

[DavidW99]

Hi Geng,

Could you share your error message?

Generally, if your previous run is successful, then the software will store the best model in the result directory and it should automatically load that model when you continue the training. However, if your previous training crashes due to an error when you run it the first time but not due to the walltime restriction, then no best model is stored and you shall remove your result directory to completely restart the training.