Spiking loss

[davipatti]

Hi @theosanderson

Firstly thanks for developing chronumental, it's been really helpful!

I've noticed that the loss and error metrics spike quite noticeably during runs:

I don't know enough about the optimizer that is being used to know whether this is expected behaviour, but it does seem a little odd.

I guess you could get unlucky if the run finished during a spike. In this run it looks like the final parameters are actually just at the end of a spike and not properly back on the plateau, which I assume would be preferable.

Anyway, I wondered whether this is something to be concerned about, whether its pointing to me doing something wrong, and/or whether it might point you to something that could be improved.

Happy to share more details and the data.

Many thanks
David

[theosanderson]

Hi David,

A couple of things:

On this very specific question:

In this run it looks like the final parameters are actually just at the end of a spike and not properly back on the plateau, which I assume would be preferable.

My belief is that this should not be an issue.

Unless this is set we should use the params that gave the lowest loss. (Unless there's a bug of which I'm not aware).

More generically though, I have a fair bit of paranoia about whether I may have broken some bits of Chronumental with some changes in the last couple of years and I have some overdue benchmarking to perform. It is certainly good to be cautious about its behaviour.

All the best

Theo

[davipatti]

Thanks for the response.

It's reassuring that by default the params with the lowest loss (and not the final set) get used.