Noise Is Not the Main Factor Behind the Gap Between Sgd and Adam on Transformers, But Sign Descent Might Be

See subfolders to reproduce our plots or see the experiment definitions

Links
OpenReview: https://openreview.net/forum?id=a65YK0cqH8g

Citation

@inproceedings{kunstner2023noise,
    title={Noise Is Not the Main Factor Behind the Gap Between Sgd and Adam on Transformers, But Sign Descent Might Be},
    author={Frederik Kunstner and Jacques Chen and Jonathan Wilder Lavington and Mark Schmidt},
    booktitle={The Eleventh International Conference on Learning Representations },
    year={2023},
    url={https://openreview.net/forum?id=a65YK0cqH8g}
}