Fix FusedAdam.zero_grad(set_to_none=True) #1579
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
crcrpar
reviewed
Feb 13, 2023
apex/optimizers/fused_adam.py
Outdated
| @@ -94,8 +94,8 @@ def __init__(self, params, lr=1e-3, bias_correction=True, | |||
| else: | |||
| raise RuntimeError('apex.optimizers.FusedAdam requires cuda extensions') | |||
|
|
|||
| def zero_grad(self): | |||
| if self.set_grad_none: | |||
| def zero_grad(self, set_to_none=None): | |||
There was a problem hiding this comment.
QQ: why is the default None? Given that the default of set_grad_none' and torch.optim.Optimizer.zero_grad's set_to_none is True, it'd make sense to make it default to True to me
There was a problem hiding this comment.
Also would it make sense to check that set_to_none and self.set_grad_none are the same? Raise a RunTimeError if they are not?
There was a problem hiding this comment.
Also note that this is a breaking change from torch. It defaults to False in 1.13 and True in master. Not sure which one would make more sense here.
NouamaneTazi
commented
Mar 10, 2023
Comment on lines
+97
to
+98
| def zero_grad(self, set_to_none=True): | ||
| if self.set_grad_none or set_to_none: |
There was a problem hiding this comment.
gently pinging @crcrpar
Does this look good to you? :)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for set_to_none for FusedAdam.
According to pytorch/pytorch#92731 we set
set_to_none=Trueby default, in accordance to current Pytorch master. see #1579 (comment)