-
Notifications
You must be signed in to change notification settings - Fork 243
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[T5 1.1] Enable v1.1 Presets #1948
Merged
divyashreepathihalli
merged 10 commits into
keras-team:master
from
DavidLandup0:feature/t5_1.1
Oct 30, 2024
Merged
[T5 1.1] Enable v1.1 Presets #1948
divyashreepathihalli
merged 10 commits into
keras-team:master
from
DavidLandup0:feature/t5_1.1
Oct 30, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
divyashreepathihalli
approved these changes
Oct 23, 2024
@DavidLandup0 the GPU tests are failing, can you pleas take a look? |
@divyashreepathihalli - Fixed - there was a missing Kaggle link for the XL preset |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Turns out we already operationally supported T5 1.1 (given the gated activations) but only supported vanilla T5 models through weight conversion and presets.
This PR updates the conversion script to include the T5 1.1 variants:
For example:
It also updates the conversion to use the
save_to_preset()
functionality and fixes assertions that raised exceptions, and saves the tokenizer as well.Numerical Equivalence
Behaves equally to:
PCA on flattened outputs running on the same input:
Notes
The XXL version (11B params, 44GB for weights) is too large to run on consumer hardware. I can't run the conversion script on it. Getting XL weights up on Kaggle as soon as the download is finished.
/cc @divyashreepathihalli