New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[QUESTION]Is V-ZB still not support overlap_grad_reduce and transformer_engine? #53

Open

LitLeo opened this issue Dec 5, 2024 · 1 comment

LitLeo commented Dec 5, 2024

overlap_grad_reduce and transformer_engine can both bring significant performance benefits. Are they still not supported?

QPHutu commented Dec 6, 2024 •

edited

Loading

transformer_engine is not supported because it's not accessible to split backward pass into weight gradient and activation gradient.

For overlap_grad_reduce, I think it's already well supported. Please provide more information if you found any issue. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment