Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reuse Existing Cuda Streams each Epoch #308

Open
wants to merge 1 commit into
base: v1.1.0
Choose a base branch
from

Conversation

warner-benjamin
Copy link
Contributor

This PR modifies the FFCV Loader and EpochIterator to create one set of Cuda streams and reuse them each epoch instead of creating new Cuda streams every epoch.

The current method of creating new Cuda streams every epoch can cause increasing memory allocation if using a GPU transform, as each epoch the GPU transform will allocate new memory in the new Cuda stream. This doesn't cause any errors, as the prior epochs' allocation can be reused. But it does make keeping track of GPU memory usage more difficult and can hide real memory overflow errors.

I don't think this will cause any issues with distributed training, but am unable to test.

If wanted, I can create a flag like recompile to recreate Cuda streams every epoch to match current behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant