Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enable data loading for data parallel training #49

Merged
merged 2 commits into from
Feb 8, 2024
Merged

enable data loading for data parallel training #49

merged 2 commits into from
Feb 8, 2024

Conversation

tianyu-l
Copy link
Contributor

@tianyu-l tianyu-l commented Feb 7, 2024 •
edited
Loading

Stack from ghstack (oldest at bottom):

Tested that data loading now have the expected behavior:

  • different dp ranks get different data
  • different tp ranks within the same dp rank get the same data

tianyu-l added a commit that referenced this pull request Feb 7, 2024
@tianyu-l
enable data loading for data parallel training
362b2c7 
ghstack-source-id: f15415d24f485fb71f08eb991d73b4921c17b484
Pull Request resolved: #49
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Feb 7, 2024
@tianyu-l tianyu-l linked an issue Feb 7, 2024 that may be closed by this pull request
alpaca dataset data parallel training #48
Closed
@tianyu-l tianyu-l requested a review from wanchaol February 7, 2024 23:29
wanchaol
wanchaol approved these changes Feb 7, 2024
View reviewed changes
Copy link
Contributor

@wanchaol wanchaol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good to me! that's a good way to unblock, have one comment

torchtrain/datasets/alpaca.py Show resolved Hide resolved
@tianyu-l
Update on "enable data loading for data parallel training"
f8a6e76 
Tested that data loading now have the expected behavior:
- different dp ranks get different data
- different tp ranks within the same dp rank get the same data 


[ghstack-poisoned]
tianyu-l added a commit that referenced this pull request Feb 8, 2024
@tianyu-l
enable data loading for data parallel training
8073c44 
ghstack-source-id: 08d335e3151097a273742be7cab615a75015d4dd
Pull Request resolved: #49
@tianyu-l tianyu-l merged commit f8a6e76 into gh/tianyu-l/1/base Feb 8, 2024
3 checks passed
tianyu-l added a commit that referenced this pull request Feb 8, 2024
@tianyu-l
enable data loading for data parallel training
e1b61c3 
ghstack-source-id: 08d335e3151097a273742be7cab615a75015d4dd
Pull Request resolved: #49
@tianyu-l tianyu-l deleted the gh/tianyu-l/1/head branch February 8, 2024 01:12
lessw2020 pushed a commit that referenced this pull request Apr 18, 2024
@tianyu-l
enable data loading for data parallel training
4f9eba6 
ghstack-source-id: 08d335e3151097a273742be7cab615a75015d4dd
Pull Request resolved: #49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wanchaol wanchaol wanchaol approved these changes

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

alpaca dataset data parallel training
3 participants
@tianyu-l @wanchaol @facebook-github-bot