Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enable aggregate mem monitoring #3042

Merged
merged 11 commits into from
Feb 23, 2024
Merged

enable aggregate mem monitoring #3042

merged 11 commits into from
Feb 23, 2024

Conversation

vchiley
Copy link
Contributor

@vchiley vchiley commented Feb 21, 2024 •
edited
Loading

What does this PR do?

Enables memory monitor to aggregate stats across GPUs.
With dynamic graph execution, some GPUs may have more memory usage than others; this update allows the user to aggregate memory stats across GPUs.

What issue(s) does this change relate to?

Discussed offline
Local testing via: https://github.com/mosaicml/llm-foundry-private/pull/153

Before submitting

  • Have you read the contributor guidelines?
  • Is this change a documentation change or typo fix? If so, skip the rest of this checklist.
  • Was this change discussed/approved in a GitHub issue first? It is much more likely to be merged if so.
    • discussed offline
  • Did you update any related docs and document your change?
  • Did you update any related tests and add any new tests related to your change? (see testing)
  • Did you run the tests locally to make sure they pass?
  • Did you run pre-commit on your change? (see the pre-commit section of prerequisites)

mvpatel2000
mvpatel2000 reviewed Feb 21, 2024
View reviewed changes
Copy link
Contributor

@mvpatel2000 mvpatel2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add a unit test for this?

In particular I would write a distributed test where rank 0 creates a large tensor and then show rank 1 gets a higher mem usage

mvpatel2000
mvpatel2000 reviewed Feb 23, 2024
View reviewed changes
Copy link
Contributor

@mvpatel2000 mvpatel2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. One small bug and a request for a comment

composer/callbacks/memory_monitor.py Outdated Show resolved Hide resolved
composer/callbacks/memory_monitor.py Show resolved Hide resolved
@vchiley vchiley requested a review from mvpatel2000 February 23, 2024 18:08
@vchiley vchiley requested a review from mvpatel2000 February 23, 2024 19:01
mvpatel2000
mvpatel2000 approved these changes Feb 23, 2024
View reviewed changes
Copy link
Contributor

@mvpatel2000 mvpatel2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 🐈 🐒

@vchiley vchiley enabled auto-merge (squash) February 23, 2024 19:16
@vchiley vchiley merged commit a042759 into mosaicml:dev Feb 23, 2024
14 checks passed
j316chuck pushed a commit that referenced this pull request Mar 29, 2024
@vchiley @mvpatel2000 @j316chuck
enable aggregate mem monitoring (#3042)
e70c409 
* enable aggregate mem monitoring

* add test

* lint

* make more deterministic

* pr comments

* Update composer/callbacks/memory_monitor.py

Co-authored-by: Mihir Patel <[email protected]>

* updt doc str

---------

Co-authored-by: Mihir Patel <[email protected]>
j316chuck pushed a commit that referenced this pull request May 16, 2024
@vchiley @mvpatel2000
enable aggregate mem monitoring (#3042)
3144900 
* enable aggregate mem monitoring

* add test

* lint

* make more deterministic

* pr comments

* Update composer/callbacks/memory_monitor.py

Co-authored-by: Mihir Patel <[email protected]>

* updt doc str

---------

Co-authored-by: Mihir Patel <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mvpatel2000 mvpatel2000 mvpatel2000 approved these changes

@dakinggg dakinggg Awaiting requested review from dakinggg

@josejg josejg Awaiting requested review from josejg

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@vchiley @mvpatel2000