add pp validation for schedule #568
Conversation
[ghstack-poisoned]
facebook-github-bot
added
the
CLA Signed
| @@ -41,6 +41,18 @@ def build_pipeline_schedule(job_config, stages, loss_fn): | |||
| if n_microbatches is None: | |||
| n_microbatches = job_config.experimental.pipeline_parallel_degree | |||
|
|
|||
| # Validation that the stages are compatible with the schedule | |||
| if isinstance(schedule_class, PipelineScheduleSingle): | |||
There was a problem hiding this comment.
hm, shouldn't this validation actually go inside of PipelineSchedule* __init__ functions?
There was a problem hiding this comment.
Hm yeah for PipelineScheduleMulti init we do check that len(stages) > 2. I guess the issue I ran into was because of this line below:
I accidentally commented out the schedule config so it defaulted to a single stage schedule, but still had the stages cut such that it was a "looped schedule". Since in the line above, only stage[0] is retrieved the schedule was created correctly, but later in runtime caused a hang.
There was a problem hiding this comment.
If we wanted to be more consistent then we could just support passing in a list of stages for both single and multi, then all the validation could be done in pytorch
I ran into this issue with config where I was using the wrong schedule so hopefully an error message can prevent this for users as well [ghstack-poisoned]
I ran into this issue with config where I was using the wrong schedule so hopefully an error message can prevent this for users as well [ghstack-poisoned]
Stack from ghstack (oldest at bottom):
I ran into this issue with config where I was using the wrong schedule so hopefully an error message can prevent this for users as well