Skip to content

Commit

Permalink
modify Distributed_Configuration.md because the link about pipeline p…
Browse files Browse the repository at this point in the history
…arallel is false (#547)
  • Loading branch information
xiaohoua authored Aug 29, 2024
1 parent 4a57a74 commit b6c00ef
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/source/tutorials/basics/Distributed_Configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ train.dist.pipeline_num_layers = model.cfg.hidden_layers
- `train.dist.pipeline_num_layers` must be set consistent with the model layers. If unset, it will use the default value `1000`,
which might trigger unexpected behavior.

- For models which have been configured with pipeline parallelism(e.g., BERT, GPT-2, T5 and ViT), you can simply update the distributed config to execute pipeline parallel training on them. If you need to train your own model with pipeline parallel strategy, please refer to [Write Models](https://libai.readthedocs.io/en/latest/tutorials/basics/Write_Models.html) for more details about configuring your own model with pipeline parallelism.
- For models which have been configured with pipeline parallelism(e.g., BERT, GPT-2, T5 and ViT), you can simply update the distributed config to execute pipeline parallel training on them. If you need to train your own model with pipeline parallel strategy, please refer to [Customize Parallelism](https://libai.readthedocs.io/en/latest/tutorials/advanced_tutorials/customize_parallel.html#write-your-own-pipeline-parallel-model) for more details about configuring your own model with pipeline parallelism.

#### **Data Parallel + Tensor Parallel for 2D Parallel Training on 8 GPUs**

Expand Down

0 comments on commit b6c00ef

Please sign in to comment.