-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add ability to submit multiple sequential worker jobs.
By setting a Run parameter of sequential_workers we will submit a series of worker jobs where each depends on the previous job's completion. This is only really useful if each job persists the current state of the overall workflow so the next job can pick up where the prevous job left off.
- Loading branch information
1 parent
ca3ded0
commit b0aaec3
Showing
6 changed files
with
69 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
python_env: python_env.yaml | ||
|
||
entry_points: | ||
main: | ||
command: "./demo.sh" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
# Example of Sequential Worker Jobs | ||
This simple example shows how to use the `sequential_workers` parameter to submit a job that will be split into multiple jobs that depend on each other. | ||
|
||
```shell | ||
mlflow run --backend slurm -c ../../slurm_config.json -P sequential_workers=3 . | ||
``` | ||
|
||
Each job appends to a file named `restart.log` with the time the job is run. | ||
MLFlow will submit three jobs that depend on each other. As soon as the first job terminates, the next job will start. This will continue until all jobs have completed. | ||
|
||
When the jobs are complete, you can check the `restart.log` file to see the order in which the jobs were run. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
#!/bin/bash | ||
echo "Hello World" | ||
sleep 2 | ||
date >> restart.log | ||
exit 0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
# Python version required to run the project. | ||
python: "3.10" | ||
# Dependencies required to run the project. | ||
dependencies: | ||
- mlflow |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters