-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #7 from Computational-Plant-Science/v0.0.3
Release 0.0.3
- Loading branch information
Showing
23 changed files
with
762 additions
and
480 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,17 @@ | ||
| # Installation | ||
|
|
||
| ## Official release | ||
|
|
||
| `slappt` is [available on the Python Package Index](https://pypi.org/project/slappt/) and can be installed with `pip`: | ||
|
|
||
| ```shell | ||
| pip install slappt | ||
| ``` | ||
|
|
||
| ## Development version | ||
|
|
||
| The latest development version of `slappt` can be installed from GitHub: | ||
|
|
||
| ```shell | ||
| pip install git+https://github.com/Computational-Plant-Science/slappt.git | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,19 +1,64 @@ | ||
| # Quickstart | ||
|
|
||
| Say you're on a Slurm cluster with an active Python3.8+ environment and permission to submit to the `batch` partition. First make sure `slappt` is installed: | ||
| Say you have access to a Slurm cluster with `apptainer` installed, and you have permission to submit to the `batch` partition. | ||
|
|
||
| Copy the `hello.yaml` file from the `examples` directory to your current working directory, then run: | ||
|
|
||
| ```shell | ||
| pip install slappt | ||
| slappt hello.yaml > job.sh | ||
| ``` | ||
|
|
||
| Then we're off to the races: | ||
| Alternatively, without the configuration file: | ||
|
|
||
| ```shell | ||
| slappt --image docker://alpine \ | ||
| --shell sh \ | ||
| --partition batch \ | ||
| --entrypoint "echo 'hello world'" | ||
| --entrypoint "echo 'hello world'" > hello.sh | ||
| ``` | ||
|
|
||
| Your `hello.sh` script should now contain: | ||
|
|
||
| ```shell | ||
| #!/bin/bash | ||
| #SBATCH --job-name=0477f4b9-e119-4354-8384-f50d7a96adad | ||
| #SBATCH --output=slappt.0477f4b9-e119-4354-8384-f50d7a96adad.%j.out | ||
| #SBATCH --error=slappt.0477f4b9-e119-4354-8384-f50d7a96adad.%j.err | ||
| #SBATCH --partition=batch | ||
| #SBATCH -c 1 | ||
| #SBATCH -N 1 | ||
| #SBATCH --ntasks=1 | ||
| #SBATCH --time=01:00:00 | ||
| #SBATCH --mem=1GB | ||
| module load apptainer # only if you need, some clusters | ||
| apptainer exec docker://alpine sh -c "echo 'hello world'" | ||
| ``` | ||
|
|
||
| **Note:** for most image definitions, specifying the `shell` is likely not necessary — the default is `bash`. However, for images that don't have `bash` installed (`alphine` only has `sh`) you'll need to specify a different shell. | ||
| If already on the cluster, use the `--submit` flag to submit the job directly. (Standard Slurm commands must be available for this to work.) In this case the job ID is shown if submission was successful. | ||
|
|
||
| You can provide authentication information to submit the script to remote clusters over SSH. For instance, assuming you have key authentication set up and your key is `~/.ssh/id_rsa`: | ||
|
|
||
| ```shell | ||
| slappt ... --host <cluster IP or FQDN> --username <username> | ||
| ``` | ||
|
|
||
| ## Caveats | ||
|
|
||
| There are a few things to note about the example above. | ||
|
|
||
| ### Shell | ||
|
|
||
| For most image definitions, specifying the `shell` is likely not necessary, as the default is `bash`. However, for images that don't have `bash` installed (e.g., `alphine` only has `sh`) a different shell must be selected. | ||
|
|
||
| ### Singularity support | ||
|
|
||
| If your cluster still uses `singularity`, pass the `--singularity` flag (or set the `singularity` key in the configuration file to `true`) to substitute `singularity` for `apptainer` in the command wrapping your workflow entrypoint. | ||
|
|
||
| ### Pre-commands | ||
|
|
||
| **Note:** if `apptainer` or `singularity` are not available by default on your cluster's compute nodes, you may need to add `--pre` commands (or a `pre` section to the configuration file), for instance `--pre "module load apptainer"`, or: | ||
|
|
||
| ```yaml | ||
| pre: | ||
| - module load apptainer | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| # YAML Specification | ||
|
|
||
| `slappt` supports declarative YAML configuration to make container workflows reusable. | ||
|
|
||
| ```yaml | ||
| # standard attributes | ||
| image: # the container image definition to use, e.g. docker://alpine (registry prefix is required) | ||
| shell: # the shell to use (default: bash) | ||
| partition: # the cluster partition to submit to | ||
| entrypoint: # the command to run inside the container | ||
| workdir: # the working directory to use | ||
| email: # the email address to send notifications to | ||
| name: # the name of the job (default: slappt.<guid>) | ||
| pre: # a list of commands to run before invoking the container (e.g. loading modules) | ||
| inputs: # a text file containing a newline-separated list of input files | ||
| environment: # a dictionary of environment variables to set | ||
| bind_mounts: # a list of bind mounts to use, in format <host path>:<container path> | ||
| no_cache: # don't use the apptainer/singularity cache, force a rebuild of the image (default: false) | ||
| gpus: # the number of GPUs to request | ||
| time: # the job's walltime | ||
| account: # the account name to associate the job with | ||
| mem: # the amount of memory to request (default: 1GB) | ||
| nodes: # the number of nodes to request (default: 1) | ||
| cores: # the number of cores to request (default: 1) | ||
| tasks: # the number of tasks to request (default: 1) | ||
| header_skip: # a list of header lines to skip when parsing the input file (can be useful e.g. for clusters which have virtual memory and reject --mem headers) | ||
| singularity: # whether to invoke singularity instead of apptainer (default: false) | ||
| # submission attributes | ||
| host: # the hostname, IP or FQDN of the remote cluster to submit to | ||
| port: # the port to use for the SSH connection (default: 22) | ||
| username: # the username to use for the SSH connection | ||
| password: # the password to use for SSH authentication | ||
| pkey: # the path to the private key to use for SSH authentication | ||
| allow_stderr: # don't raise an error if sshlurm encounters stderr output (default: false) | ||
| timeout: # the timeout for the SSH connection (default: 10) | ||
| ``` |
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.