Skip to content

Latest commit

 

History

History
114 lines (98 loc) · 2.64 KB

slurm.md

File metadata and controls

114 lines (98 loc) · 2.64 KB
Raw

Connect

  1. Connect to TAU vpn: https://computing.tau.ac.il/helpdesk/remote-access/communication/vpn
  2. Connect to the login server (used to submit jobs to the worker nodes) Start with:
ssh <user-name>@c-002.cs.tau.ac.il

And enter your password.

  1. Navigate to your course folder
cd /home/joberant/NLP_2324b/<user-name>

Not the first time?

Initialize the conda environment:

bash
conda activate llm2vec

And start working!

Useful commands

Submit a job

sbatch mntp.slurm

Examine all jobs for user

sacct -u <user-name>

Check status of queue

squeue --job -u <user-name>

First time setup

Just on first time

  1. Create cache folders (Just on first time)
mkdir cache
mkdir llm2vec
mkdir cache/hf_cache
mkdir cache/transformers_cache
mkdir cache/hf_dataset_cache
mkdir cache/torch_cache
  1. Install Anaconda and create a virtual environment
wget repo.anaconda.com/archive/Anaconda3-2020.11-Linux-x86_64.sh

This will download a script called Anaconda3-2020.11-Linux-x86_64.sh. Now run it, and press enter until you need to accept the license by typing "yes"

sh Anaconda3-2020.11-Linux-x86_64.sh

Do not user the default location, instead:

/home/joberant/NLP_2324b/<user-name>/anaconda3

After (the long) installation, when you're asked: Do you wish the installer to initialize Anaconda3 by running conda init? just enter yes.

  1. Create virtual environment First, use bash to activate the base virtual environment:
bash

And create a new virtual environment:

conda create -n llm2vec python=3.9

And now you can activate it using:

conda activate llm2vec
conda install pip
  1. Copy the project into slurm

The general command to copy files is:

rsync -vrah <path-on-local-machine> <user-name>@c-002.cs.tau.ac.il:<path-on-slurm> --stats --progress

Now, lets copy the entire project excluding our virtual env. Make sure you run this command on your local machine from the projects' base directory.

On Windows:

scp -r experiments llm2vec nlp_course scripts test_configs train_configs .env README.md setup.cfg setup.py <user-name>@c-002.cs.tau.ac.il:/home/joberant/NLP_2324b/<user-name>/llm2vec

On Mac:

rsync -vrah experiments llm2vec nlp_course scripts test_configs train_configs .env README.md setup.cfg setup.py <user-name>@c-002.cs.tau.ac.il:/home/joberant/NLP_2324b/<user-name>/llm2vec --stats --progress
  1. install required packages First make sure you're using the created virtual environment (llm2vec). If not:
bash
conda activate llm2vec

Navigate to the llm2vec folder and run:

./scripts/install.sh