Copyright (c) 2020-2021 Yannic Kilcher (yk), Lucas Nestler (clashluke), Shawn Presser (shawwn), Jan (xmaster96)
First, create your VM through google cloud shell with ctpu up --vm-only. This way it has all the necessary permissions to connect to your Buckets and TPUs.
Next, install the requirements with pip on your VM using git clone https://github.com/tensorfork/obst && cd obst && python3 -m pip install -r requirements.txt.
Finally, start a TPU to kick off a training run using python3 main.py --model configs/big_ctx.json --tpu ${YOUR_TPU_NAME}.
- Mesh Tensorflow as machine learning library
- Intial code forked from Eleuther AI's GPT-Neo
We also want to explicitly thank
- tensorfork and TRC for providing us with the required compute (TPUs)
- Ben Wang (kindiana) and Shawn Presser for their invaluable knowledge about TensorFlow, TPU, and language models
- Gwern Branwen, Tri Songz and Aleph Alpha for financing our storage and servers