RT-1

This is the completion of google's rt-1 project code and can run directly.

You can view the google source code here: robotics_transformer

Using Method

1. 下载language-table数据集，详见下文Downloading the dataset，下载Universal Sentence Encoder模型

2. 通过language_table_data_reconstruction文件夹下代码对instrucion进行编码，原数据集为UTF-8编码格式

3. 运行前需要解决tensorflow版本兼容性问题，参见我的回答contrib_answer,运行distribute_train.py,保存模型

4. 使用language_table文件夹下代码进行测试，仿真环境详见language_table，整体流程如上图所示

Features

• Film efficient net based image tokenizer backbone
• Token learner based compression of input tokens
• Transformer for end to end robotic control
• Testing utilities

Getting Started

Downloading the dataset

RT-1 dataset: robotics_transformer_dataset

Language-table dataset: language_table_dataset

Both datasets are in RLDS format

• Clone the repository

cd Foundation-Models-for-Robots-main
git clone https://github.com/google-research/tensor2robot

• Install protobuf using pip

pip install protobuf
cd tensor2robot/proto

• Compile the protobuf file

protoc -I=./ --python_out=`pwd` t2r.proto
cd ../..

• Create a conda environment from the provided YAML file

conda env create -f tf-rt1_environment.yaml

while creating new enviroonment, the memory of /home disk may too small to load

• install others by pip in conda env without pip cache can solve it

conda activate tf-rt1
pip install --no-cache-dir -r piprequirments.txt -i https://pypi.tuna.tsinghua.edu.cn/simpl
python3 -m pip install tensorflow[and-cuda]
pip install --upgrade "jax[cuda11_pip]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html

• Run distributed code

python -m robotics_transformer.distribute_train

Using trained checkpoints

Checkpoints are included in trained_checkpoints/ folder for three models:

1. RT-1 trained on 700 tasks
2. RT-1 jointly trained on EDR and Kuka data
3. RT-1 jointly trained on sim and real data

They are tensorflow SavedModel files. Instructions on usage can be found here

Future Releases

The current repository includes an initial set of libraries for early adoption. More components may come in future releases.

License

The Robotics Transformer library is licensed under the terms of the Apache license.

Acknowledgements

Special thanks to these people for their help in this project:kpertsch

Contact

The project will continue to improve and update, if you have any questions about the use of this project or suggestions for modification, please contact us by email [email protected]