Semantic predictive control (SPC) is a policy learning framework that predicts future semantic segmentation and events by aggregating multi-scale feature maps. It utilizes dense supervision from semantic segmentation for feature learning and greatly improves policy learning efficiency. The learned features are explainable as they depict future scenes with semantic segmentation and explicit events.
This repository contains a PyTorch implementation of SPC, as well as some training scripts to reproduce policy learning results reported in our paper.
Our model is composed of four sub-modules:- The feature extraction module extracts multi-scale intermediate features from RGB observations;
- The extracted features are then concatenated with tiled actions and feed to the multi-scale prediction module that sequentially predicts future features;
- The information prediction module takes in the predicted latent feature representation and outputs corresponding future frame semantic segmentation and task-related signals, such as collision, off-road, and speed;
- The guidance network module that predicts action distribution for efficient sampling-based optimization.
Our results in the paper can be reproduced with the provided scripts by running
cd scripts/
bash train_#ENVNAME.sh