Awesome Radar-Camera Fusion

Overview

• Surveys
• Datasets
• Methods
• Radar-Camera
  • Detection
  • Tracking
  • Segmentation
  • Depth Estimation
  • Odometry
  • Multi-Task
• Radar-LiDAR
  • Detection
• Camera-LiDAR
  • Detection
• Camera-LiDAR-Radar
  • Detection
• Citation

Surveys

• 2023 - Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review TIV [Paper] [Website] [GitHub]
• 2021 - Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges TITS [Paper] [Website]
• 2023 - Vision-RADAR fusion for Robotics BEV Detections: A Survey IV [Paper]
• 2023 - Radar Perception in Autonomous Driving: Exploring Different Data Representations arXiv [Paper] [Website] [GitHub]
• 2024 - Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Survey arXiv [Paper]

Datasets

Id	Name	Year	Task	Annotation	Radar Data Representation	Link
1	nuScenes	2019	Object Detection Object Tracking	3D box	Point Cloud	Website Github
2	Astyx	2019	Object Detection	3D box	Point Cloud	Website
3	SeeingThroughFog	2020	Object Detection	2D box 3D box	Point Cloud	Website
4	CARRADA	2020	Object Detection Semantic Segmentation Object Tracking Trajectory Prediction	2D box 2D pixel	Range-Doppler Tensor Range-Azimuth Tensor	Website
5	HawkEye	2020	Semantic Segmentation	3D point	Point Cloud	Website
6	Zendar	2020	Object Detection Mapping Localization	2D box	Range-Doppler Tensor Range-Azimuth Tensor Point Cloud	Website
7	RADIATE	2020	Object Detection Object Tracking SLAM Scene Understanding	2D box	Range-Azimuth Tensor	Website
8	AIODrive	2020	Object Detection Object Tracking Semantic Segmentation Trajectory Prediction Depth Estimation	2D box 3D box	Point Cloud	Website
9	CRUW	2021	Object Detection	2D box	Range-Azimuth Tensor	Website
10	RaDICaL	2021	Object Detection	2D box	ADC Signal	Website
11	RadarScenes	2021	Object Detection Semantic Segmentation Object Tracking	2D pixel 3D point	Point Cloud	Website
12	RADDet	2021	Object Detection	2D box 3D box	Range-Azimuth-Doppler Tensor	Github
13	FloW	2021	Object Detection	2D box	Range-Doppler Tensor Point Cloud	Website Github
14	RADIal	2021	Object Detection Semantic Segmentation	2D box	ADC Signal Range-Azimuth-Doppler Tensor Range-Azimuth Tensor Range-Doppler Tensor Point Cloud	Github
15	VoD	2022	Object Detection	2D box 3D box	4D Point Cloud	Website
16	Boreas	2022	Object Detection Localization Odometry	2D box	Range-Azimuth Tensor	Website
17	TJ4DRadSet	2022	Object Detection Object Tracking	3D box	4D Point Cloud	Website
18	K-Radar	2022	Object Detection Object Tracking SLAM	3D box	4D Range-Azimuth-Doppler Tensor	Github
19	aiMotive	2022	Object Detection	3D box	Point cloud	Website
20	WaterScenes	2023	Instance Segmentation Semantic Segmentation Free-space Segmentation Waterline Segmentation Panoptic Perception	2D box 2D pixel 2D line 3D point	4D Point cloud	Paper Website GitHub
21	ThermRad	2023	Object Detection	3D box	4D Point Cloud	Paper
22	Dual Radar	2023	Object Detection Object Tracking	3D box	4D Point Cloud	Paper GitHub

Methods

Id	Short Name	Year	Task	Annotation	Radar Data Representation	Fusion Level	Dataset	Link
1	Chadwick et al.	2019	Object Detection	2D box	Point Cloud	Feature Level	Self-Recorded
2	RRPN	2019	Object Detection	2D box	Point Cloud	Data Level	nuScenes	Code
3	Jha et al.	2019	Object Detection	2D box	Point Cloud	Object Level	Self-Recorded
4	CMGGAN	2019	Semantic Segmentation	2D point	Grid Map	Feature Level	Self-Recorded
5	Meyer and Kuschk	2019	Object Detection	3D box	Point Cloud	Data Level	Astyx
6	RVNet	2019	Object Detection	2D box	Point Cloud	Feature Level	nuScenes
7	FusionNet	2019	Object Detection Object Classification	2D box	Range-Azimuth Tensor	Feature Level	Self-Recorded
8	SO-Net	2020	Object Detection Semantic Segmentation	2D box 2D pixel	Point Cloud	Feature Level	nuScenes
9	SAF-FCOS	2020	Object Detection	2D box	Point Cloud	Feature Level	nuScenes	Code
10	CRF-Net	2019	Object Detection	2D box	Point Cloud	Data Level	nuScenes Self-Recorded	Code
11	Bijelic et al.	2020	Object Detection	2D box	Point Cloud	Feature Level	DENSE	Code
12	BIRANet	2020	Object Detection	2D box	Point Cloud	Feature Level	nuScenes	Code
13	Nabati and Qi	2020	Object Detection Depth Estimation	2D box	Point Cloud	Mixed Level	nuScenes
14	YOdar	2020	Object Detection	2D box	Point Cloud	Feature Level	nuScenes
15	CenterNet	2020	Object Detection	3D box	Point Cloud	Feature Level	nuScenes	Code
16	RODNet	2020	Object Detection	2D box	Range-Azimuth Tensor	Feature Level	CRUW	Code
17	RAMP-CNN	2021	Object Detection	2D box	Range-Azimuth-Doppler Tensor	Feature Level	CRUW
18	Li and Xie	2021	Object Detection	3D box	Point Cloud	Feature Level	nuScenes
19	Kim et al.	2020	Object Detection	3D box	Range-Azimuth Tensor	Feature Level	Self-Recorded
20	AssociationNet	2021	Object Detection	2D box	Point Cloud	Object Level	Self-Recorded
21	RVF-Net	2021	Object Detection	3D box	Point Cloud	Data Level	nuScenes
22	Cui et al.	2021	Object Detection	3D box	Point Cloud	Mixed Level	Astyx
23	RISFNet	2021	Object Detection	2D box	Point Cloud	Feature Level	FloW
24	GRIF Net	2021	Object Detection	3D box	Point Cloud	Feature Level	nuScenes
25	Stacker et al.	2021	Object Detection	2D box	Point Cloud	Feature Level	nuScenes
26	Harley et al.	2021	Semantic Segmentation	2D pixel	Point Cloud	Feature Level	nuScenes
27	RadSegNet	2022	Object Detection	2D box 2D pixel	Point Cloud Range-Azimuth Tensor	Data	Astyx RADIATE
28	RCBEV	2022	Object Detection	3D box	Point Cloud	Feature Level	nuScenes
29	CRAFT	2022	Object Detection	3D box	Point Cloud	Data Level	nuScenes
30	DeepFusion	2022	Object Detection	3D box	Point Cloud	Feature Level	Self-reorded nuScenes
31	CramNet	2022	Object Detection	3D box	Range-Azimuth Tensor	Feature Level	RADIATE
32	MVFusion	2023	Object Detection	3D box	Point Cloud	Feature Level	nuScenes
33	CRN	2023	Object Detection	3D box	Point Cloud	Feature Level	nuScenes

Radar-Camera

Detection

• 2022 - Detecting Darting Out Pedestrians With Occlusion Aware Sensor Fusion of Radar and Stereo Camera TIV [Paper]
• 2023 - RCFusion: Fusing 4-D Radar and Camera With Bird’s-Eye View Features for 3-D Object Detection [VoD TJ4DRadSet] TIM [Paper]
• 2023 - LXL: LiDAR Exclusive Lean 3D Object Detection with 4D Imaging Radar and Camera Fusion [VoD TJ4DRadSet] TIV [Paper]
• 2023 - REDFormer: Radar Enlightens the Darkness of Camera Perception with Transformers [nuScenes] TIV [Paper] [Code]
• 2023 - SparseFusion3D: Sparse Sensor Fusion for 3D object detection by Radar and Camera in Environmental Perception [nuScenes] TIV [Paper]
• 2023 - CRN: Camera Radar Net for Accurate, Robust, Efficient 3D Perception [nuScenes] ICCV [Paper] [Code]
• 2023 - RADIANT: Radar-Image Association Network for 3D Object Detection [nuScenes] AAAI [Paper] [Code]
• 2024 - RCBEVDet: Radar-camera Fusion in Bird’s Eye View for 3D Object Detection [nuScenes] CVPR [Paper] [Code]
• 2024 - CRKD: Enhanced Camera-Radar Object Detection with Cross-modality Knowledge Distillation [nuScenes] CVPR [Paper]
• 2024 - RCBEVDet++: Toward High-accuracy Radar-Camera Fusion 3D Perception Network [nuScenes] arXiv [Paper] [Code]

Tracking

• 2016 - On-Road Vehicle Detection and Tracking Using MMW Radar and Monovision Fusion TITS [Paper]
• 2019 - Target Detection Algorithm Based on MMW Radar and Camera Fusion ITSC [Paper]
• 2021 - A Novel Spatio-Temporal Synchronization Method of Roadside Asynchronous MMW Radar-Camera for Sensor Fusion TITS [Paper]
• 2021 - Robust Detection and Tracking Method for Moving Object Based on Radar and Camera Data Fusion IEEE Sensors [Paper]
• 2021 - CFTrack: Center-based Radar and Camera Fusion for 3D Multi-Object Tracking IV Workshops [Paper]
• 2021 - 3D Detection and Tracking for On-road Vehicles with a Monovision Camera and Dual Low-cost 4D mmWave Radars ITSC [Paper]
• 2022 - Robust Target Recognition and Tracking of Self-Driving Cars With Radar and Camera Information Fusion Under Severe Weather Conditions TITS [Paper]
• 2022 - 3D Multiple Object Tracking with Multi-modal Fusion of Low-cost Sensors for Autonomous Driving ITSC [Paper]
• 2022 - Robust multiobject tracking using mmwave radar-camera sensor fusion IEEE Sensors Letters [Paper]
• 2023 - CRN: Camera Radar Net for Accurate, Robust, Efficient 3D Perception [nuScenes] ICCV [Paper] [Code]
• 2023 - Online Multi-pedestrian Tracking based on Fused Detections of Millimeter Wave Radar and Vision IEEE Sensors Journal [Paper]
• 2023 - Asynchronous Information Fusion in Intelligent Driving Systems for Target Tracking Using Cameras and Radars TIE [Paper]
• 2024 - Deep Learning-Based Robust Multi-Object Tracking via Fusion of mmWave Radar and Camera Sensors TITS [Paper]

Segmentation

• 2024 - BEVCar: Camera-Radar Fusion for BEV Map and Object Segmentation [nuScenes] CVPR [Paper] [Website] [Code]

Depth Estimation

• 2023 - Depth Estimation From Camera Image and mmWave Radar Point Cloud [nuScenes] CVPR [Paper] [Code]

Odometry

• 2023 - 4DRVO-Net: Deep 4D Radar–Visual Odometry Using Multi-Modal and Multi-Scale Adaptive Fusion [VoD] TIV [Paper]

Multi-Task

• 2023 - Achelous: A Fast Unified Water-surface Panoptic Perception Framework based on Fusion of Monocular Camera and 4D mmWave Radar [WaterScenes] ITSC [Paper] [GitHub]
• 2023 - Mask-VRDet: Mask-VRDet: A robust riverway panoptic perception model based on dual graph fusion of vision and 4D mmWave radar[WaterScenes] RAS [Paper] [GitHub]
• 2023 - Efficient-VRNet: An Exquisite Fusion Network for Riverway Panoptic Perception based on Asymmetric Fair Fusion of Vision and 4D mmWave Radar [WaterScenes] arXiv [Paper] [GitHub]
• 2023 - CRN: Camera Radar Net for Accurate, Robust, Efficient 3D Perception [nuScenes] ICCV [Paper] [Code]

Radar-LiDAR

Detection

• 2021 - Robust multimodal vehicle detection in foggy weather using complementary lidar and radar signals CVPR [Paper]
• 2022 - RaLiBEV: Radar and LiDAR BEV Fusion Learning for Anchor Box Free Object Detection System [Paper]
• 2022 - Modality-agnostic learning for radar-lidar fusion in vehicle detection CVPR [Paper]
• 2022 - [ST-MVDNet] Modality-Agnostic Learning for Radar-Lidar Fusion in Vehicle Detection [Oxford Radar Robotcar]CVPR [Paper]
• 2023 - Bi-LRFusion: Bi-Directional LiDAR-Radar Fusion for 3D Dynamic Object Detection CVPR [Paper]
• 2023 - Multi-Modal and Multi-Scale Fusion 3D Object Detection of 4D Radar and LiDAR for Autonomous Driving [Astyx] TVT [Paper]
• 2023 - ST-MVDNet++: Improve vehicle detection with lidar-radar geometrical augmentation via self-training ICASSP [Paper]
• 2023 - ACF-Net: Asymmetric Cascade Fusion for 3D Detection With LiDAR Point Clouds and Images TIV [Paper]

Camera-LiDAR

Detection

• 2021 - Joint Multi-Object Detection and Tracking with Camera-LiDAR Fusion for Autonomous Driving IROS [Paper]
• 2023 - RoboBEV: Towards Robust Bird’s Eye View Perception under Corruptions [nuScenes] [Paper Code]

Camera-LiDAR-Radar

Detection

• 2023 - [BEV-Guide] BEV-Guided Multi-Modality Fusion for Driving Perception CVPR [Paper]
• 2023 - DeepFusion: A Robust and Modular 3D Object Detector for Lidars, Cameras and Radars IROS [Paper]

Citation

Please use the following citation when referencing

@article{yao2023radar,
  author={Yao, Shanliang and Guan, Runwei and Huang, Xiaoyu and Li, Zhuoxiao and Sha, Xiangyu and Yue, Yong and Lim, Eng Gee and Seo, Hyungjoon and Man, Ka Lok and Zhu, Xiaohui and Yue, Yutao},
  journal={IEEE Transactions on Intelligent Vehicles}, 
  title={Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review}, 
  year={2024},
  volume={9},
  number={1},
  pages={2094-2128},
  doi={10.1109/TIV.2023.3307157}}