In this paper, we introduce a new deep learning architecture for camera and Lidar sensor fusion. The proposed scheme performs 2D object detection using the RGB camera image and the depth, height, and intensity images generated by projecting the 3D Lidar point cloud into camera image plane. The proposed object detector consists of two convolutional neural networks (CNNs) that process the RGB and Lidar images separately as well as the fusion network that combines the feature maps produced at the intermediate layers of the CNNs. We aim to develop a robust object detector that maintains good object detection accuracy even when the quality of the sensor signals is degraded for object detection. Towards this end, we devise the gated fusion unit (GFU) that adjusts the contribution of the feature maps generated by two CNN structures via gating mechanism. Using the GFU, the proposed object detector can fuse the high level feature maps drawn from two modalities with appropriate weights to achieve robust performance. Experiments conducted on the challenging KITTI benchmark show that the proposed camera and Lidar fusion network outperforms the conventional sensor fusion methods even when either of the camera and Lidar sensor signals is corrupted by missing data, occlusion, noise, and illumination change.
Robust Camera Lidar Sensor Fusion Via Deep Gated Information Fusion Network
2018 IEEE Intelligent Vehicles Symposium (IV) ; 1620-1625
2018-06-01
3840012 byte
Aufsatz (Konferenz)
Elektronische Ressource
Englisch
LIDAR-Camera Fusion Where LIDAR and Camera Validly See Different Things
Europäisches Patentamt | 2022
|Deep Learning-based Radar, Camera, and Lidar Fusion for Object Detection
TIBKAT | 2022
|Multi-Classifier Based LIDAR and Camera Fusion
IEEE | 2007
|