Multimodality fusion based on deep neural networks (DNN) is a significant method for intelligent vehicles. The special characteristics of DNN lead to the issue of AI safety and safety test. In this paper, we firstly propose a multimodality fusion framework called Integrated Multimodality Fusion Deep Neural Network (IMF-DNN), which can flexibly accomplish both object detection and end-to-end driving policy for prediction of steering angle and speed. Then, we propose a DNN safety test strategy, which systematically analyzes DNN's robustness and generalization ability in large amounts of diverse driving environment conditions. The test in this paper is based on our IMF-DNN model and the strategy can be widely used for other DNNs. Finally, the experiment analysis is performed on KITTI for object detection and the dateset DBNet for end-to-end tasks. The results show the superior accuracy of the proposed IMF-DNN model and the test strategy's potential ability to improve the robustness and generalization of autonomous vehicle deep learning model. Code is available at https://github.com/ennisnie/IMF-DNN
A Multimodality Fusion Deep Neural Network and Safety Test Strategy for Intelligent Vehicles
IEEE Transactions on Intelligent Vehicles ; 6 , 2 ; 310-322
2021-06-01
4937884 byte
Article (Journal)
Electronic Resource
English
Transportation Research Record | 2007
|Experiments in Multimodality Image Classification and Data Fusion
British Library Conference Proceedings | 2002
|Network connectivity and multimodality: the context
Taylor & Francis Verlag | 2000
|