Multimodality fusion based on deep neural networks (DNN) is a significant method for intelligent vehicles. The special characteristics of DNN lead to the issue of AI safety and safety test. In this paper, we firstly propose a multimodality fusion framework called Integrated Multimodality Fusion Deep Neural Network (IMF-DNN), which can flexibly accomplish both object detection and end-to-end driving policy for prediction of steering angle and speed. Then, we propose a DNN safety test strategy, which systematically analyzes DNN's robustness and generalization ability in large amounts of diverse driving environment conditions. The test in this paper is based on our IMF-DNN model and the strategy can be widely used for other DNNs. Finally, the experiment analysis is performed on KITTI for object detection and the dateset DBNet for end-to-end tasks. The results show the superior accuracy of the proposed IMF-DNN model and the test strategy's potential ability to improve the robustness and generalization of autonomous vehicle deep learning model. Code is available at https://github.com/ennisnie/IMF-DNN


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    A Multimodality Fusion Deep Neural Network and Safety Test Strategy for Intelligent Vehicles


    Contributors:
    Nie, Jian (author) / Yan, Jun (author) / Yin, Huilin (author) / Ren, Lei (author) / Meng, Qian (author)

    Published in:

    Publication date :

    2021-06-01


    Size :

    4937884 byte




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    Experiments in multimodality image classification and data fusion

    Farag, A.A. / Mohamed, R. / Mahdi, H. | IEEE | 2002


    Multimodality

    Nobis, Claudia | Transportation Research Record | 2007


    Experiments in Multimodality Image Classification and Data Fusion

    Farag, A. / Mahdi, H. / Mohamed, R. et al. | British Library Conference Proceedings | 2002


    Supervised Hash Coding With Deep Neural Network for Environment Perception of Intelligent Vehicles

    Yan, Chenggang / Xie, Hongtao / Yang, Dongbao et al. | IEEE | 2018


    Network connectivity and multimodality: the context

    Nijkamp, Peter / Priemus, Hugo / Shefer, Dani | Taylor & Francis Verlag | 2000