Driver fatigue detection is one of the crucial methods to ensure driving safety. This paper proposes a driver fatigue detection model through multimodal fusion based on convolutional neural networks (CNN) and gated transformer networks (GTN). Specifically, the driver’s facial features are extracted using CNN and combined with the driver’s electrocardiogram (ECG) and vehicle state features to form multivariate time series. Then, the GTN employs self-attention and masking mechanisms to capture the channel-wise and step-wise correlations of the input, fully leveraging the advantages of complementarity among different modalities. To verify the validity of the proposed model, extensive experiments are conducted on our dataset collected from 15 participants using a driving simulator. Experimental results demonstrate that the fusion of three modalities achieves higher accuracy than a single modality. The trimodal model achieved an average prediction accuracy of 90% across the four fatigue levels. Therefore, the multimodal-based driver fatigue detection method is feasible for autonomous vehicles (AVs).


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Multimodal Features Fusion for Driver Fatigue Detection Based on CNN-GTN Learning


    Contributors:
    Zhao, Shuaijie (author) / Du, Aimin (author) / Han, Yeyang (author) / Yi, Jiwei (author) / Zhu, Zhongpan (author)


    Publication date :

    2023-10-27


    Size :

    4222712 byte




    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    Multimodal Features for Detection of Driver Stress and Fatigue: Review

    Nemcova, Andrea / Svozilova, Veronika / Bucsuhazy, Katerina et al. | IEEE | 2021


    Multimodal Detection of Driver Distraction

    M. Eskenazi / L. Morency / A. W. Black et al. | NTIS | 2017


    Driver Fatigue Detection

    Abdullah, Muhamad Hafiz / Raman, Kirbana Jai / Azman, Afizan et al. | Springer Verlag | 2016


    Driver Drowsiness Behavior Detection and Analysis Using Vision-Based Multimodal Features for Driving Safety

    Gopinath, Aditya / Li, Bing / Brand, Howard et al. | SAE Technical Papers | 2020


    Driver Drowsiness Behavior Detection and Analysis Using Vision-Based Multimodal Features for Driving Safety

    Li, Rui / Brand, Howard / Gopinath, Aditya et al. | British Library Conference Proceedings | 2020