Highlights A novel multi-scale driver behavior modeling framework for driver physical and mental states recognition. A multi-task learning-oriented encoder-decoder framework is designed to fuse multi-modal signals. Joint modeling of driver intention and emotion with statistical analysis on the two cognitive process. Model recognition and real-time performance evaluation based on naturalistic driving dataset.

    Abstract The mutual understanding between driver and vehicle is critical to the realization of intelligent vehicles and customized interaction interface. In this study, a unified driver behavior modeling system toward multi-scale behavior recognition is proposed to enhance the driver behavior reasoning ability for intelligent vehicles. Specifically, the driver behavior recognition system is designed to simultaneously recognize the driver's physical and mental states based on a deep encoder-decoder framework. The model jointly learns to recognize three driver behaviors with different time scales: mirror checking and facial expression state, and two mental behaviors, including intention and emotion. The encoder module is designed based on a deep convolutional neural network (CNN) to capture spatial information from the input video stream. Then, several decoders for different driver states estimation are proposed with fully-connected (FC) and long short-term memory (LSTM) based recurrent neural networks (RNN). Two naturalistic datasets are used in this study to investigate the model performance, which is a local highway dataset, namely, CranData, and one public dataset from Brain4Cars. Based on the spatial–temporal representation of driver physical behavior, it shows that the observed physical behaviors can be used to model the latent mental behaviors through the proposed end-to-end learning process. The testing results on these two datasets show state-of-the-art results on mirror checking behavior, intention, and emotion recognition. With the proposed system, intelligent vehicles can gain a holistic understanding of the driver's physical and phycological behaviors to better collaborate and interact with the human driver, and the driver behavior reasoning system helps to reduce the conflicts between the human and vehicle automation.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Multi-scale driver behavior modeling based on deep spatial-temporal representation for intelligent vehicles


    Contributors:
    Xing, Yang (author) / Lv, Chen (author) / Cao, Dongpu (author) / Velenis, Efstathios (author)


    Publication date :

    2021-06-29




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English




    Driver Behavior Modeling in Critical Situations for Threat Assessment of Intelligent Vehicles

    Zhou, Huajian / Qin, Xiaohui / Wang, Xiaowei et al. | IEEE | 2019




    Examining Heterogeneity of Driver Behavior with Temporal and Spatial Factors

    Ellison, Adrian B. / Greaves, Stephen / Bliemer, Michiel | Transportation Research Record | 2013