Individuals with Autism Spectrum Disorder (ASD) typically present difficulties in engaging and interacting with their peers. Thus, researchers have been developing different technological solutions as support tools for children with ASD. Social robots, one example of these technological solutions, are often unaware of their game partners, preventing the automatic adaptation of their behavior to the user. Information that can be used to enrich this interaction and, consequently, adapt the system behavior is the recognition of different actions of the user by using RGB cameras or/and depth sensors. The present work proposes a method to automatically detect in real-time typical and stereotypical actions of children with ASD by using the Intel RealSense and the Nuitrack SDK to detect and extract the user joint coordinates. The pipeline starts by mapping the temporal and spatial joints dynamics onto a color image-based representation. Usually, the position of the joints in the final image is clustered into groups. In order to verify if the sequence of the joints in the final image representation can influence the model’s performance, two main experiments were conducted where in the first, the order of the grouped joints in the sequence was changed, and in the second, the joints were randomly ordered. In each experiment, statistical methods were used in the analysis. Based on the experiments conducted, it was found statistically significant differences concerning the joints sequence in the image, indicating that the order of the joints might impact the model’s performance. The final model, a Convolutional Neural Network (CNN), trained on the different actions (typical and stereotypical), was used to classify the different patterns of behavior, achieving a mean accuracy of 92.4% ± 0.0% on the test data. The entire pipeline ran on average at 31 FPS. ; This work has been supported by FCT—Fundação para a Ciência e Tecnologia within the R&D Units Project Scope: UIDB/00319/2020. Vinicius Silva thanks FCT for the PhD scholarship SFRH/BD/SFRH/BD/133314/2017.


    Access

    Download


    Export, share and cite



    Title :

    Skeleton driven action recognition using an image-based spatial-temporal representation and convolution neural network



    Publication date :

    2021-06-25


    Remarks:

    34201991



    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    Classification :

    DDC:    629




    A Novel Spatial-Temporal Graph for Skeleton-based Driver Action Recognition

    Li, Peng / Lu, Meiqi / Zhang, Zhiwei et al. | IEEE | 2019



    ST-Tracking: Spatial-temporal Graph Convolution Neural Network for Multi-object Tracking

    Xue, Yaqing / Luo, Guiyang / Yuan, Quan et al. | IEEE | 2021


    Bi-GRU-Attention Enhanced Unsupervised Network for Skeleton-Based Action Recognition

    Chen, Li / Ma, Nan / Zhang, Guoping | Springer Verlag | 2022


    Dual Dynamic Spatial-Temporal Graph Convolution Network for Traffic Prediction

    Sun, Yanfeng / Jiang, Xiangheng / Hu, Yongli et al. | IEEE | 2022