Automatically recognising surgical gestures from surgical data is an important building block of automated activity recognition and analytics, technical skill assessment, intra-operative assistance and eventually robotic automation. The complexity of articulated instrument trajectories and the inherent variability due to surgical style and patient anatomy make analysis and fine-grained segmentation of surgical motion patterns from robot kinematics alone very difficult. Surgical video provides crucial information from the surgical site with context for the kinematic data and the interaction between the instruments and tissue. Yet sensor fusion between the robot data and surgical video stream is non-trivial because the data have different frequency, dimensions and discriminative capability. In this paper, we integrate multimodal attention mechanisms in a two-stream temporal convolutional network to compute relevance scores and weight kinematic and visual feature representations dynamically in time, aiming to aid multimodal network training and achieve effective sensor fusion. We report the results of our system on the JIGSAWS benchmark dataset and on a new in vivo dataset of suturing segments from robotic prostatectomy procedures. Our results are promising and obtain multimodal prediction sequences with higher accuracy and better temporal structure than corresponding unimodal solutions. Visualization of attention scores also gives physically interpretable insights on network understanding of strengths and weaknesses of each sensor.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    Gesture Recognition in Robotic Surgery with Multimodal Attention


    Beteiligte:
    Van Amsterdam, B (Autor:in) / Funke, I (Autor:in) / Edwards, E (Autor:in) / Speidel, S (Autor:in) / Collins, J (Autor:in) / Sridhar, A (Autor:in) / Kelly, J (Autor:in) / Clarkson, MJ (Autor:in) / Stoyanov, D (Autor:in)

    Erscheinungsdatum :

    01.01.2022


    Anmerkungen:

    IEEE Transactions on Medical Imaging (2022) (In press).


    Medientyp :

    Aufsatz (Zeitschrift)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    Klassifikation :

    DDC:    629



    Robotic Control of Dynamic and Static Gesture Recognition

    Zhang, Xuexiang / Wu, Xuncheng | IEEE | 2019


    Lightweight wireless gesture recognition method based on attention enhancement

    LI XIAOFAN / YUAN JIAHUI / FU ZHENGJUN et al. | Europäisches Patentamt | 2024

    Freier Zugriff



    GESTURE RECOGNITION DEVICE AND GESTURE RECOGNITION METHOD

    HIGUCHI DAIKI / OHASHI DAISUKE | Europäisches Patentamt | 2022

    Freier Zugriff