Automatically recognising surgical gestures from surgical data is an important building block of automated activity recognition and analytics, technical skill assessment, intra-operative assistance and eventually robotic automation. The complexity of articulated instrument trajectories and the inherent variability due to surgical style and patient anatomy make analysis and fine-grained segmentation of surgical motion patterns from robot kinematics alone very difficult. Surgical video provides crucial information from the surgical site with context for the kinematic data and the interaction between the instruments and tissue. Yet sensor fusion between the robot data and surgical video stream is non-trivial because the data have different frequency, dimensions and discriminative capability. In this paper, we integrate multimodal attention mechanisms in a two-stream temporal convolutional network to compute relevance scores and weight kinematic and visual feature representations dynamically in time, aiming to aid multimodal network training and achieve effective sensor fusion. We report the results of our system on the JIGSAWS benchmark dataset and on a new in vivo dataset of suturing segments from robotic prostatectomy procedures. Our results are promising and obtain multimodal prediction sequences with higher accuracy and better temporal structure than corresponding unimodal solutions. Visualization of attention scores also gives physically interpretable insights on network understanding of strengths and weaknesses of each sensor.


    Access

    Download


    Export, share and cite



    Title :

    Gesture Recognition in Robotic Surgery with Multimodal Attention


    Contributors:
    Van Amsterdam, B (author) / Funke, I (author) / Edwards, E (author) / Speidel, S (author) / Collins, J (author) / Sridhar, A (author) / Kelly, J (author) / Clarkson, MJ (author) / Stoyanov, D (author)

    Publication date :

    2022-01-01


    Remarks:

    IEEE Transactions on Medical Imaging (2022) (In press).


    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    Classification :

    DDC:    629



    Robotic Control of Dynamic and Static Gesture Recognition

    Zhang, Xuexiang / Wu, Xuncheng | IEEE | 2019


    Lightweight wireless gesture recognition method based on attention enhancement

    LI XIAOFAN / YUAN JIAHUI / FU ZHENGJUN et al. | European Patent Office | 2024

    Free access



    Gesture Control of Robotic Arm

    Y., Dharshan / S., Vivek / S., Saranya et al. | BASE | 2017

    Free access