This paper presents a mixed reality system that, using the sensors mounted on the Microsoft Hololens headset and a cloud service, acquires and processes in real-time data to detect and track different kinds of objects and finally superimposes geographically coherent holographic texts on the detected objects. Such a goal has been achieved dealing with the intrinsic headset hardware limitations, by performing part of the overall computation in a edge/cloud environment. In particular, the heavier object detection algorithms, based on Deep Neural Networks (DNNs), are executed in the cloud. At the same time we compensate for cloud transmission and computation latencies by running light scene detection and object tracking on board the headset. The proposed pipeline allows meeting the real-time constraint by exploiting at the same time the power of state of art DNNs and the potential of Microsoft Hololens. This paper presents the design choices and describes the original algorithmic steps we devised to achieve real time tracking in mixed reality. Finally, the proposed system is experimentally validated.


    Access

    Download


    Export, share and cite



    Title :

    Real-time object detection and tracking in mixed reality using Microsoft HoloLens


    Contributors:
    Farasin A. (author) / Peciarolo F. (author) / Grangetto M. (author) / Gianaria E. (author) / Garza P. (author)

    Publication date :

    2020-01-01



    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    Classification :

    DDC:    006 / 629



    Sensorless hand guidance using microsoft hololens

    Puljiz, David / Stöhr, Erik / Riesterer, Katharina S. et al. | BASE | 2019

    Free access

    Evaluating the Microsoft HoloLens through an augmented reality assembly application

    Evans, Gabriel / Miller, Jack / Iglesias Pena, Mariangely et al. | SPIE | 2017


    General Hand Guidance Framework using Microsoft HoloLens

    Puljiz, David / Stöhr, Erik / Riesterer, Katharina S. et al. | BASE | 2019

    Free access

    Single Pilot Operations with AR-Glasses using Microsoft HoloLens

    Tran, The Huy / Behrend, Ferdinand / Funning, Niels et al. | IEEE | 2018