We introduce FaceVR, a novel method for gaze-aware facial reenactment in the Virtual Reality (VR) context. The key component of FaceVR is a robust algorithm to perform real-time facial motion capture of an actor who is wearing a head-mounted display (HMD), as well as a new data-driven approach for eye tracking from monocular videos. In addition to these face reconstruction components, FaceVR incorporates photo-realistic re-rendering in real time, thus allowing artificial modifications of face and eye appearances. For instance, we can alter facial expressions, change gaze directions, or remove the VR goggles in realistic re-renderings. In a live setup with a source and a target actor, we apply these newly-introduced algorithmic components. We assume that the source actor is wearing a VR device, and we capture his facial expressions and eye movement in real-time. For the target video, we mimic a similar tracking process; however, we use the source input to drive the animations of the target video, thus enabling gaze-aware facial reenactment. To render the modified target video on a stereo display, we augment our capture and reconstruction process with stereo data. In the end, FaceVR produces compelling results for a variety of applications, such as gaze-aware facial reenactment, reenactment in virtual reality, removal of VR goggles, and re-targeting of somebody's gaze direction in a video conferencing call.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality


    Beteiligte:
    Thies, J. (Autor:in) / Zollhöfer, M. (Autor:in) / Stamminger, M. (Autor:in) / Theobalt, C. (Autor:in) / Nießner, M. (Autor:in)

    Erscheinungsdatum :

    2016-01-01


    Medientyp :

    Paper


    Format :

    Elektronische Ressource


    Sprache :

    Englisch


    Klassifikation :

    DDC:    629


    Lizenzbestimmungen :


    HeadOn: Real-time Reenactment of Human Portrait Videos ; {HeadOn}: {R}eal-time Reenactment of Human Portrait Videos

    Thies, J. / Zollhöfer, M. / Theobalt, C. et al. | BASE | 2018

    Freier Zugriff


    Comparison of gaze accuracy and precision in real-world and virtual reality

    Pastel, Stefan / Chen, Chien-Hsi / Martin, Luca et al. | BASE | 2021

    Freier Zugriff

    Predictive gaze in action selection within virtual reality

    Bampouni, Elpida | BASE | 2020

    Freier Zugriff

    EgoFace: Egocentric Face Performance Capture and Videorealistic Reenactment

    Elgharib, M. / Mallikarjun B R / Tewari, A. et al. | BASE | 2019

    Freier Zugriff