We introduce MotioNet, a deep neural network that directly reconstructs the motion of a 3D human skeleton from monocular video. While previous methods rely on either rigging or inverse kinematics (IK) to associate a consistent skeleton with temporally coherent joint rotations, our method is the first data-driven approach that directly outputs a kinematic skeleton, which is a complete, commonly used, motion representation. At the crux of our approach lies a deep neural network with embedded kinematic priors, which decomposes sequences of 2D joint positions into two separate attributes: a single, symmetric, skeleton, encoded by bone lengths, and a sequence of 3D joint rotations associated with global root positions and foot contact labels. These attributes are fed into an integrated forward kinematics (FK) layer that outputs 3D positions, which are compared to a ground truth. In addition, an adversarial loss is applied to the velocities of the recovered rotations, to ensure that they lie on the manifold of natural joint rotations. The key advantage of our approach is that it learns to infer natural joint rotations directly from the training data, rather than assuming an underlying model, or inferring them from joint positions using a data-agnostic IK solver. We show that enforcing a single consistent skeleton along with temporally coherent joint rotations constrains the solution space, leading to a more robust handling of self-occlusions and depth ambiguities. ; This work has been partly supported by the project that has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 739578 (RISE – Call: H2020-WIDESPREAD-01-2016-2017-TeamingPhase2) and the Government of the Republic of Cyprus through the Directorate General for European Programmes, Coordination and Development.


    Access

    Download


    Export, share and cite



    Title :

    MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency


    Contributors:

    Publication date :

    2020-09-01


    Remarks:

    oai:zenodo.org:4572649



    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    Classification :

    DDC:    629




    Automatic reconstruction of 3D human arm motion from a monocular image sequence

    Filova, V. / Solina, F. / Lenarcic, J. | British Library Online Contents | 1998



    Robust deformable shape reconstruction from monocular video with manifold forests

    Tao, L. / Matuszewski, B. J. | British Library Online Contents | 2016


    Self-Supervised 3D Reconstruction and Ego-Motion Estimation Via On-Board Monocular Video

    Jia, Shaocheng / Pei, Xin / Jing, Xiao et al. | IEEE | 2022