Many emerging applications of nano-sized unmanned aerial vehicles (UAVs), with a few cm(2) form-factor, revolve around safely interacting with humans in complex scenarios, for example, monitoring their activities or looking after people needing care. Such sophisticated autonomous functionality must be achieved while dealing with severe constraints in payload, battery, and power budget (similar to 100 mW). In this work, we attack a complex task going from perception to control: to estimate and maintain the nano-UAV's relative 3-D pose with respect to a person while they freely move in the environment-a task that, to the best of our knowledge, has never previously been targeted with fully onboard computation on a nano-sized UAV. Our approach is centered around a novel vision-based deep neural network (DNN), called Frontnet, designed for deployment on top of a parallel ultra-low power (PULP) processor aboard a nano-UAV. We present a vertically integrated approach starting from the DNN model design, training, and dataset augmentation down to 8-bit quantization and deployment in-field. PULP-Frontnet can operate in real-time (up to 135 frame/s), consuming less than 87 mW for processing at peak throughput and down to 0.43 mJ/frame in the most energy-efficient operating point. Field experiments demonstrate a closed-loop top-notch autonomous navigation capability, with a tiny 27-g Crazyflie 2.1 nano-UAV. Compared against an ideal sensing setup, onboard pose inference yields excellent drone behavior in terms of median absolute errors, such as positional (onboard: 41 cm, ideal: 26 cm) and angular (onboard: 3.7 degrees, ideal: 4.1 degrees). We publicly release videos and the source code of our work.


    Access

    Download


    Export, share and cite



    Title :

    Fully Onboard AI-Powered Human-Drone Pose Estimation on Ultralow-Power Autonomous Flying Nano-UAVs


    Contributors:
    Palossi, D (author) / Zimmerman, N (author) / Burrello, A (author) / Conti, F (author) / Muller, H (author) / Gambardella, LM (author) / Benini, L (author) / Giusti, A (author) / Guzzi, J (author) / Palossi, D

    Publication date :

    2022-01-01


    Remarks:

    doi:10.1109/JIOT.2021.3091643



    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    Classification :

    DDC:    629



    Fully Onboard AI-powered Human-Drone Pose Estimation on Ultra-low Power Autonomous Flying Nano-UAVs

    Palossi D. / Zimmerman N. / Burrello A. et al. | BASE | 2022

    Free access

    Onboard drone human-machine interface for autonomous operation

    BALTRUSCH ROBERT ERIC / RECKER SHAWN THOMAS / BUTKIEWICZ MARK THOMAS | European Patent Office | 2021

    Free access

    FULLY AUTONOMOUS DRONE FLIGHTS

    HINTERMANN MATHIAS ANDREAS / HUANG JUSTIN / PATTON RUSSELL DOUGLAS | European Patent Office | 2023

    Free access