UAVs (Unmanned Aerial Vehicles) and drones have been used in various fields such as delivery of goods, fire suppression, and traffic monitoring. As a result, the number of beginners in UAV control is also increasing. Traditional methods of UAV control include manual flight using twin-stick controllers or automatic flight by uploading Waypoint-based missions to the FC (Flight Controller) in advance using GCS (Ground Control System) software. However, these conventional UAV control methods take a long time for beginners to get used to it. In manual flight, considerable proficiency is required for UAV control, which increases the control fatigue of novice operators. For beginners, it may be difficult not only to control UAV, but also to handle other tasks such as manipulating a gimbal-camera or starting/pausing missions at the same time as UAV operations. Especially in emergency situations, simultaneous work can be very burdensome because the ability to respond quickly is required. Therefore, it is necessary to develop an operator-friendly control system with HMI (Human Machine Interface) (Zimmer D, Rhodes D (2006) Human–machine interfaces. IEEE Indus Appl Mag 12:29–35) that reduces the operator's controlling stress and does not require high-level control capabilities. We have developed the UAV control system with speech recognition, which is used for AI speakers that are easily seen around us, and gesture recognition, which is used for motion recognition cameras like KINECT, based on the existing control systems. We have implemented speech recognition techniques using PyAudio and SpeechRecognition, speech recognition open-source libraries, and gesture recognition techniques using YOLOv3, an object-detecting open-source library. First of all, to select speech and gesture commands, we analyze the UAV operation procedure and divided step by step. Second, the commands are adopted according to the divided UAV operation steps. Finally, we develop modules to process these commands and do the SITL (Simulation In The Loop). In this paper, we have designed a prototype of an Operator-friendly Control System with HMI (Human Machine Interface) capable of simple speech and gesture commands, and proceed with the SITL. Based on our UAV Control System with HMI, we can identify which sensors and which commands are Operator-friendly. The entire process is simulated using ROS Melodic and gazebo9 in Ubuntu 18.04 environments. As a result, we expect that the people who have no experience in operating UAVs can operate the UAV with HMI control system using speech and gesture recognition in the simulation.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Operator-Friendly UAV Control System with HMI Using Speech and Gesture Recognition


    Additional title:

    Lect. Notes Electrical Eng.


    Contributors:

    Conference:

    Asia-Pacific International Symposium on Aerospace Technology ; 2021 ; Korea (Republic of) November 15, 2021 - November 17, 2021



    Publication date :

    2022-09-30


    Size :

    14 pages





    Type of media :

    Article/Chapter (Book)


    Type of material :

    Electronic Resource


    Language :

    English




    Control Virtual Human with Speech Recognition and Gesture Recognition Technology

    Zhao, Wei / Xie, XiaoFang / Yang, XiangHong | Springer Verlag | 2012


    A Multimodal Control System for Autonomous Vehicles Using Speech, Gesture, and Gaze Recognition

    Kitaoka, Norihide / Nakagawa, Takuma / Nishimura, Ryota et al. | TIBKAT | 2020




    Control system, gesture recognition system, vehicle, and method for controlling gesture recognition system

    HOLLAND CHRISTOPHER MICHAEL | European Patent Office | 2022

    Free access