There have been many recent advancements in imitation and reinforcement learning for autonomous driving, but existing metrics generally lack the means to capture a wide range of driving behaviors and compare the severity of different failure cases. To address this shortcoming, we introduce Quan-titative Evaluation for Driving (QED), which assesses different aspects of driving behavior including the ability to stay in the center of the lane, avoid weaving and erratic behavior, follow the speed limit, and avoid collisions. We compare scores generated by QED against scores assigned by human evaluators on 30 different drivers and 6 different towns in the CARLA driving simulator. In "easy" evaluation scenarios where better drivers are easily distinguished from worse drivers, QED attains 0.96 Pearson correlation and 0.97 Spearman correlation with human evaluators, similar to the baseline inter-human-evaluator 0.96 Pearson correlation and 0.95 Spearman correlation. In "hard" evaluation scenarios where ranking drivers is more ambiguous, QED attains 0.84 Pearson correlation and 0.74 Spearman correlation with human evaluators, slighter higher than the baseline inter-human-evaluator 0.78 Pearson correlation and 0.7 Spearman correlation. While QED may not capture every characteristic that defines good driving, we consider it an important foundation for reproducibility and standardization in the community.


    Zugriff

    Zugriff prüfen

    Verfügbarkeit in meiner Bibliothek prüfen

    Bestellung bei Subito €


    Exportieren, teilen und zitieren



    Titel :

    Quantitative Evaluation of Autonomous Driving in CARLA


    Beteiligte:
    Gao, Shang (Autor:in) / Paulissen, Spencer (Autor:in) / Coletti, Mark (Autor:in) / Patton, Robert (Autor:in)


    Erscheinungsdatum :

    2021-07-11


    Format / Umfang :

    987101 byte




    Medientyp :

    Aufsatz (Konferenz)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    ANTI-CARLA: An Adversarial Testing Framework for Autonomous Vehicles in CARLA

    Ramakrishna, Shreyas / Luo, Baiting / Kuhn, Christopher B. et al. | IEEE | 2022


    An Evaluation of "Crash Prediction Networks" (CPN) for Autonomous Driving Scenarios in CARLA Simulator

    Nair, Saasha / Shafaei, Sina / Auge, Daniel et al. | TIBKAT | 2021

    Freier Zugriff

    Autonomous Navigation with Deep Reinforcement Learning in Carla Simulator

    Wang, Peilin / Technische Universität Dresden | SLUB | 2023


    No Cost Autonomous Vehicle Advancements in CARLA through ROS

    Prescinotti Vivan, Gabriel / Rojas, Johan / Goberville, Nick et al. | SAE Technical Papers | 2021


    AD PerDevKit: An Autonomous Driving Perception Development Kit using CARLA simulator and ROS

    de la Pena, Javier / Bergasa, Luis M. / Antunes, Miguel et al. | IEEE | 2022