There have been many recent advancements in imitation and reinforcement learning for autonomous driving, but existing metrics generally lack the means to capture a wide range of driving behaviors and compare the severity of different failure cases. To address this shortcoming, we introduce Quan-titative Evaluation for Driving (QED), which assesses different aspects of driving behavior including the ability to stay in the center of the lane, avoid weaving and erratic behavior, follow the speed limit, and avoid collisions. We compare scores generated by QED against scores assigned by human evaluators on 30 different drivers and 6 different towns in the CARLA driving simulator. In "easy" evaluation scenarios where better drivers are easily distinguished from worse drivers, QED attains 0.96 Pearson correlation and 0.97 Spearman correlation with human evaluators, similar to the baseline inter-human-evaluator 0.96 Pearson correlation and 0.95 Spearman correlation. In "hard" evaluation scenarios where ranking drivers is more ambiguous, QED attains 0.84 Pearson correlation and 0.74 Spearman correlation with human evaluators, slighter higher than the baseline inter-human-evaluator 0.78 Pearson correlation and 0.7 Spearman correlation. While QED may not capture every characteristic that defines good driving, we consider it an important foundation for reproducibility and standardization in the community.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Quantitative Evaluation of Autonomous Driving in CARLA


    Contributors:


    Publication date :

    2021-07-11


    Size :

    987101 byte




    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    Injecting Conflict Situations in Autonomous Driving Simulation Using CARLA

    Mihaylova, Tsvetomila / Reitmann, Stefan / Topp, Elin A. et al. | IEEE | 2025


    Injecting Conflict Situations in Autonomous Driving Simulation using CARLA

    Mihaylova, Tsvetomila / Reitmann, Stefan / Topp, Elin A. et al. | ArXiv | 2025

    Free access

    ANTI-CARLA: An Adversarial Testing Framework for Autonomous Vehicles in CARLA

    Ramakrishna, Shreyas / Luo, Baiting / Kuhn, Christopher B. et al. | IEEE | 2022


    CARLA Real Traffic Scenarios -- novel training ground and benchmark for autonomous driving

    Osiński, Błażej / Miłoś, Piotr / Jakubowski, Adam et al. | ArXiv | 2020

    Free access

    Rationale-aware Autonomous Driving Policy utilizing Safety Force Field implemented on CARLA Simulator

    Suk, Ho / Kim, Taewoo / Park, Hyungbin et al. | ArXiv | 2022

    Free access