Personalized driving refers to an autonomous vehicle's ability to adapt its driving behavior or control strategies to match individual users' preferences and driving styles while maintaining safety and comfort standards. However, existing works either fail to capture every individual preference precisely or become computationally inefficient as the user base expands. Vision-Language Models (VLMs) offer promising solutions to this front through their natural language understanding and scene reasoning capabilities. In this work, we propose a lightweight yet effective on-board VLM framework that provides low-latency personalized driving performance while maintaining strong reasoning capabilities. Our solution incorporates a Retrieval-Augmented Generation (RAG)-based memory module that enables continuous learning of individual driving preferences through human feedback. Through comprehensive real-world vehicle deployment and experiments, our system has demonstrated the ability to provide safe, comfortable, and personalized driving experiences across various scenarios and significantly reduce takeover rates by up to 76.9%. To the best of our knowledge, this work represents the first end-to-end VLM-based motion control system in real-world autonomous vehicles.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validation


    Beteiligte:
    Cui, Can (Autor:in) / Yang, Zichong (Autor:in) / Zhou, Yupeng (Autor:in) / Peng, Juntong (Autor:in) / Park, Sung-Yeon (Autor:in) / Zhang, Cong (Autor:in) / Ma, Yunsheng (Autor:in) / Cao, Xu (Autor:in) / Ye, Wenqian (Autor:in) / Feng, Yiheng (Autor:in)

    Erscheinungsdatum :

    2024


    Anmerkungen:

    arXiv admin note: substantial text overlap with arXiv:2410.15281


    Medientyp :

    Preprint


    Format :

    Elektronische Ressource


    Sprache :

    Englisch




    Autonomous Vehicle Validation using Real-World Adversarial Events

    BIN-NUN AMITAI / TEBBENS RADBOUD DUINTJER / COLLIN ANNE et al. | Europäisches Patentamt | 2023

    Freier Zugriff

    AUTONOMOUS VEHICLE VALIDATION USING REAL-WORLD ADVERSARIAL EVENTS

    BIN-NUN AMITAI / TEBBENS RADBOUD DUINTJER / COLLIN ANNE et al. | Europäisches Patentamt | 2023

    Freier Zugriff

    Personalized Autonomous Driving with Large Language Models: Field Experiments

    Cui, Can / Yang, Zichong / Zhou, Yupeng et al. | IEEE | 2024


    Real- World Maneuver Extraction for Autonomous Vehicle Validation: A Comparative Study

    Erdogan, Ahmetcan / Ugranli, Burak / Adali, Erkan et al. | IEEE | 2019


    REAL- WORLD MANEUVER EXTRACTION FOR AUTONOMOUS VEHICLE VALIDATION: A COMPARATIVE STUDY

    Erdogan, Ahmetcan / Ugranli, Burak / Adali, Erkan et al. | British Library Conference Proceedings | 2019