While advances in prompt engineering and RAG have improved LLM proficiency in field-specific, specialized tasks, there has yet to be an industry standard or accepted evaluation metric of the highly fragmented RAG solutions that are currently being deployed. Thus, in this work, we focused on building a robust LLM and RAG evaluation platform. We contribute 1) a platform that evaluates an RAG system's performance on a multimodal input context for LLM question answering and 2) MRAFE: Multimodal Retrieval Augmented Feature Extractor, which processes information from the input to our platform. Through various automated and systematic hand testing, we find that our evaluation benchmarks are useful in determining noise robustness, negative rejection, information integration, and counterfactual rejection. Such a platform would serve as a useful tool for developers iterating on retrieval systems and regulatory bodies creating AI-focused governance alike.


    Zugriff

    Zugriff prüfen

    Verfügbarkeit in meiner Bibliothek prüfen

    Bestellung bei Subito €


    Exportieren, teilen und zitieren



    Titel :

    Multimodal Retrieval Augmented Generation Evaluation Benchmark


    Beteiligte:
    Sun, Tienlan (Autor:in) / Somalwar, Anaiy (Autor:in) / Chan, Hinson (Autor:in)


    Erscheinungsdatum :

    24.06.2024


    Format / Umfang :

    1275043 byte





    Medientyp :

    Aufsatz (Konferenz)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    A comparison of 3D shape retrieval methods based on a large-scale benchmark supporting multimodal queries

    Li, B. / Lu, Y. / Li, C. et al. | British Library Online Contents | 2015



    Global Optimization of Multimodal Aerodynamic Optimization Benchmark Case

    Poole, Daniel J. / Allen, Christian B. / Rendall, T. | AIAA | 2017


    RAGTraffic: Utilizing Retrieval-Augmented Generation for Intelligent Traffic Signal Control

    Zhang, Zhendong / Shen, Zhen / Yuan, Min et al. | IEEE | 2024


    A Retrieval-Augmented Generation-Based Method for Aviation Accident Data Analysis

    Yang, Jianzhong / Xiang, Xinyu / Chen, Xiyuan | IEEE | 2024