Can I read A System for Comprehensive Assessment of RAG Frameworks on EtoBox?

A System for Comprehensive Assessment of RAG Frameworks by Rengo, Mattia; Beadini, Senad; Alfano, Domenico; Abbruzzese, Roberto is a scholarly article available to read on EtoBox.

What is A System for Comprehensive Assessment of RAG Frameworks about?

Retrieval Augmented Generation (RAG) has emerged as a standard paradigm for enhancing the factual accuracy and contextual relevance of Large Language Models (LLMs) by integrating retrieval mechanisms. However, existing evaluation frameworks fail to provide a holistic black-box approach to assessing RAG systems, especially in real-world deployment scenarios. To address this gap, we introduce SCARF (System for Comprehensive Assessment of RAG Frameworks), a modular and flexible evaluation framework designed to benchmark deployed RAG applications systematically. SCARF provides an end-to-end, black-box evaluation methodology, enabling a limited-effort comparison across diverse RAG frameworks. Our framework supports multiple deployment configurations and facilitates automated testing across vector databases and LLM serving strategies, producing a detailed performance report. Moreover, SCARF integrates practical considerations such as response coherence, providing a scalable and adaptable solution for researchers and industry professionals evaluating RAG applications. Using the REST APIs interface, we demonstrate how SCARF can be applied to real-world scenarios, showcasing its flexibility

Author

Rengo, Mattia; Beadini, Senad; Alfano, Domenico; Abbruzzese, Roberto

Published

2025

Language

EN