REVIEW 4 cited by
MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on multi-turn RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation is an important and often overlooked task with several additional challenges. We present MTRAG: an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. MTRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judge evaluation. Our human and automatic evaluations show that even state-of-the-art LLM RAG systems struggle on MTRAG. We demonstrate the need for strong retrieval and generation systems that can handle later turns, unanswerable questions, non-standalone questions, and multiple domains. MTRAG is available at https://github.com/ibm/mt-rag-benchmark.
Forward citations
Cited by 4 Pith papers
-
FlexRAG: A Flexible and Comprehensive Framework for Retrieval-Augmented Generation
FlexRAG is a modular, open-source RAG framework with text, multimodal, and web retrieval, plus evaluation tools and efficient memory-mapped indexing.
-
Benchmarking Poisoning Attacks against Retrieval-Augmented Generation
A unified benchmark evaluation finds that existing RAG poisoning attacks remain effective on standard QA datasets, drop on expanded knowledge bases, and are only partially mitigated by current defenses.
-
Granite Embedding R2 Models
Granite Embedding R2 is an Apache-2.0 family of ModernBERT-based retrieval and reranking models that posts high average scores on several benchmarks but falls short of top code-retrieval and reranking baselines.
-
Conversational Search: From Fundamentals to Frontiers in the LLM Era
A four-page proposal for a SIGIR 2025 half-day tutorial connecting conversational search fundamentals with LLM-era techniques; it contains no new experiments, data, or results.
Discussion (0). Sign in to comment.