pith. sign in

arxiv: 2501.03468 · v1 · pith:OPQDGLA5new · submitted 2025-01-07 · 💻 cs.CL · cs.AI

MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems

Yannis Katsis , Sara Rosenthal , Kshitij Fadnis , Chulaka Gunasekara , Young-Suk Lee , Lucian Popa , Vraj Shah , Huaiyu Zhu
show 2 more authors
Danish Contractor Marina Danilevsky
This is my paper
classification 💻 cs.CL cs.AI
keywords mtragevaluatinggenerationmulti-turnsystemsacrossbenchmarkconversations
0
0 comments X
read the original abstract

Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on multi-turn RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation is an important and often overlooked task with several additional challenges. We present MTRAG: an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. MTRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judge evaluation. Our human and automatic evaluations show that even state-of-the-art LLM RAG systems struggle on MTRAG. We demonstrate the need for strong retrieval and generation systems that can handle later turns, unanswerable questions, non-standalone questions, and multiple domains. MTRAG is available at https://github.com/ibm/mt-rag-benchmark.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer?

    cs.CR 2026-05 unverdicted novelty 6.0

    GroundedCache reduces unsafe-served rate in RAG answer caching to 0-1.5% (vs 15-51.5% naive) via four validation gates while keeping p50 latency within 1.07x of no-cache baseline.

  2. Detecting Is Not Resolving: The Monitoring Control Gap in Retrieval Augmented LLMs

    cs.AI 2026-05 unverdicted novelty 6.0

    RAG models exhibit a monitoring-control gap: they acknowledge epistemic conflicts in accumulating documents yet fail to constrain unsafe recommendations, with single-turn tests overestimating safety.

  3. DQA: Diagnostic Question Answering for IT Support

    cs.CL 2026-04 unverdicted novelty 6.0

    DQA maintains persistent diagnostic state and aggregates retrievals at the root-cause level to reach 78.7% success on 150 enterprise IT scenarios versus 41.3% for standard multi-turn RAG while cutting average turns fr...

  4. RAG-DIVE: A Dynamic Approach for Multi-Turn Dialogue Evaluation in Retrieval-Augmented Generation

    cs.IR 2026-01 unverdicted novelty 5.0

    RAG-DIVE uses an LLM to dynamically generate, validate, and evaluate multi-turn dialogues for assessing RAG system performance in interactive settings.

  5. Little Brains, Big Feats: Exploring Compact Language Models

    cs.CL 2026-06 unverdicted novelty 3.0

    Small language models can run RAG generation on-device without GPUs in reasonable time.

  6. 5ting at SemEval-2026 Task 8: Strong End-to-End Multi-Turn RAG via LLM-Based Reranking and Faithfulness Control

    cs.CL 2026-06 unverdicted novelty 2.0

    5ting achieves nDCG@5 of 0.4719 on Task A and harmonic score 0.5597 with RL_F 0.7692 on Task C for multi-turn RAG via standard dense retrieval plus LLM reranking and faithfulness constraints.