Pith. sign in

REVIEW 11 cited by

Generating Diverse Q&A Benchmarks for RAG Evaluation with DataMorgana

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2501.12789 v1 pith:MUWRI5DU submitted 2025-01-22 cs.CL cs.IR

Simone Filice , Guy Horowitz , David Carmel , Zohar Karnin , Liane Lewin-Eytan , Yoelle Maarek This is my paper
classification cs.CLcs.IR
keywords datamorganabenchmarksdiversegeneratingquestiondatadomain-specificexisting
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Evaluating Retrieval-Augmented Generation (RAG) systems, especially in domain-specific contexts, requires benchmarks that address the distinctive requirements of the applicative scenario. Since real data can be hard to obtain, a common strategy is to use LLM-based methods to generate synthetic data. Existing solutions are general purpose: given a document, they generate a question to build a Q&A pair. However, although the generated questions can be individually good, they are typically not diverse enough to reasonably cover the different ways real end-users can interact with the RAG system. We introduce here DataMorgana, a tool for generating highly customizable and diverse synthetic Q&A benchmarks tailored to RAG applications. DataMorgana enables detailed configurations of user and question categories and provides control over their distribution within the benchmark. It uses a lightweight two-stage process, ensuring efficiency and fast iterations, while generating benchmarks that reflect the expected traffic. We conduct a thorough line of experiments, showing quantitatively and qualitatively that DataMorgana surpasses existing tools and approaches in producing lexically, syntactically, and semantically diverse question sets across domain-specific and general-knowledge corpora. DataMorgana will be made available to selected teams in the research community, as first beta testers, in the context of the upcoming SIGIR'2025 LiveRAG challenge to be announced in early February 2025.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Millions of $\text{GeAR}$-s: Extending GraphRAG to Millions of Documents

    cs.CL 2025-07 conditional novelty 5.0 of 10

    An online pseudo-alignment between retrieved passages and Wikidata triples lets a graph-based RAG pipeline run over millions of documents without offline triple extraction, at the cost of frequent topic misalignment.

  2. UiS-IAI@LiveRAG: Retrieval-Augmented Information Nugget-Based Generation of Responses

    cs.IR 2025-06 conditional novelty 4.0 of 10

    A nugget-based RAG pipeline with query rewriting and cluster-based summarization is applied to the LiveRAG challenge, where few rewrites plus the original query improve recall and larger document cutoffs hit diminishi...

  3. Leveraging LLM-Assisted Query Understanding for Live Retrieval-Augmented Generation

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Omni-RAG, a query-rewriting and decomposition pipeline on top of standard retrieval and reranking, achieved rank 2 in the SIGIR 2025 LiveRAG Challenge.

  4. Knowledge-Aware Diverse Reranking for Cross-Source Question Answering

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A knowledge-aware diverse reranking pipeline, combining LLM-based knowledge element extraction with dual-summary reranking, won the SIGIR 2025 LiveRAG competition.

  5. RMIT-ADM+S at the SIGIR 2025 LiveRAG Challenge

    cs.IR 2025-06 conditional novelty 4.0 of 10

    The RMIT-ADM+S G-RAG system, combining HyDE-style hypothetical answers with hybrid retrieval and pointwise LLM re-ranking, was declared the winner of the SIGIR 2025 LiveRAG Challenge.

  6. RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition

    cs.IR 2025-06 conditional novelty 4.0 of 10

    A RAG pipeline using InstructRAG, Pinecone, and BGE placed third in the 2025 LiveRAG Challenge, though internal evaluation only weakly predicted official scores.

  7. DoTA-RAG: Dynamic of Thought Aggregation RAG

    cs.CL 2025-06 conditional novelty 4.0 of 10

    DoTA-RAG combines query rewriting, namespace routing, dense retrieval, BM25 pruning, and reranking to answer questions over a 15M-document corpus, with reported correctness gains but fragile faithfulness under output caps.

  8. CIIR@LiveRAG 2025: Optimizing Multi-Agent Retrieval Augmented Generation through Self-Training

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A multi-agent RAG framework trained by self-supervision on high-reward interaction trajectories outperforms a vanilla RAG baseline on DataMorgana-generated questions and places 7th in the LiveRAG 2025 competition.

  9. SIGIR 2025 -- LiveRAG Challenge Report

    cs.CL 2025-07 conditional novelty 3.0 of 10

    In the SIGIR 2025 LiveRAG Challenge, all 25 active RAG teams beat the no-RAG baseline on LLM-judged correctness, and LLM scores correlated with human scores at r=0.88.

  10. Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge

    cs.CL 2025-06 conditional novelty 3.0 of 10

    A hybrid BM25-plus-dense RAG system finished 4th in faithfulness and 11th in correctness at LiveRAG 2025, and its analysis found vocabulary alignment between questions and documents to be the most important performanc...

  11. A Survey of the State-of-the-Art in Conversational Question Answering Systems

    cs.CL 2025-09 conditional novelty 2.0 of 10

    A review that categorizes ConvQA components, techniques, models, and datasets, with no new experimental result.

Pith tools