QO-Bench shows RAG systems retrieve relevant text but often discard typed values required for query operators, with paradigm performance inverting across operators and execution remaining a bottleneck even with gold evidence.
hub
Ruiliu Fu, Han Wang, Xuejun Zhang, Jun Zhou, and Yonghong Yan
20 Pith papers cite this work, alongside 8 external citations. Polarity classification is still indexing.
20
Pith papers citing it
8
external citations · Pith
abstract
Retrieval-Augmented Generation (RAG) has become a standard architectural pattern for incorporating domain-specific knowledge into user-facing chat applications powered by Large Language Models (LLMs). RAG systems are characterized by (1) a document retriever that queries a domain-specific corpus for context information relevant to an input query, and (2) an LLM that generates a response based on the provided query and context. However, comprehensive evaluation of RAG systems remains a challenge due to the lack of unified evaluation criteria and annotated datasets. In response, we introduce RAGBench: the first comprehensive, large-scale RAG benchmark dataset of 100k examples. It covers five unique industry-specific domains and various RAG task types. RAGBench examples are sourced from industry corpora such as user manuals, making it particularly relevant for industry applications. Further, we formalize the TRACe evaluation framework: a set of explainable and actionable RAG evaluation metrics applicable across all RAG domains. We release the labeled dataset at https://huggingface.co/datasets/rungalileo/ragbench. RAGBench explainable labels facilitate holistic evaluation of RAG systems, enabling actionable feedback for continuous improvement of production applications. Thorough extensive benchmarking, we find that LLM-based RAG evaluation methods struggle to compete with a finetuned RoBERTa model on the RAG evaluation task. We identify areas where existing approaches fall short and propose the adoption of RAGBench with TRACe towards advancing the state of RAG evaluation systems.
hub tools
citation-role summary
background 3
citation-polarity summary
roles
background 3representative citing papers
The work creates a new ground truth dataset for mapping queries to verbatim text spans in research papers and shows a 150M-parameter ModernBERT token classifier achieving 53.6 word-level F1, outperforming LLM extractors at 48.7.
OPD-Evolver uses on-policy self-distillation in fast interaction and slow attribution loops to build agents with holistic memory competence, outperforming prior systems by up to 11.5% and allowing a 9B model to compete with much larger ones.
GroundedCache reduces unsafe-served rate in RAG answer caching to 0-1.5% (vs 15-51.5% naive) via four validation gates while keeping p50 latency within 1.07x of no-cache baseline.
DualGraph combines semantic textual KGs with symbolic KGs for semi-structured QA and introduces the SpecsQA benchmark, outperforming baselines on both open and specification questions.
CARE, a context-aware LLM judge, outperforms standard methods when evaluating multi-hop retrieval quality in RAG systems.
Introduces a facet-level diagnostics framework using Facet x Chunk matrices and controlled inference modes to show that RAG hallucinations arise mainly from evidence integration failures rather than retrieval errors.
MemoryAgentBench is a multi-turn benchmark covering four memory competencies, and current memory agents fail at selective forgetting and long-range understanding.
The survey organizes RAG methods via a taxonomy of query-based, logits-based, latent, and parametric fusion with comparisons on accessibility, efficiency, applications, and challenges.
Survey mapping persistent state in LLM agents along six axes and proposing the AOEP-v0 protocol to evaluate governance and recovery obligations.
RAIDS proposes making responsibility an execution-level property in data systems via composable operator contracts and a preservation objective.
A conformal prediction filter for retrieval chunks plus an attention-based factuality classifier can raise RAG answer quality by up to 6% and detect inconsistent generations up to 77% of the time.
RAG systems optimize for factual certainty and ignore opinion diversity; O-RAG reduces Wasserstein distance to corpus sentiment and is preferred by humans 79% of the time.
Systematic tests show that specific PDF parsers combined with overlapping chunking strategies better preserve structure and improve RAG answer correctness on financial QA benchmarks including the new TableQuest dataset.
A new RAG method that retrieves chunks through aggregated entity descriptions performs on par with or slightly better than plain vector RAG, and both beat Microsoft's GraphRAG on three QA benchmarks.
LLMs judge document relevance at a level comparable to humans but frequently highlight different passages, indicating they are often not right for the right reasons and cannot fully replace human assessors.
Adding handwritten Cypher graph tools to an agentic RAG system roughly doubled factual-correctness precision and recall on MoNaCo complex questions and improved fine-grained truthfulness compared with vector-only RAG.
Human-AI teams with RAG assistants outperform AI-only systems in information-seeking tasks independent of model size, with similar perceived usability across 3B to 70B models.
Small language models can run RAG generation on-device without GPUs in reasonable time.
citing papers explorer
-
QO-Bench: Diagnosing Query-Operator-Preserving Retrieval over Typed Event Tuples
QO-Bench shows RAG systems retrieve relevant text but often discard typed values required for query operators, with paradigm performance inverting across operators and execution remaining a bottleneck even with gold evidence.
-
ACL-Verbatim: hallucination-free question answering for research
The work creates a new ground truth dataset for mapping queries to verbatim text spans in research papers and shows a 150M-parameter ModernBERT token classifier achieving 53.6 word-level F1, outperforming LLM extractors at 48.7.
-
OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation
OPD-Evolver uses on-policy self-distillation in fast interaction and slow attribution loops to build agents with holistic memory competence, outperforming prior systems by up to 11.5% and allowing a 9B model to compete with much larger ones.
-
Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer?
GroundedCache reduces unsafe-served rate in RAG answer caching to 0-1.5% (vs 15-51.5% naive) via four validation gates while keeping p50 latency within 1.07x of no-cache baseline.
-
Query Symbolically or Retrieve Semantically? A Dataset and Method for Semi-Structured Question Answering
DualGraph combines semantic textual KGs with symbolic KGs for semi-structured QA and introduces the SpecsQA benchmark, outperforming baselines on both open and specification questions.
-
Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategies
CARE, a context-aware LLM judge, outperforms standard methods when evaluating multi-hop retrieval quality in RAG systems.
-
Facet-Level Tracing of Evidence Uncertainty and Hallucination in RAG
Introduces a facet-level diagnostics framework using Facet x Chunk matrices and controlled inference modes to show that RAG hallucinations arise mainly from evidence integration failures rather than retrieval errors.
-
Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
MemoryAgentBench is a multi-turn benchmark covering four memory competencies, and current memory agents fail at selective forgetting and long-range understanding.
-
Retrieval-Augmented Generation for Natural Language Processing: A Survey
The survey organizes RAG methods via a taxonomy of query-based, logits-based, latent, and parametric fusion with comparisons on accessibility, efficiency, applications, and challenges.
-
Always-OnAgents:A Survey of Persistent Memory, State, and Governance in LLMAgents
Survey mapping persistent state in LLM agents along six axes and proposing the AOEP-v0 protocol to evaluate governance and recovery obligations.
-
RAIDS: Rethinking Data Systems as Responsible Intelligent Infrastructure
RAIDS proposes making responsibility an execution-level property in data systems via composable operator contracts and a preservation objective.
-
Towards Dependable Retrieval-Augmented Generation Using Factual Confidence Prediction
A conformal prediction filter for retrieval chunks plus an attention-based factuality classifier can raise RAG answer quality by up to 6% and detect inconsistent generations up to 77% of the time.
-
Retrieval-Augmented Generation Must Move Beyond Factual Grounding to Represent Diverse Opinions
RAG systems optimize for factual certainty and ignore opinion diversity; O-RAG reduces Wasserstein distance to corpus sentiment and is preferred by humans 79% of the time.
-
Empirical Evaluation of PDF Parsing and Chunking for Financial Question Answering with RAG
Systematic tests show that specific PDF parsers combined with overlapping chunking strategies better preserve structure and improve RAG answer correctness on financial QA benchmarks including the new TableQuest dataset.
-
UnWeaving the knots of GraphRAG -- turns out VectorRAG is almost enough
A new RAG method that retrieves chunks through aggregated entity descriptions performs on par with or slightly better than plain vector RAG, and both beat Microsoft's GraphRAG on three QA benchmarks.
-
LLMs as Assessors: Right for the Right Reason?
LLMs judge document relevance at a level comparable to humans but frequently highlight different passages, indicating they are often not right for the right reasons and cannot fully replace human assessors.
-
Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)
Adding handwritten Cypher graph tools to an agentic RAG system roughly doubled factual-correctness precision and recall on MoNaCo complex questions and improved fine-grained truthfulness compared with vector-only RAG.
-
Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?
Human-AI teams with RAG assistants outperform AI-only systems in information-seeking tasks independent of model size, with similar perceived usability across 3B to 70B models.
-
Little Brains, Big Feats: Exploring Compact Language Models
Small language models can run RAG generation on-device without GPUs in reasonable time.
- PolyWorkBench: Benchmarking LLM Agents for Cross-Lingual Long-Horizon Workflows