Pith. sign in

REVIEW 13 cited by

Long Context RAG Performance of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.03538 v1 pith:TREJVLMQ submitted 2024-11-05 cs.LG cs.CL

Quinn Leng , Jacob Portes , Sam Havens , Matei Zaharia , Michael Carbin This is my paper · ORCID
classification cs.LGcs.CL
keywords contextlongllmsmodelsperformancetokensaccuracyimprove
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Retrieval Augmented Generation (RAG) has emerged as a crucial technique for enhancing the accuracy of Large Language Models (LLMs) by incorporating external information. With the advent of LLMs that support increasingly longer context lengths, there is a growing interest in understanding how these models perform in RAG scenarios. Can these new long context models improve RAG performance? This paper presents a comprehensive study of the impact of increased context length on RAG performance across 20 popular open source and commercial LLMs. We ran RAG workflows while varying the total context length from 2,000 to 128,000 tokens (and 2 million tokens when possible) on three domain-specific datasets, and report key insights on the benefits and limitations of long context in RAG applications. Our findings reveal that while retrieving more documents can improve performance, only a handful of the most recent state of the art LLMs can maintain consistent accuracy at long context above 64k tokens. We also identify distinct failure modes in long context scenarios, suggesting areas for future research.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Evidence Interfaces Shape How Retrieval-Augmented Readers Use Support

    cs.AI 2026-07 accept novelty 7.0 of 10

    Top-k retrieval scores should be reported with complete-support coverage, because the same score can mix missing-support failures with reader-side interface effects.

  2. Replicating Belief, Not Bits: Epistemic State Replication for Agentic Systems

    cs.AI 2026-07 conditional novelty 7.0 of 10

    ESR separates an immutable evidence log from a stochastic belief lineage so agent replicas stay semantically compatible without bitwise state equality.

  3. LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework

    cs.CL 2025-07 conditional novelty 6.0 of 10

    LOOM-Scope is a framework that standardizes long-context LLM evaluation across 22 benchmarks and integrates a lightweight 12-benchmark suite, LOOMBench, for fast comprehensive assessment.

  4. Long Context vs. RAG for LLMs: An Evaluation and Revisits

    cs.CL 2024-12 conditional novelty 6.0 of 10

    LC generally outperforms RAG on filtered long-context QA, but RAG wins on dialogue and general yes/no questions.

  5. Drowning in Documents: Consequences of Scaling Reranker Inference

    cs.IR 2024-11 conditional novelty 6.0 of 10

    Modern cross-encoder rerankers improve Recall@10 at small candidate counts but degrade when reranking thousands of documents, sometimes falling below first-stage dense retrieval alone.

  6. Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs

    cs.CL 2026-01 conditional novelty 5.0 of 10

    On a new extended needle-in-a-haystack benchmark, explicit anti-hallucination prompts and dispersed fact placement cause some long-context LLMs to over-refuse or collapse in accuracy, while others remain robust.

  7. Agent Identity Evals: Measuring Agentic Identity

    cs.AI 2025-07 conditional novelty 5.0 of 10

    Introduces Agent Identity Evals (AIE), five similarity-based metrics for LMA identity stability, with pilot experiments showing identifiability always at zero and no statistical support.

  8. MacRAG: Compress, Slice, and Scale-up for Multi-Scale Adaptive Context RAG

    cs.CL 2025-05 conditional novelty 5.0 of 10

    MacRAG retrieves small slices of compressed summaries, scales up candidate chunks, and merges neighboring chunks, achieving higher average F1 than RAPTOR and LongRAG on three LongBench multi-hop QA datasets.

  9. Optimizing Retrieval-Augmented Generation: Analysis of Hyperparameter Impact on Performance and Efficiency

    cs.LG 2025-05 conditional novelty 4.0 of 10

    A systematic test of RAG hyperparameters on 100 questions shows naive small-window chunking outperforms semantic chunking and re-ranking carries a fivefold runtime cost.

  10. Enhancing Cache-Augmented Generation (CAG) with Adaptive Contextual Compression for Scalable Knowledge Integration

    cs.CL 2025-05 reject novelty 4.0 of 10

    Adaptive Contextual Compression and a Hybrid CAG-RAG framework are proposed to compress cached context and selectively retrieve new knowledge, but the evaluation is incomplete and not reproducible.

  11. Stay Hungry, Stay Foolish: On the Extended Reading Articles Generation with LLMs

    cs.CL 2025-04 reject novelty 4.0 of 10

    An LLM pipeline generates TED-Ed extended reading articles and course recommendations, but its recommendation hit rate drops when the generation stages are added, undercutting the claimed accuracy.

  12. Existential Crisis: A Social Robot's Reason for Being

    cs.RO 2025-01 conditional novelty 4.0 of 10

    A 12-person pilot found higher positive affect and likeability scores for a joking NAO robot than a formal one, but only descriptive statistics support the claim.

  13. Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks

    cs.CL 2024-12 reject novelty 4.0 of 10

    Preloading a small knowledge base into a long-context LLM with a cached KV cache can beat traditional RAG on accuracy and latency, but the paper's evaluation gives CAG an unfair advantage by feeding it the exact answe...

Pith tools