REVIEW 25 cited by
Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases. Leveraging LCLMs' ability to natively ingest and process entire corpora of information offers numerous advantages. It enhances user-friendliness by eliminating the need for specialized knowledge of tools, provides robust end-to-end modeling that minimizes cascading errors in complex pipelines, and allows for the application of sophisticated prompting techniques across the entire system. To assess this paradigm shift, we introduce LOFT, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs' performance on in-context retrieval and reasoning. Our findings reveal LCLMs' surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks. However, LCLMs still face challenges in areas like compositional reasoning that are required in SQL-like tasks. Notably, prompting strategies significantly influence performance, emphasizing the need for continued research as context lengths grow. Overall, LOFT provides a rigorous testing ground for LCLMs, showcasing their potential to supplant existing paradigms and tackle novel tasks as model capabilities scale.
Forward citations
Cited by 25 Pith papers
-
Evidence Interfaces Shape How Retrieval-Augmented Readers Use Support
Top-k retrieval scores should be reported with complete-support coverage, because the same score can mix missing-support failures with reader-side interface effects.
-
LLM-guided Hierarchical Search for End-to-end Reasoning Intensive Retrieval
An LLM directly traverses a hierarchical semantic index of a corpus, using calibrated path-relevance scores to retrieve documents for reasoning-intensive queries.
-
CompLLM: Compression for Long Context Q&A
A segment-wise soft compression method that provides up to 4x TTFT speedup, 2x KV cache reduction, and comparable or better QA quality at 2x compression.
-
LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework
LOOM-Scope is a framework that standardizes long-context LLM evaluation across 22 benchmarks and integrates a lightweight 12-benchmark suite, LOOMBench, for fast comprehensive assessment.
-
AbsenceBench: Language Models Can't Tell What's Missing
LLMs that ace Needle-in-a-Haystack struggle to identify deliberately omitted content, a new benchmark called AbsenceBench shows.
-
CORG: Generating Answers from Complex, Interrelated Contexts
CORG, a graph-based context grouping framework, improves disambiguated answer recall on QA with distracting, ambiguous, counterfactual, and duplicated contexts, and reaches performance comparable to per-document proce...
-
APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding
Adaptive Parallel Encoding restores sequential-style attention in parallel KV-cache RAG/ICL with three training-free tweaks, enabling fast pre-cached long-context generation.
-
NoLiMa: Long-Context Evaluation Beyond Literal Matching
NoLiMa shows that long-context language models' strong retrieval performance largely depends on literal matches, and that accuracy drops sharply once those matches are removed.
-
GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?
A new synthetic benchmark reveals that LLM reasoning accuracy decays sigmoidally with problem complexity and that repeated sampling has poor scaling efficiency.
-
Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQL
Feeding Gemini 1.5 a long context with all tables, sample column values, hints, and many synthetic examples reaches 67.41% execution accuracy on BIRD dev without fine-tuning or self-consistency.
-
PRISM: Efficient Long-Range Reasoning With Short-Context LLMs
PRISM enables short-context LLMs to outperform incremental and hierarchical merging baselines on long-range reasoning with 4 to 50 times shorter contexts and up to 54% lower cost via structured memory, programmatic re...
-
Bootstrap Your Own Context Length
The authors synthesize long-context training data from short chunks and fine-tune Llama-3 models to reach 1M-token contexts with top open-source RULER scores.
-
Efficient Long Context Language Model Retrieval with Compression
CoLoR, a preference-optimized passage compressor, cuts LCLM retrieval context by 1.91x while improving average retrieval accuracy by 6% over nine benchmarks.
-
Revisiting In-Context Learning with Long Context Language Models
With long-context language models, in-context example selection methods give no reliable gain over random sampling, while adding synthetic examples improves performance by about five percent.
-
MRAG: A Modular Retrieval Framework for Time-Sensitive Question Answering
A modular, training-free retrieval framework that decomposes time-sensitive questions into semantic content and temporal constraints, then ranks evidence by combined semantic and symbolic temporal scores, outperforms ...
-
Systematic Evaluation of Long-Context LLMs on Financial Concepts
GPT-4o and GPT-4-Turbo degrade significantly on financial news retrieval as context length grows, with performance collapsing on multi-concept tasks beyond 32K tokens.
-
Monte Carlo Tree Search for Table-to-Multimodal Report Generation
MCTS-Report applies Monte Carlo Tree Search to multimodal table-to-report generation, reaching a 77.9 overall score on the new MMRBench benchmark, but its evaluation relies on a single, unvalidated LLM judge.
-
LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing
The paper reports that LongCat Sparse Attention matches full attention quality while reducing indexing overhead and supporting one-million-token training.
-
Evaluating the Retrieval Robustness of Large Language Models
A realistic benchmark with three metrics shows modern LLMs are generally robust to imperfect retrieval, though not perfectly so.
-
MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
MAPLE uses graph-influence scores to select and pseudo-label the most useful unlabeled examples, then adaptively chooses demonstrations per query, improving many-shot in-context learning with few human labels.
-
CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capability
A two-stage coarse-to-fine attention-based filtering and steering method, CAFE, improves multi-document QA accuracy across three LLMs and five benchmarks without additional training.
-
Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models
A new benchmark with hard confounders shows long-context LMs struggle at in-context retrieval, and a retrieve-then-generate fine-tune plus attention-probing decoding substantially improves them.
-
CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models
A causal reasoning benchmark with graph and tabular data shows open-source LLMs are weak at causal discovery from tables, and reveals that task correlations do not align with the benchmark's three categories.
-
On the Surprising Efficacy of LLMs for Penetration-Testing
A critical review arguing that LLMs are surprisingly effective for penetration testing because the task is largely pattern-matching, while noting serious reliability, safety, and cost barriers to autonomous use.
-
Building Task Bots with Self-learning for Enhanced Adaptability, Extensibility, and Factuality
A thesis that combines self-learning from dialog logs, schema-guided prompting, and self-aligned factuality to build task bots with minimal human intervention.
Discussion (0). Continue with ORCID to comment.