REVIEW 13 cited by
Retrieval meets Long Context Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Retrieval meets Long Context Large Language Models
read the original abstract
Extending the context window of large language models (LLMs) is getting popular recently, while the solution of augmenting LLMs with retrieval has existed for years. The natural questions are: i) Retrieval-augmentation versus long context window, which one is better for downstream tasks? ii) Can both methods be combined to get the best of both worlds? In this work, we answer these questions by studying both solutions using two state-of-the-art pretrained LLMs, i.e., a proprietary 43B GPT and Llama2-70B. Perhaps surprisingly, we find that LLM with 4K context window using simple retrieval-augmentation at generation can achieve comparable performance to finetuned LLM with 16K context window via positional interpolation on long context tasks, while taking much less computation. More importantly, we demonstrate that retrieval can significantly improve the performance of LLMs regardless of their extended context window sizes. Our best model, retrieval-augmented Llama2-70B with 32K context window, outperforms GPT-3.5-turbo-16k and Davinci003 in terms of average score on nine long context tasks including question answering, query-based summarization, and in-context few-shot learning tasks. It also outperforms its non-retrieval Llama2-70B-32k baseline by a margin, while being much faster at generation. Our study provides general insights on the choice of retrieval-augmentation versus long context extension of LLM for practitioners.
Forward citations
Cited by 13 Pith papers
-
Don't Stir the Pot! Authorized Vector Data Retrieval via Access-Aware Indexing
Veda and EffVeda build access-aware lattice indexes on role-partitioned vector blocks to support authorized top-k queries with controlled duplication and pruned search.
-
Don't Stir the Pot! Authorized Vector Data Retrieval via Access-Aware Indexing
Veda and EffVeda partition vectors into disjoint role-combination blocks, apply lattice-based copy and merge operations within a storage budget, index large nodes with HNSW, and use coordinated search with distance bo...
-
When Knowledge Changes: Metamorphic Testing of RAG Systems with Mutations
Eleven corpus mutations expose 4.9–10.2% metamorphic violations in RAG pipelines, with an oracle F1 of 0.927–1.000 versus at most 0.570 for RAGAS.
-
Token Reduction Is Not Cost Reduction
Token reduction of tool output does not reliably lower billed coding-agent cost; prompt-cache traffic dominates, and aggressive compression can also destroy task success.
-
Token Reduction Is Not Cost Reduction
In paid coding-agent runs, compressing 38% of tool-output tokens did not lower billed cost; cache traffic and extra turns dominated the bill.
-
Token Reduction Is Not Cost Reduction
In 2,848 paired provider-billed Claude Code runs, removing 38% of tool-output tokens raised cost by 6.8%; cache traffic dominated the bill, and compression broke code anchors agents need for patches.
-
Beyond Summaries: Structure-Aware Labeling of Code Changes with Large Language Models
A two-stage LLM pipeline for taxonomy-based labeling of code changes in patches achieves up to 84% recall and 81% precision on a manually curated benchmark of natural and synthetic patches.
-
AV-SQL: Decomposing Complex Text-to-SQL Queries with Agentic Views
AV-SQL uses a pipeline of LLM agents to generate intermediate CTE views that decompose complex Text-to-SQL queries, reaching 70.38% execution accuracy on Spider 2.0.
-
Exploratory Retrieval-Augmented Planning For Continual Embodied Instruction Following
ExRAP couples LLM planning with a temporal knowledge-graph memory and information-based exploration, improving success and efficiency for continual embodied instruction following.
-
Don't Stir the Pot! Authorized Vector Data Retrieval via Access-Aware Indexing
Veda and EffVeda partition vector data by role combinations, apply lattice-based copy/merge under storage budget, index large nodes with HNSW and small nodes with linear scan, then use query plans and coordinated sear...
-
Fine-Tuning Small Language Models for Solution-Oriented Windows Event Log Analysis
Fine-tuned small language models trained on a synthetic Windows event log dataset with remediation steps outperform larger models in issue detection and solution generation with lower computational cost.
-
Interfaze: The Future of AI is built on Task-Specific Small Models
Interfaze-Beta uses small specialist models and tools to build a compact context that a general-purpose LLM answers from, reporting competitive benchmark scores without reproducible evidence.
-
Retrieval-Augmented Generation for Large Language Models: A Survey
A survey of RAG paradigms, components, benchmarks, and challenges for improving LLMs on knowledge-intensive tasks.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.