REVIEW 6 cited by
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss
read the original abstract
This paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark designed to assess model capabilities in extracting and processing distributed facts within extensive texts. Our evaluation, which includes benchmarks for GPT-4 and RAG, reveals that common methods are effective only for sequences up to $10^4$ elements. In contrast, fine-tuning GPT-2 with recurrent memory augmentations enables it to handle tasks involving up to $11\times 10^6$ elements. This achievement marks a substantial leap, as it is by far the longest input processed by any neural network model to date, demonstrating a significant improvement in the processing capabilities for long sequences.
Forward citations
Cited by 6 Pith papers
-
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.
-
Efficient Retrieval-Augmented Generation via Token Co-occurrence Graphs
TIGRAG constructs token co-occurrence graphs for scalable graph-augmented RAG and uses iterative entity-driven retrieval to improve multi-hop QA performance over dense and prior graph methods.
-
All Relations Lead to Rome: Automated Knowledge Graph Creation and Question Generation
ARLtR is a framework for jointly constructing knowledge graphs, embeddings, and grounded QA pairs from text, demonstrated on a Roman Empire dataset with over 19,000 entities and 8,400 QA pairs.
-
All Relations Lead to Rome: Automated Knowledge Graph Creation and Question Generation
ARLtR is a framework for jointly constructing knowledge graphs, embeddings, and grounded QA pairs from text, released as a Roman Empire dataset with over 19,000 entities and 8,400 QA pairs.
-
Retrieval-Augmented Generation for AI-Generated Content: A Survey
A survey classifying RAG foundations for AIGC, summarizing enhancements, cross-modal applications, benchmarks, limitations, and future directions.
-
LightRAG: Simple and Fast Retrieval-Augmented Generation
LightRAG builds graph structures into RAG indexing and retrieval with dual-level search and incremental updates to improve accuracy and speed.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.