Pith. sign in

REVIEW 8 cited by

In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.10790 v2 pith:NTTZIGJ6 submitted 2024-02-16 cs.CL cs.AIcs.LG

Yuri Kuratov , Aydar Bulatov , Petr Anokhin , Dmitry Sorokin , Artyom Sorokin , Mikhail Burtsev This is my paper
classification cs.CLcs.AIcs.LG
keywords processingcapabilitieselementslongmemorymodelrecurrentsequences
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

This paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark designed to assess model capabilities in extracting and processing distributed facts within extensive texts. Our evaluation, which includes benchmarks for GPT-4 and RAG, reveals that common methods are effective only for sequences up to $10^4$ elements. In contrast, fine-tuning GPT-2 with recurrent memory augmentations enables it to handle tasks involving up to $11\times 10^6$ elements. This achievement marks a substantial leap, as it is by far the longest input processed by any neural network model to date, demonstrating a significant improvement in the processing capabilities for long sequences.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  2. Efficient Retrieval-Augmented Generation via Token Co-occurrence Graphs

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    TIGRAG constructs token co-occurrence graphs for scalable graph-augmented RAG and uses iterative entity-driven retrieval to improve multi-hop QA performance over dense and prior graph methods.

  3. All Relations Lead to Rome: Automated Knowledge Graph Creation and Question Generation

    cs.IR 2026-06 unverdicted novelty 6.0 of 10

    ARLtR is a framework for jointly constructing knowledge graphs, embeddings, and grounded QA pairs from text, demonstrated on a Roman Empire dataset with over 19,000 entities and 8,400 QA pairs.

  4. ActMem: Bridging the Gap Between Memory Retrieval and Reasoning in LLM Agents

    cs.CL 2026-02 conditional novelty 6.0 of 10

    ActMem stores agent dialogue as a causal and semantic knowledge graph and uses LLM counterfactual reasoning to retrieve implicit constraints, outperforming RAG-style baselines on a new reasoning benchmark.

  5. All Relations Lead to Rome: Automated Knowledge Graph Creation and Question Generation

    cs.IR 2026-06 unverdicted novelty 5.0 of 10

    ARLtR is a framework for jointly constructing knowledge graphs, embeddings, and grounded QA pairs from text, released as a Roman Empire dataset with over 19,000 entities and 8,400 QA pairs.

  6. NeMo: Needle in a Montage for Video-Language Understanding

    cs.CV 2025-09 conditional novelty 5.0 of 10

    NeMoBench, an automatically generated benchmark with 31,378 QA pairs, shows that video LLMs struggle with temporal grounding of relevant clips hidden in long montages.

  7. Retrieval-Augmented Generation for AI-Generated Content: A Survey

    cs.CV 2024-02 accept novelty 5.0 of 10

    A survey classifying RAG foundations for AIGC, summarizing enhancements, cross-modal applications, benchmarks, limitations, and future directions.

  8. LightRAG: Simple and Fast Retrieval-Augmented Generation

    cs.IR 2024-10 unverdicted novelty 4.0 of 10

    LightRAG builds graph structures into RAG indexing and retrieval with dual-level search and incremental updates to improve accuracy and speed.

Pith tools