Pith. sign in

REVIEW 3 cited by

In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.10790 v2 pith:NTTZIGJ6 submitted 2024-02-16 cs.CL cs.AIcs.LG

Yuri Kuratov , Aydar Bulatov , Petr Anokhin , Dmitry Sorokin , Artyom Sorokin , Mikhail Burtsev This is my paper
classification cs.CLcs.AIcs.LG
keywords processingcapabilitieselementslongmemorymodelrecurrentsequences
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

This paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark designed to assess model capabilities in extracting and processing distributed facts within extensive texts. Our evaluation, which includes benchmarks for GPT-4 and RAG, reveals that common methods are effective only for sequences up to $10^4$ elements. In contrast, fine-tuning GPT-2 with recurrent memory augmentations enables it to handle tasks involving up to $11\times 10^6$ elements. This achievement marks a substantial leap, as it is by far the longest input processed by any neural network model to date, demonstrating a significant improvement in the processing capabilities for long sequences.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  2. ActMem: Bridging the Gap Between Memory Retrieval and Reasoning in LLM Agents

    cs.CL 2026-02 conditional novelty 6.0 of 10

    ActMem stores agent dialogue as a causal and semantic knowledge graph and uses LLM counterfactual reasoning to retrieve implicit constraints, outperforming RAG-style baselines on a new reasoning benchmark.

  3. NeMo: Needle in a Montage for Video-Language Understanding

    cs.CV 2025-09 conditional novelty 5.0 of 10

    NeMoBench, an automatically generated benchmark with 31,378 QA pairs, shows that video LLMs struggle with temporal grounding of relevant clips hidden in long montages.

Pith tools