pith. sign in

arxiv: 2402.10790 · v2 · pith:NTTZIGJ6new · submitted 2024-02-16 · 💻 cs.CL · cs.AI· cs.LG

In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss

Yuri Kuratov , Aydar Bulatov , Petr Anokhin , Dmitry Sorokin , Artyom Sorokin , Mikhail Burtsev This is my paper
classification 💻 cs.CL cs.AIcs.LG
keywords processingcapabilitieselementslongmemorymodelrecurrentsequences
0
0 comments X
read the original abstract

This paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark designed to assess model capabilities in extracting and processing distributed facts within extensive texts. Our evaluation, which includes benchmarks for GPT-4 and RAG, reveals that common methods are effective only for sequences up to $10^4$ elements. In contrast, fine-tuning GPT-2 with recurrent memory augmentations enables it to handle tasks involving up to $11\times 10^6$ elements. This achievement marks a substantial leap, as it is by far the longest input processed by any neural network model to date, demonstrating a significant improvement in the processing capabilities for long sequences.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Efficient Retrieval-Augmented Generation via Token Co-occurrence Graphs

    cs.CL 2026-06 unverdicted novelty 6.0

    TIGRAG constructs token co-occurrence graphs for scalable graph-augmented RAG and uses iterative entity-driven retrieval to improve multi-hop QA performance over dense and prior graph methods.

  2. All Relations Lead to Rome: Automated Knowledge Graph Creation and Question Generation

    cs.IR 2026-06 unverdicted novelty 6.0

    ARLtR is a framework for jointly constructing knowledge graphs, embeddings, and grounded QA pairs from text, demonstrated on a Roman Empire dataset with over 19,000 entities and 8,400 QA pairs.

  3. All Relations Lead to Rome: Automated Knowledge Graph Creation and Question Generation

    cs.IR 2026-06 unverdicted novelty 5.0

    ARLtR is a framework for jointly constructing knowledge graphs, embeddings, and grounded QA pairs from text, released as a Roman Empire dataset with over 19,000 entities and 8,400 QA pairs.

  4. Retrieval-Augmented Generation for AI-Generated Content: A Survey

    cs.CV 2024-02 accept novelty 5.0

    A survey classifying RAG foundations for AIGC, summarizing enhancements, cross-modal applications, benchmarks, limitations, and future directions.

  5. LightRAG: Simple and Fast Retrieval-Augmented Generation

    cs.IR 2024-10 unverdicted novelty 4.0

    LightRAG builds graph structures into RAG indexing and retrieval with dual-level search and incremental updates to improve accuracy and speed.