Pith. sign in

REVIEW 9 cited by

BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.10149 v2 pith:YI3P2FKQ submitted 2024-06-14 cs.CL cs.AI

Yuri Kuratov , Aydar Bulatov , Petr Anokhin , Ivan Rodkin , Dmitry Sorokin , Artyom Sorokin , Mikhail Burtsev This is my paper
classification cs.CLcs.AI
keywords contextbabilonglongmodelsincreasedllmsmethodsreasoning
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In recent years, the input context sizes of large language models (LLMs) have increased dramatically. However, existing evaluation methods have not kept pace, failing to comprehensively assess the efficiency of models in handling long contexts. To bridge this gap, we introduce the BABILong benchmark, designed to test language models' ability to reason across facts distributed in extremely long documents. BABILong includes a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets. These tasks are challenging on their own, and even more demanding when the required facts are scattered across long natural text. Our evaluations show that popular LLMs effectively utilize only 10-20\% of the context and their performance declines sharply with increased reasoning complexity. Among alternatives to in-context reasoning, Retrieval-Augmented Generation methods achieve a modest 60\% accuracy on single-fact question answering, independent of context length. Among context extension methods, the highest performance is demonstrated by recurrent memory transformers after fine-tuning, enabling the processing of lengths up to 50 million tokens. The BABILong benchmark is extendable to any length to support the evaluation of new upcoming models with increased capabilities, and we provide splits up to 10 million token lengths.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WildTrace: Benchmarking Natural Evidence Trails in Long-Context Reasoning

    cs.CL 2026-07 conditional novelty 8.0 of 10

    WILDTRACE evaluates long-context models on 481 natural multi-hop evidence trails from 214 real documents, with top systems at 75.3% and geometry-specific weaknesses.

  2. UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A unified KG-plus-live-web data pipeline covering all three multimodal BrowseComp information-flow patterns, plus an exploration-degree filter, yields a 35B agent at 54.4 avg accuracy.

  3. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  4. Dropping the Anchor: Statistical Context Summarization for Distributed Systems via Pulsar Attention

    cs.CL 2026-05 conditional novelty 6.0 of 10

    IDF-selected chunk summaries replace Star Attention's static anchor, cutting Phase 1 FLOPs up to 3.3× and reportedly beating dense attention on RULER at 32K–128K.

  5. Positional Biases Shift as Inputs Approach Context Window Limits

    cs.CL 2025-08 conditional novelty 6.0 of 10

    The lost-in-the-middle effect in LLMs appears mainly when inputs fill up to half the model's context window; beyond that, accuracy favors information closest to the end.

  6. LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework

    cs.CL 2025-07 conditional novelty 6.0 of 10

    LOOM-Scope is a framework that standardizes long-context LLM evaluation across 22 benchmarks and integrates a lightweight 12-benchmark suite, LOOMBench, for fast comprehensive assessment.

  7. IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments

    cs.CV 2025-06 conditional novelty 6.0 of 10

    IntPhys 2 evaluates models on permanence, immutability, continuity, and solidity in synthetic videos, and finds most models near chance while humans near perfect.

  8. Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs

    cs.CL 2026-01 conditional novelty 5.0 of 10

    On a new extended needle-in-a-haystack benchmark, explicit anti-hallucination prompts and dispersed fact placement cause some long-context LLMs to over-refuse or collapse in accuracy, while others remain robust.

  9. A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A unified taxonomy and comparative meta-evaluation of automatic evaluation methods across text, vision, and speech generation, concluding that LLM-based evaluators dominate current practice.

Pith tools