Pith. sign in

REVIEW 23 cited by

Landmark Attention: Random-Access Infinite Context Length for Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.16300 v2 pith:BL7MJCYO submitted 2023-05-25 cs.CL cs.LG

Amirkeivan Mohtashami , Martin Jaggi This is my paper · ORCID
classification cs.CLcs.LG
keywords attentioncontextlandmarkmechanismmemorymethodrandom-accesswhile
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

While Transformers have shown remarkable success in natural language processing, their attention mechanism's large memory requirements have limited their ability to handle longer contexts. Prior approaches, such as recurrent memory or retrieval-based augmentation, have either compromised the random-access flexibility of attention (i.e., the capability to select any token in the entire context) or relied on separate mechanisms for relevant context retrieval, which may not be compatible with the model's attention. In this paper, we present a novel approach that allows access to the complete context while retaining random-access flexibility, closely resembling running attention on the entire context. Our method uses a landmark token to represent each block of the input and trains the attention to use it for selecting relevant blocks, enabling retrieval of blocks directly through the attention mechanism instead of by relying on a separate mechanism. Our approach seamlessly integrates with specialized data structures and the system's memory hierarchy, enabling processing of arbitrarily long context lengths. We demonstrate that our method can obtain comparable performance with Transformer-XL while significantly reducing the number of retrieved tokens in each step. Finally, we show that fine-tuning LLaMA 7B with our method successfully extends its context length capacity to over 32k tokens, allowing for inference at the context lengths of GPT-4. We release the implementation of landmark attention and the code to reproduce our experiments at https://github.com/epfml/landmark-attention/.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 23 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding

    cs.LG 2026-07 accept novelty 7.0 of 10

    Page-local rank-8 spectral key summaries let sparse decode selection track the exact mass oracle and match FullKV quality at ~2% attended tokens with 2× latency cut at 1M context.

  2. Anti-Periodic Positional Encoding: M\"obius Boundary Conditions Make In-Context Retrieval Reliable

    cs.CL 2026-07 conditional novelty 7.0 of 10

    An anti-periodic rotary positional encoding (M\u00f6bius RoPE) on 25% of heads collapses seed-to-seed variance in needle-in-a-haystack retrieval at matched perplexity.

  3. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

    cs.DC 2026-06 accept novelty 6.5 of 10

    KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.

  4. CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agents

    cs.LG 2026-08 conditional novelty 6.0 of 10

    AI agents that use tools remember each past step as cached tokens; CommitKV deletes a chunk only when its influence drops from high before a tool call to low after the observation returns.

  5. Addressable Memory for Video World Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Video world models can recall revisited scenes far beyond their training horizon by storing compressed memory at fixed in-distribution positions and averaging keys in a rotation-free space.

  6. Understanding Is Done Early: A Depth Division of Labor in Large Language Models and Its Use for Unbounded-Context Memory

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Long-context memory can be organized by caching intermediate residual states and recomputing only query-conditioned upper layers over a bounded retrieved pack.

  7. Raven: High-Recall Sequence Modeling with Sparse Memory Routing

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Raven is a linear-time sequence model whose sparse, input-dependent routing writes tokens into dedicated memory slots, preserving long-context recall and extrapolating 16x beyond training length.

  8. Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A DP-means allocate-on-novelty cache matches full-attention associative recall while storing only distinct items, and a minimal novelty gate recovers the rule end-to-end.

  9. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  10. CoDe-R: Refining Decompiler Output with LLMs via Rationale Guidance and Adaptive Inference

    cs.SE 2026-04 unverdicted novelty 6.0 of 10

    Rationale-guided fine-tuning plus dual-path adaptive inference lifts a 1.3B decompiler refiner to 50% average re-executability, a new lightweight SOTA on HumanEval-Decompile.

  11. Balancing Computation Load and Representation Expressivity in Parallel Hybrid Neural Networks

    cs.CL 2025-05 conditional novelty 6.0 of 10

    FlowHN splits input tokens between parallel attention and SSM branches to balance compute load, improving throughput and hardware utilization in small autoregressive language models.

  12. Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning

    cs.AI 2025-05 conditional novelty 6.0 of 10

    Enhancing a model's long-context ability before supervised fine-tuning improves math reasoning accuracy, including on short-input benchmarks.

  13. Overflow Prevention Enhances Long-Context Recurrent LLMs

    cs.LG 2025-05 conditional novelty 6.0 of 10

    OPRM, a training-free chunk-and-select inference method, improves recurrent LLMs by 14-51% on LongBench and sets a 7B-class LongBench v2 record, while raising doubts about whether recurrent models exploit long-range d...

  14. LiveLongBench: Tackling Long-Context Understanding for Spoken Texts from Live Streams

    cs.CL 2025-04 conditional novelty 6.0 of 10

    LiveLongBench, a new spoken-text long-context benchmark from live streams, shows that current long-context models and compression methods degrade on redundant speech and that hybrid compression works best.

  15. Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encoding

    cs.CL 2025-01 conditional novelty 6.0 of 10

    TAPE makes positional embeddings content-aware and equivariant, improving Transformer performance on arithmetic and long-context tasks and extending representational power to NC1-complete algorithms.

  16. Sliding Windows Are Not the End: Exploring Full Ranking with Long-Context Large Language Models

    cs.IR 2024-12 conditional novelty 6.0 of 10

    With a multi-pass teacher label construction and a position-weighted loss, a fine-tuned long-context LLM can rank 100 passages in one pass, beating sliding-window rerankers in both NDCG@10 and latency.

  17. V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding

    cs.CV 2024-12 conditional novelty 6.0 of 10

    V2PE assigns visual tokens smaller and variable positional increments than text tokens, which allows a 2B vision-language model to effectively process multimodal sequences up to 1M tokens.

  18. Selective Attention: Enhancing Transformer through Principled Context Control

    cs.LG 2024-11 conditional novelty 6.0 of 10

    Selective Self-Attention adds query- and value-dependent temperature scaling to transformer attention, improving language modeling accuracy and passkey retrieval with under 0.5% extra parameters.

  19. Maglev: Sliding Recurrent Memory

    cs.LG 2026-08 conditional novelty 5.0 of 10

    Using a full-context prefiller to write memory targets, Maglev trains a sliding-window recurrent transformer that keeps bounded memory at inference and improves language-modeling benchmarks.

  20. Curse of High Dimensionality Issue in Transformer for Long-context Modeling

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A group attention mechanism that keeps important tokens and merges the rest cuts long-context decoding cost, but the theoretical case for grouping is only weakly supported.

  21. 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    100-LongBench and LongScore evaluate LLMs at controlled context lengths and rank models by relative performance drop from a short-context baseline, not by raw accuracy.

  22. SELF: Self-Extend the Context Length With Logistic Growth Function

    cs.CL 2025-05 conditional novelty 5.0 of 10

    SELF replaces Self-Extend's fixed token grouping with a logistic-growth grouping schedule, giving mixed but sometimes large gains on long-context benchmarks.

  23. DENIAHL: In-Context Features Influence LLM Needle-In-A-Haystack Abilities

    cs.CL 2024-11 conditional novelty 4.0 of 10

    DENIAHL, a new synthetic benchmark, shows that LLM needle-in-a-haystack recall is influenced by data type, item length, and pattern structure, not just context length.

Pith tools