pith. sign in

arxiv: 2606.28560 · v1 · pith:IXPGLPLSnew · submitted 2026-06-26 · 💻 cs.CL · cs.LG

Depth-Staggered Fibonacci Spacing for Sparse Attention: Static Schedules Beat Learned Dilation and Extrapolate Where Dense Attention Fails

Chad A. Capps This is my paper

Pith reviewed 2026-06-30 01:09 UTC · model grok-4.3

classification 💻 cs.CL cs.LG
keywords sparse attentionfibonacci spacingdepth staggercontext extrapolationlanguage modelingstatic schedulesperplexity
0
0 comments X

The pith

A static per-layer stagger on Fibonacci-spaced offsets improves perplexity over fixed and learned alpha and lets sparse attention extrapolate to four times training length where dense attention collapses.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper trains 21 language models under one matched recipe and tests four schedules for the per-layer scalar alpha that sets Fibonacci offset spacing in sparse self-attention. A static stagger schedule beats both a single fixed alpha and per-layer learned alpha, and the same stagger also improves a power-of-2 base. Every sparse variant keeps perplexity nearly flat when context is extended to four times the training length, while a matched dense model sees a 201 percent rise. The authors tie the extrapolation result to the fact that fixed offsets only ever attend to relative positions that appeared during training. At training length the best sparse model still trails the dense baseline by roughly 26 percent, and the stagger gain appears uniformly rather than only at long range.

Core claim

Across the matched set of models a static per-layer stagger improves perplexity over both fixed and learned alpha, the gain holds for both Fibonacci and power-of-2 bases, learning alpha per layer adds no benefit and multiplies inference latency by about five, and all sparse variants extrapolate to four times training length with little or no degradation while the dense baseline collapses; the authors attribute the extrapolation success to fixed-offset attention querying only relative positions seen in training.

What carries the argument

Depth-staggered Fibonacci spacing, the use of a static per-layer adjustment to the scalar alpha that expands or compresses a set of Fibonacci-spaced attention offsets in addition to a dense local window.

If this is right

  • Static per-layer stagger beats both fixed alpha and learned alpha in perplexity.
  • The stagger improvement is base-agnostic and also lifts power-of-2 spacing to parity with learned Fibonacci.
  • Per-layer learning of alpha adds no accuracy and multiplies inference cost by roughly five times.
  • All sparse variants maintain performance at four times training length while a matched dense model does not.
  • The stagger benefit appears uniformly across context positions rather than only at long range.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Fixed-offset sparse patterns may support longer contexts without any need for long-sequence fine-tuning.
  • Stagger schedules could be tested on other sparse attention patterns such as local-plus-global or random offsets.
  • The uniform gain suggests the benefit is in overall attention coverage rather than targeted long-range modeling.
  • The method might be combined with other length-extrapolation techniques such as position interpolation to push context even farther.

Load-bearing premise

The extrapolation advantage holds because fixed-offset attention only ever queries relative positions that were seen during training.

What would settle it

Modify the offsets at inference time so that some queries attend to relative positions never encountered in training and measure whether perplexity at four times context length then rises sharply.

read the original abstract

We study sparse self-attention in which each query attends to a dense local window plus a set of Fibonacci-spaced offsets, with a per-layer scalar alpha that compresses or expands the spacing. Across 21 language models trained under one matched recipe (60M parameters, 512 hidden, 16 layers, 426M tokens), we compare four ways of setting alpha across depth: fixed, per-layer learned, a static linear stagger, and a coprime (anti-gridding) reassignment of that stagger, together with a reach-matched power-of-2 control. Three results stand out. First, a static per-layer stagger improves perplexity over both fixed and learned alpha, and the gain is base-agnostic: applying the same stagger to a power-of-2 base lifts it above fixed Fibonacci and to parity with learned Fibonacci attention. Second, learning per layer is inert: it does not beat the static schedule and costs roughly five times the inference latency. Third, and most consequential, all sparse variants extrapolate to four times their training length with little or no degradation, whereas a recipe-matched dense baseline collapses (perplexity rises by 201% at 4x length); we attribute this to fixed-offset attention only ever querying relative positions seen during training. We also report two honest negatives: at training length the best sparse model has about 26% higher perplexity than the dense baseline, and the staggering gain is uniform across context positions rather than concentrated at long range.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper empirically compares four schedules for setting a per-layer scalar alpha in Fibonacci-spaced sparse self-attention (fixed, per-layer learned, static linear stagger, coprime reassignment) plus a reach-matched power-of-2 control and a dense baseline. Across 21 matched 60M-parameter models trained on 426M tokens, it reports that a static per-layer stagger yields lower perplexity than fixed or learned alpha (and the gain transfers to a power-of-2 base), that learned alpha is inert and slower at inference, and that all sparse variants maintain perplexity when evaluated at 4 imes training length while a matched dense model degrades by 201%; two honest negatives are also noted (best sparse is 26% worse at training length; stagger gain is uniform across positions).

Significance. If the empirical deltas hold under replication, the work supplies a simple, static, base-agnostic recipe for sparse attention that demonstrably enables length extrapolation where dense attention fails, while documenting the remaining gap at training length. The direct head-to-head design across 21 models and the explicit reporting of negative results strengthen the practical takeaway for efficient long-context modeling.

major comments (2)
  1. [Abstract / results tables] Abstract and results: the reported perplexity improvements for the static stagger (and the 201% dense degradation at 4 imes length) are presented without error bars, standard deviations, or any measure of variance across the 21 models or multiple random seeds; this leaves open whether the observed deltas exceed training stochasticity.
  2. [Abstract / experimental setup] Abstract: exact training details (optimizer, learning-rate schedule, batch size, data mixture, and initialization) for the 21 matched models are not supplied, which is required to confirm that the recipe is truly identical across the five attention variants.
minor comments (1)
  1. [Abstract] The parenthetical mechanistic attribution for extrapolation success is offered without a dedicated control experiment; while not required for the validity of the measured deltas, it should be clearly labeled as speculative.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the positive evaluation and the constructive feedback. We address each major comment below.

read point-by-point responses

  1. Referee: [Abstract / results tables] Abstract and results: the reported perplexity improvements for the static stagger (and the 201% dense degradation at 4 imes length) are presented without error bars, standard deviations, or any measure of variance across the 21 models or multiple random seeds; this leaves open whether the observed deltas exceed training stochasticity.

    Authors: We acknowledge the value of variance estimates. All 21 models were trained from the same random seed under identical conditions to isolate the effect of the attention schedule; repeating the full suite with multiple seeds was not feasible given the compute budget. The 201% degradation for the dense baseline at 4x length is an order of magnitude larger than typical run-to-run variation in this regime, making stochasticity an implausible explanation for the reported pattern. In the revision we will add an explicit statement that all results are single-seed and will note the scale of the extrapolation failure to contextualize the absence of error bars. revision: partial

  2. Referee: [Abstract / experimental setup] Abstract: exact training details (optimizer, learning-rate schedule, batch size, data mixture, and initialization) for the 21 matched models are not supplied, which is required to confirm that the recipe is truly identical across the five attention variants.

    Authors: The abstract states that the models share 'one matched recipe,' but the referee is correct that the abstract itself does not enumerate the concrete hyperparameters. These details (AdamW optimizer, cosine schedule with linear warmup, batch size 512, data mixture, and initialization) appear in full in Section 3. We will revise the abstract to include a short parenthetical reference to the matched training recipe and a pointer to Section 3 so that readers can immediately locate the verification details. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper reports direct empirical results from training 21 matched 60M-parameter models under four alpha schedules plus a dense baseline, measuring perplexity at training length and at 4x length. No equations or derivations are present that reduce the reported metrics to fitted parameters by construction; the central claims are measured deltas between training runs. The parenthetical attribution for extrapolation is offered as a possible explanation but is not load-bearing for the validity of the observed numbers. No self-citation chains, ansatzes, or uniqueness theorems appear in the provided text.

Axiom & Free-Parameter Ledger

1 free parameters · 0 axioms · 0 invented entities

The work is purely empirical; the central claims rest on the matched training recipe across variants and the interpretation of fixed offsets, with no new mathematical axioms or postulated entities.

free parameters (1)
  • per-layer alpha
    scalar that compresses or expands Fibonacci spacing; set statically or learned per the four schedules tested

pith-pipeline@v0.9.1-grok · 5806 in / 1252 out tokens · 42186 ms · 2026-06-30T01:09:02.652626+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · 7 internal anchors

  1. [1]

    Longformer: The Long-Document Transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150,

    work page internal anchor Pith review Pith/arXiv arXiv 2004

  2. [2]

    arXiv preprint arXiv:2503.03588 , year=

    Lida Chen, Dong Xu, Chenxin An, Xintao Wang, Yikai Zhang, Jiangjie Chen, Zujie Liang, Feng Wei, Jiaqing Liang, Yanghua Xiao, and Wei Wang. Powerattention: Exponentially scaling of receptive fields for effective sparse attention.arXiv preprint arXiv:2503.03588,

    work page arXiv

  3. [3]

    Generating Long Sequences with Sparse Transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509,

    work page internal anchor Pith review Pith/arXiv arXiv 1904

  4. [4]

    Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei

    Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens.arXiv preprint arXiv:2307.02486,

    work page arXiv

  5. [5]

    TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

    Ronen Eldan and Yuanzhi Li. TinyStories: How small can language models be and still speak coherent English?arXiv preprint arXiv:2305.07759,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Measuring Mathematical Problem Solving With the MATH Dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset.arXiv preprint arXiv:2103.03874,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Dilateformer: Multi-scale dilated transformer for visual recognition.arXiv preprint arXiv:2302.01791,

    Jiayu Jiao, Yu-Ming Tang, Kun-Yu Lin, Yipeng Gao, Jinhua Ma, Yaowei Wang, and Wei-Shi Zheng. Dilateformer: Multi-scale dilated transformer for visual recognition.arXiv preprint arXiv:2302.01791,

    work page arXiv

  8. [8]

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf

    arXiv:2112.03740. Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale.arXiv preprint arXiv:2406.17557,

    work page arXiv

  9. [9]

    Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

    Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Rahimian, Manish K

    Ali K. Rahimian, Manish K. Govind, Subhajit Maity, Dominick Reilly, Christian Kümmerle, Srijan Das, and Aritra Dutta. Fibottention: Inceptive visual representation learning with diverse attention across heads.arXiv preprint arXiv:2406.19391,

    work page arXiv

  11. [11]

    RoFormer: Enhanced Transformer with Rotary Position Embedding

    10 Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.arXiv preprint arXiv:2104.09864,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    Wan, F., Zhong, L., Yang, Z., Chen, R., and Quan, X

    Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. OpenMathInstruct-2: Accelerating AI for math with massive open-source instruction data.arXiv preprint arXiv:2410.01560,

    work page arXiv

  13. [13]

    WaveNet: A Generative Model for Raw Audio

    Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio.arXiv preprint arXiv:1609.03499,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Yixing Xu, Shivank Nag, Dong Li, Lu Tian, and Emad Barsoum

    Accessed via the Hugging Facewikimedia/wikipediadataset. Yixing Xu, Shivank Nag, Dong Li, Lu Tian, and Emad Barsoum. Mswa: Refining local attention with multi-scale window attention.arXiv preprint arXiv:2501.01039,

    work page arXiv

  15. [15]

    Unveiling transformers with lego: A synthetic reasoning task.arXiv preprint arXiv:2206.04301,

    Yi Zhang, Arturs Backurs, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, and Tal Wagner. Unveiling transformers with lego: A synthetic reasoning task.arXiv preprint arXiv:2206.04301,

    work page arXiv