Breaking Quadratic Barriers: A Non-Attention LLM for Ultra-Long Context Horizons

Andrew Kiruluta; Preethi Raju; Priscilla Burity

REVIEW 4 major objections 5 minor 1 cited by

Breaking Quadratic Barriers: A Non-Attention LLM for Ultra-Long Context Horizons

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims a fully attention-free LLM can match efficient transformers on standard benchmarks while scaling near-linearly to million-token contexts.

desk verdict A plausible architecture proposal undone by unverifiable empirical claims; the synthesis is real but the paper asks to be taken on faith. read the letter →

arxiv 2506.01963 v1 pith:2UMOZRFM submitted 2025-05-09 cs.LG cs.CL

Andrew Kiruluta , Preethi Raju , Priscilla Burity This is my paper · ORCID

classification cs.LGcs.CL

keywords non-attentionLLMstate-spacemodelsmulti-resolutionconvolutionrecurrentsupervisorretrieval-augmentedmemoryultra-longcontextnear-linearscalinglanguagemodeling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a language model can eliminate token-to-token self-attention entirely and still match efficient transformers on standard benchmarks, while extending to contexts of hundreds of thousands or millions of tokens. The proposed architecture splits the input into chunks, mixes tokens inside each chunk with state-space and multi-resolution convolution layers, and bridges chunks with a recurrent hidden state plus retrieval from an external memory of chunk summaries. If true, this matters because the main obstacle to ultra-long-context modeling—the quadratic cost of the $QK^\top V$ attention matrix—would be removed rather than approximated or sparsified. The paper reports 18.7 perplexity on WikiText-103 and 1.04 bits per character on Enwik8, and describes scaling experiments on sequences up to about 200,000 tokens.

What carries the argument

The central object is the chunked processing pipeline, with the chunk summary vector as the unit of global memory. Inside each chunk, an S4-inspired state-space block provides near-linear long-range mixing, and parallel dilated convolutions refine local patterns at several scales; after mean pooling, one vector per chunk becomes both the input to a recurrent supervisor and the key stored in an external key-value memory. A gated fusion combines the current chunk summary with retrieved summaries, and the resulting vector updates the recurrent hidden state that carries coherence across chunks. This machinery carries the argument because no step ever forms a pairwise token interaction, so total cost scales with the number of chunks rather than with the square of the token count.

What would settle it

Train the model on long synthetic documents in which a unique token appears exactly once and must be recalled more than one chunk later, then measure exact-match recall in next-token prediction as the chunk size grows. If recall collapses as $c$ increases or falls well below a same-size token-level attention model, the claim that one pooled vector per chunk plus top-$k$ retrieval preserves long-range information is falsified.

Watch

Extended reading notes

Core claim

The central claim is that a non-attention architecture can serve as a general-purpose language model, not merely a long-context special case. The defining move is to never construct an $n\times n$ token-to-token attention map: each chunk is transformed by state-space kernels inspired by S4 and by dilated convolutions at multiple resolutions, then pooled into a single chunk embedding; a recurrent supervisor updates one global hidden state per chunk, and a retrieval-augmented memory returns a small number of past chunk summaries that are gated into the current representation. Token-level predictions come from an LM head applied to the locally mixed representations, with no attention anywhere in the pipeline. The paper argues that because every component costs $O(c)$ or $O(c\log c)$ per chunk of size $c$, the total cost is $O(n)$ or $O(n\log c)$, and it presents benchmark numbers intended to show that removing attention does not cost quality.

Load-bearing premise

The load-bearing premise is that compressing each chunk to one compact summary vector, then passing only that vector and a few retrieved summaries across chunk boundaries, preserves enough information that removing token-to-token attention does not degrade language modeling on long contexts.

Editorial extensions

If this is right

With the paper's reported numbers, a roughly 120M-parameter attention-free model reaches 18.7 perplexity on WikiText-103 and 1.04 bpc on Enwik8, which it presents as competitive with or better than efficient transformer baselines.
Context length stops being the main driver of memory: the active training and inference footprint is set by chunk size and batch size rather than by total tokens, so contexts of 100,000 to 1,000,000 tokens become feasible on a single GPU.
The paper reports stable chunked training and accurate recall of patterns planted up to 100,000 tokens earlier in synthetic and book-length inputs, attributing that recall to the recurrent supervisor and retrieval memory.
Because there is no attention, the effective context is bounded by the external memory index and by chunked decoding rather than by a fixed window, so processing an entire corpus becomes one pass instead of a pipeline of overlapping slices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: if the claim holds, the practical ceiling shifts from model capacity to the retrieval index; scaling to millions of tokens becomes primarily an engineering problem of chunk summarization and nearest-neighbor search.
Editorial extension: a direct test of the information bottleneck would vary chunk size $c$ at fixed total context. If the recurrent supervisor truly compresses history, perplexity should degrade smoothly as $c$ grows; a sharp collapse would point to the per-chunk pooling step as the operative limitation.
Editorial extension: the external memory could be read hierarchically—first retrieving whole chunks, then local windows within them—which might recover token-level precision that a single pooled vector loses while keeping near-linear cost.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a coherent proposal for a non-attention LLM (S4-style SSM + dilated convs + GRU chunk supervisor + FAISS retrieval), and the complexity analysis is standard and true by construction. But the paper is not a position piece: it reports trained-model numbers (18.7 PPL WikiText-103, 1.04 bpc Enwik8) with no code, no data, no logs, and a GitHub link that is not verifiable from the manuscript. The empirical core is unsupported, and the baseline table doesn't line up with published numbers (GPT-2 small at 20.5 PPL on WikiText-103 is off from the commonly cited ~29). So the headline result—competitive quality at near-linear cost—cannot be checked.

What's genuinely here: the four-component combination is a legitimate synthesis, and the authors are honest that each part has precedent. The chunked architecture has a real chance of avoiding quadratic blowups, and the per-chunk cost analysis is fine as far as it goes. The ablation logic (retrieval helps cross-chunk recall, RNN helps coherence) is sensible, but it is reported as narrative, not as numbers anyone can inspect.

The soft spots run deeper than missing code. The load-bearing assumption is that one GRU hidden state per chunk plus top-1 or top-2 retrieved pooled chunk embeddings is enough cross-chunk information flow to support the 100k-token recall claims in Section 7.6. The paper offers no capacity bound, no analysis, and no released experiment to show that. The training-memory claim of O(B*c*d) also glosses over backprop through the recurrent supervisor, which needs per-chunk hidden states stored; the real footprint is O(B*c*d + M*B*h) or similar. That's not a fatal flaw in the design, but it is a hole in the stated complexity.

The writing is clear, the related work is mostly on point, and the authors avoid overclaiming novelty at the component level. If they released code, checkpoints, and a corrected experimental table, the architecture would be worth a look. As submitted, the central claim rests on assertion. I'd advise engaging with it only to extract the proposal and complexity sketch, not to trust the numbers. A serious editor should desk-reject pending evidence; there is not enough reproducible content here to justify referee time on the empirical claims.

Referee Report

4 major / 5 minor

Summary. The paper proposes a non-attention LLM architecture for ultra-long contexts. The input is split into chunks of length c; each chunk is processed by an S4-inspired state-space block and multi-resolution dilated convolutions, then pooled into a single chunk embedding. Cross-chunk information is carried by a GRU-based recurrent supervisor and by an external memory that retrieves the top-k most similar pooled chunk embeddings. The authors claim this avoids the QK^T V operation entirely, achieves O(n) or O(n log c) complexity in sequence length n, and report competitive perplexity on WikiText-103 (18.7 PPL) and Enwik8 (1.04 bpc), with qualitative claims of successful recall across up to 100,000 tokens and scaling to hundreds of thousands of tokens.

Significance. If the empirical claims were substantiated, the paper would present a useful modular alternative to attention for long-context language modeling, combining SSM blocks, multi-scale convolution, recurrence, and retrieval. The complexity analysis is transparent and the near-linear scaling is true by construction for chunked O(c) blocks. However, the central contributions depend on experimental results that are not reproducible from the manuscript, and the claimed training memory footprint omits a component that grows with the number of chunks. The paper is therefore best read as a conceptual architecture proposal; as a research contribution with quantitative claims, it is not currently supported.

major comments (4)

[§7.5, Table 2] The two headline numbers, 18.7 PPL on WikiText-103 and 1.04 bpc on Enwik8, are not backed by reproducible evidence. No training configuration, data split description, seed, hyperparameter sweep, or evaluation log is provided; the GitHub URL in §8 is not accompanied by code, checkpoints, or a README in the manuscript. In addition, the baseline values in Table 2 are not sourced: the GPT-2 Small entry of 20.5 PPL on WikiText-103 is not a standard published number for a 124M GPT-2 on that dataset, and BigBird's 19.2 PPL is reported without a training setup. These inconsistencies make the claim of 'competitive or superior' performance impossible to verify.
[§4.10, §4.5, §4.16] The claimed training memory footprint of approximately O(B*c*d) ignores the recurrent supervisor's backpropagation cost. Equation (12) and Equation (19) update the global hidden state once per chunk, so training through M = n/c chunks requires storing that hidden state at each chunk boundary (or using truncated BPTT, which is not stated). The additional memory is O(B*M*h) and the additional compute is O(M*h^2); for the million-token scenario with c = 1024, M is roughly 1000, and this is not a negligible overhead. The paper should state the actual training-time memory and specify whether full BPTT, truncated BPTT, or gradient checkpointing is used.
[§4.4, §4.5, §4.16, §7.6] The long-context capability rests on an unexamined information bottleneck. All cross-chunk information must pass through a single GRU hidden state updated once per chunk, plus top-1 or top-2 pooled chunk embeddings retrieved from external memory (§7.4). The paper provides no capacity analysis, information-theoretic bound, or controlled experiment demonstrating that this compressed channel is sufficient to maintain coherent dependencies across 100k+ tokens. The claim in §7.6 of recalling a pattern introduced 100,000 tokens earlier is anecdotal: no task definition, accuracy numbers, or comparison protocol is given.
[§7.6, §8, Abstract] The abstract and conclusion claim scaling to 'hundreds of thousands to potentially millions of tokens,' but the described experiments cover sequences of 50,000 to 200,000 tokens, with no scaling plot, wall-clock measurements beyond a coarse relative statement, or peak-memory curve. The million-token figure is explicitly speculative ('potentially'), so the central scalability claim is not empirically established at the stated scale.

minor comments (5)

[General] Equation numbers restart between Section 3 and Section 4: Equations (1)-(5) are reused as (6)-(20). This makes cross-references such as 'Eq. 4' and 'Eq. 12' ambiguous and should be fixed with continuous numbering.
[§7.5] The text says 'Table 1 summarizes the test perplexity...' but the actual results table is Table 2; Table 1 in §2.9 is a qualitative comparison table and does not report the benchmark numbers claimed in the sentence.
[Table 2] The entry 'Proposed (SSM+Conv 120M 32k (chunk))' is unclear because §7.4 reports chunk sizes of 1024, 2048, or 4096 tokens; the meaning of '32k (chunk)' should be clarified.
[§2.1, Reference [11]] The related-work discussion of 'DeepSeek LLM' cites DeepSeek-R1 (arXiv:2501.12948), which is a reasoning-model report and not an architecture paper on long-context extension. The citation does not support the claims made about DeepSeek's chunk-processing and caching mechanisms.
[§6.4, References] The FAISS citation is given as 'Johnson et al., 2017' but has no entry in the reference list; similarly, 'Wang et al., 2020' is used in the text for Linformer while the reference list item [21] is incomplete. All citations should be matched to complete bibliography entries.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the complexity claim is a deductive design property and the perplexity results are empirical claims, not fitted-input predictions.

full rationale

The paper's derivation chain is self-contained in the sense relevant to circularity. The near-linear complexity claim is not a separately fitted prediction or an empirical output disguised as an input; it follows directly from the explicit design equations: chunking into M = ceil(n/c) chunks (Sections 3 and 4.1), per-chunk S4/convolution costs of O(c) or O(c log c) (Eqs. 6-8), a per-chunk RNN update (Eqs. 12 and 19), and sublinear FAISS retrieval (Eq. 3). If one grants the stated component costs, the total O(n log c) or O(n) scaling is a deductive consequence of the architecture, not a circular restatement of an assumed conclusion. The WikiText-103 and Enwik8 numbers (Table 2) are reported measurements, not quantities fitted to one subset and then re-issued as predictions of a closely related quantity. No parameter-fitting loop appears anywhere in the paper. There are no load-bearing self-citations: the references to S4, RWKV, Transformer-XL, and related work are external prior art, and no uniqueness theorem or author-imported constraint is invoked to force the design choice. The most serious concerns with the paper are empirical credibility and reproducibility - the baseline numbers are nonstandard or inconsistent (e.g., GPT-2 small listed at 20.5 PPL on WikiText-103) and no code or checkpoints are verifiable - but those are correctness and evidence issues, not circularity. Per the review rules, under-verification and inconsistency with published baselines should not be converted into a circularity finding. Therefore the honest circularity score is 0, with no circular steps identified.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The ledger shows the architecture is a synthesis of existing modules. The scaling result follows by construction from chunk size c and O(c) blocks; the only genuinely new combination relies on two empirical assumptions, namely that the recurrent state and top-k retrieval replace attention's information routing. Those assumptions are not validated by released artifacts.

free parameters (5)

Chunk size c = 1024, 2048, or 4096
Sets the granularity of all cross-chunk communication; the O(n log c) claim and the amount of information retained per chunk depend on it.
State-space kernel size = 16 or 32
Defines the receptive field of the depthwise-convolution approximation to the S4 layer.
Number of dilation branches K = 2 to 3, with factors [1, 2, 4]
Chosen by hand in Section 7.4; no ablation is reported to justify the choice.
GRU hidden size = 512
Size of the global recurrent supervisor; determines how much cross-chunk state can be carried.
Top-k retrieved vectors = 1 or 2
Number of chunk summaries fused into each chunk; controls how much historical context is used.

assumptions (4)

domain assumption A single GRU state plus top-1 or top-2 retrieved chunk embeddings carries enough global context for coherent long-sequence modeling.
Central to the architecture; introduced in Section 3.4 and Sections 4.15 to 4.16. No theoretical or released empirical evidence supports it.
domain assumption The S4-like operation can be reduced to depthwise convolution with fixed kernels without losing the long-range modeling covered by the state-space equations.
The paper calls this 'in practice' in Sections 3.1 and 4.2, but provides no approximation-error analysis.
domain assumption Approximate nearest-neighbor lookup in FAISS over pooled chunk embeddings returns semantically relevant prior chunks.
Needed for the retrieval branch in Section 4.4; retrieval quality is asserted, not measured.
domain assumption WikiText-103 and Enwik8 perplexity and bpc are appropriate evidence for the ultra-long-context claim.
These benchmarks are dominated by local statistics; the paper does not analyze their sensitivity to long-range context.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Breaking Quadratic Barriers: A Non-Attention LLM for Ultra-Long Context Horizons." pith.science (2026). https://pith.science/paper/2UMOZRFM

@misc{pith2026250601963,
  author       = {Pith},
  title        = {Pith review of: Breaking Quadratic Barriers: A Non-Attention LLM for Ultra-Long Context Horizons},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2UMOZRFM}},
  note         = {Machine review of arXiv:2506.01963}
}

read the original abstract

We present a novel non attention based architecture for large language models (LLMs) that efficiently handles very long context windows, on the order of hundreds of thousands to potentially millions of tokens. Unlike traditional Transformer designs, which suffer from quadratic memory and computation overload due to the nature of the self attention mechanism, our model avoids token to token attention entirely. Instead, it combines the following complementary components: State Space blocks (inspired by S4) that learn continuous time convolution kernels and scale near linearly with sequence length, Multi Resolution Convolution layers that capture local context at different dilation levels, a lightweight Recurrent Supervisor to maintain a global hidden state across sequential chunks, and Retrieval Augmented External Memory that stores and retrieves high-level chunk embeddings without reintroducing quadratic operations.

Figures

Figures reproduced from arXiv: 2506.01963 by the authors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Wavelet Logic Machines: Learning and Reasoning in the Spectral Domain Without Neural Networks
cs.LG 2025-07 reject novelty 4.0 of 10

The paper claims that a fully spectral wavelet-domain model can reach near-Transformer accuracy on GLUE tasks while using 72% fewer parameters and no attention or convolution layers.

Reference graph

Works this paper leans on

21 extracted references · 15 canonical work pages · cited by 1 Pith paper

[1]

Radford and et al

A. Radford and et al. Language models are unsupervised multitask learners. OpenAI blog, 2019

work page 2019
[2]

Devlin, M.-W

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of NAACL-HLT, 2019

work page 2019
[3]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwi...

2020
[4]

Gpt-4 technical report.https://openai.com/research/gpt-4, 2023

OpenAI. Gpt-4 technical report.https://openai.com/research/gpt-4, 2023

work page 2023
[5]

Gomez, Łukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems, volume 30, pages 5998–6008, 2017. 35

work page 2017
[6]

Zaheer and et al

M. Zaheer and et al. Big bird: Transformers for longer sequences. InNeurIPS, 2020

work page 2020
[7]

Beltagy, M

I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv:2004.05150, 2020

arXiv 2004
[8]

Choromanski and et al

K. Choromanski and et al. Rethinking attention with performers. InInternational Confer- ence on Learning Representations (ICLR), 2021

work page 2021

Show all 21 references

[9]

Z. Dai, Z. Yang, Y . Yang, J. Carbonell, Q. V . Le, and R. Salakhutdinov. Transformer- xl: Attentive language models beyond a fixed-length context. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019

2019
[10]

J. W. Rae, A. Potapenko, S. M. Jayakumar, and T. Lillicrap. Compressive transformers for long-range sequence modelling. InInternational Conference on Learning Representations (ICLR), 2020

2020
[11]

Guo and et al

B. Guo and et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv:2501.12948, 2025

2025 arXiv
[12]

Lee, M.-W

K. Lee, M.-W. Chang, and K. Toutanova. Latent retrieval for weakly supervised open domain question answering. InACL, 2019

2019
[13]

Retrieval-augmented generation for knowledge-intensive nlp tasks

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Vladimir Karpukhin, Naman Goyal, Hein- rich Mohiuddin, Wen-tau Yih Wu, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. InAdvances in Neural Information Processi...

2020
[14]

Borgeaud and et al

M. Borgeaud and et al. Improving language models by retrieving from trillions of tokens. InInternational Conference on Machine Learning (ICML), 2022. DeepMind RETRO

2022
[15]

Hochreiter and J

S. Hochreiter and J. Schmidhuber. Long short-term memory.Neural Computation, 1997

1997
[16]

Rwkv-lm: Reinventing rnns for the transformer era.https://github

BlinkDL. Rwkv-lm: Reinventing rnns for the transformer era.https://github. com/BlinkDL/RWKV-LM
[17]

van den Oord and et al

A. van den Oord and et al. Wavenet: A generative model for raw audio. InSSW, 2016

2016
[18]

Kalchbrenner and et al

J. Kalchbrenner and et al. Neural machine translation in linear time. arXiv:1610.10099, 2016

2016 arXiv
[19]

Gu and et al

J. Gu and et al. Combining recurrent, convolutional, and continuous-time models with structured state spaces (s4). arXiv preprint arXiv:2111.00396, 2021

2021 arXiv
[20]

Dao and et al

M. Dao and et al. S5: A unified framework for state-space approximation. arXiv preprint arXiv:2210.09467, 2022

2022 arXiv
[21]

S. Wang, Y . Tay, L. Liu, and et al. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020. 36

2006 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Radford and et al

A. Radford and et al. Language models are unsupervised multitask learners. OpenAI blog, 2019

work page 2019

[2] [2]

Devlin, M.-W

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of NAACL-HLT, 2019

work page 2019

[3] [3]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwi...

2020

[4] [4]

Gpt-4 technical report.https://openai.com/research/gpt-4, 2023

OpenAI. Gpt-4 technical report.https://openai.com/research/gpt-4, 2023

work page 2023

[5] [5]

Gomez, Łukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems, volume 30, pages 5998–6008, 2017. 35

work page 2017

[6] [6]

Zaheer and et al

M. Zaheer and et al. Big bird: Transformers for longer sequences. InNeurIPS, 2020

work page 2020

[7] [7]

Beltagy, M

I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv:2004.05150, 2020

arXiv 2004

[8] [8]

Choromanski and et al

K. Choromanski and et al. Rethinking attention with performers. InInternational Confer- ence on Learning Representations (ICLR), 2021

work page 2021