Pith. sign in

REVIEW 3 major objections 5 minor 26 references

A physics-derived resonant mixer can replace attention and deliver both better quality and faster long-context decoding in a matched small language model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 05:27 UTC pith:OY3H7SPS

load-bearing objection Clean dual-mode damped-resonator mixer with real small-scale quality+decode gains vs a matched transformer; efficiency is solid, quality edge is still baseline-limited. the 3 major comments →

arxiv 2607.05583 v2 pith:OY3H7SPS submitted 2026-07-06 cs.CL cs.AI

ResonatorLM: Causal Resonant Field Mixing for Efficient Long-Context Language Modeling

Archie Chaudhury This is my paper
classification cs.CL cs.AI
keywords long-context language modelingefficient decodingsequence modelingalternative to attentionresonant kernelscausal convolutionstate-space models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that language models do not need self-attention to model long sequences well. Instead of token-to-token similarity scores, ResonatorLM treats a sequence as one driven latent field and mixes context with learned damped resonators—kernels that decay and oscillate over time. The same kernels run as causal FFT convolution during training and prefill, then as fixed-size recurrent state during generation, so decode memory does not grow with context length. In a carefully matched 6M-parameter character-level WikiText setting, the model improves accuracy and perplexity over a transformer baseline while becoming faster as sequences lengthen, reaching 6.47× decode speed at 32K tokens. A sympathetic reader would care because the work offers a concrete, physics-motivated path to long-context efficiency without abandoning the usual language-model block stack.

Core claim

In a matched ~6M-parameter setting, replacing self-attention with causal resonant field mixing improves WikiText-2 character accuracy from 55.32% to 61.31% and perplexity from 4.617 to 3.764, while practical block decode speed reaches 6.47× that of an optimized transformer at 32K tokens, with train and prefill speedups also rising with sequence length.

What carries the argument

Causal resonant field mixing: each head uses a damped oscillator kernel kh[t] = exp(−αh t) cos(ωh t + ϕh), applied by causal FFT convolution for full sequences and by a fixed-size complex recurrent state for one-step decoding.

Load-bearing premise

The paper's central claim rests on the premise that quality and speed gains seen at 6M parameters on character-level WikiText and TinyStories will still matter for larger models, subword tokenizers, and real production long-context workloads.

What would settle it

Train matched ResonatorLM and transformer models at substantially larger scale (or on subword WikiText / standard long-context retrieval tasks) and check whether the accuracy lead and the decode speedup at 32K+ tokens both disappear.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Long-context language models can keep a conventional residual + MLP stack while swapping only the mixer for O(T log T) prefill and constant-size decode state.
  • Decode cost and memory need not grow linearly with context length if the mixer is a fixed-size recurrent resonator state rather than a key-value cache.
  • Learned half-lives and impulse responses can serve as interpretable diagnostics of which timescales a model actually uses.
  • Quality need not trade off against long-context efficiency: the same small ResonatorLM improves both perplexity/accuracy and practical block speedups as length grows.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the recurrent resonator state truly preserves the same dynamics as the FFT path, similar physics kernels could be dropped into other sequence stacks (vision, audio, multimodal) without redesigning the whole model.
  • The reported crossover where decode becomes faster only after a few thousand tokens suggests the method is most relevant for workloads that routinely exceed that length.
  • Head coupling and the local lexical path appear secondary; the load-bearing idea is the damped resonant kernel itself, so future work can stress-test minimal resonator-only blocks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ResonatorLM, a language-model block that replaces self-attention with causal resonant field mixing. Token sequences are treated as a driven 1-D latent field; each head uses a learned damped resonator kernel k_h[t]=exp(-α_h t) cos(ω_h t+φ_h) (Eq. 1). Full-sequence training/prefill is implemented by causal FFT convolution (O(T log T)); autoregressive decode uses a fixed-size complex recurrent state per head (Eqs. 4–5). The block also includes optional head coupling and a local causal depthwise convolution, plus a conventional RMSNorm + SwiGLU stack. In a matched ~6M-parameter character-level WikiText-2 setting (six seeds), ResonatorLM reports lower test perplexity (3.764 vs 4.617) and higher accuracy (61.31% vs 55.32%) than a transformer baseline, with practical block decode speedup rising to 6.47× at 32K tokens (Table 4) and much larger kernel-only speedups (Table 5). Ablations, transfer, longer-context, and short scale sweeps are also reported.

Significance. If the quality and efficiency results hold under stronger baselines and more realistic tokenizers/scales, the work would add a concrete, physics-motivated alternative to attention and to existing linear-time mixers. Strengths that deserve credit include: (i) a clean dual-mode design that keeps the same kernel family for FFT prefill and fixed-size recurrent decode; (ii) multi-seed matched experiments with means/stds, ablations of coupling and local path, transfer and long-context tables, and explicit physics diagnostics (causality prefix error, half-life range); (iii) a clear separation of practical block wall-clock from asymptotic kernel-tail benchmarks. The Limitations section already flags the small scale and limited baselines. The contribution is therefore best read as a solid foundational demonstration rather than a production claim.

major comments (3)
  1. §5.1 / Table 1 and Related Work (§2): The joint quality+efficiency claim is load-bearing, but quality is measured only against a matched transformer (d_model=248, 6 layers, 8 heads, 10k steps, character WikiText-2). Related Work cites S4, Hyena, and Mamba as already competitive linear-time alternatives, yet none appear as baselines. At 6M parameters and character level, a weak transformer can inflate the apparent quality edge of any structured recurrence. A head-to-head against at least one strong SSM or linear-attention baseline (same parameter budget, same protocol) is needed to support the claim that the resonant mixer itself, rather than recurrence structure in general, drives the accuracy/PPL gains.
  2. §4 Experimental Protocol and Limitations: All primary quality numbers use character (or byte) tokenization on WikiText-2/103 and TinyStories at ~6M parameters. The abstract and strongest claim present these as evidence for long-context language modeling. Character-level 6M results are informative for the mixer but are a weak proxy for subword LMs at realistic scale. Either add a subword experiment (even small) or substantially qualify the abstract/conclusion so that the central claim is scoped to the regime actually tested.
  3. Table 4 vs Table 5 and §5.3: Decode crossover and 6.47× at 32K are credible under the disclosed practical-block protocol, but the kernel-tail numbers (440× / 576×) are easy to over-read. The manuscript already separates the two protocols; the abstract and conclusion should lead exclusively with the practical-block figures and treat kernel-tail as asymptotic only, so that the efficiency claim remains proportionate to what an end-to-end block actually delivers.
minor comments (5)
  1. Eq. (1) and §3.1: State explicitly how α_h, ω_h, φ_h are parameterized (softplus/sigmoid ranges, log-spaced half-life init) in the main text, not only in prose; a short parameter table would help reproducibility.
  2. Figure 1: The block diagram is useful but low-resolution in the manuscript text; ensure labels for FFT path, recurrent state, local path, and head coupling are legible in the camera-ready version.
  3. Table 1: Training throughput favors the transformer (262k vs 173k tok/s). Mention this tradeoff in the abstract or early results so quality gains are not read as free.
  4. Reference [1] (Hugging Face forum post) is cited as independent related work; clarify date/access and whether any code or formulation was shared, to avoid novelty ambiguity.
  5. Minor typos/spacing: e.g. "Currentalternativestothetransformer", "Inaseparateexperiment", and similar concatenated words appear in the provided text; a full proofread pass is needed.

Circularity Check

0 steps flagged

No significant circularity: architecture is an explicit ansatz with learned parameters, and all quality/speed claims are empirical measurements against an external baseline.

full rationale

ResonatorLM defines a damped-resonator kernel (Eq. 1), implements it via causal FFT convolution for training/prefill and fixed-size recurrence for decode (Eqs. 2–5), and then trains the free parameters end-to-end. The reported accuracy (61.31 % vs 55.32 %), perplexity, and wall-clock speedups (Table 4) are obtained by ordinary supervised training and timed inference on held-out WikiText/TinyStories data; none of these numbers is algebraically forced by the kernel definition or by any fitted constant that is later re-presented as a prediction. Ablations (Table 2) and physics diagnostics merely verify the learned modes remain causal and multi-scale; they do not close a definitional loop. The single related-work forum citation is non-load-bearing. The paper therefore contains no self-definitional step, no fitted-input-called-prediction, and no uniqueness claim imported from self-citation. Score 0 is the correct honest finding.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 1 invented entities

The central empirical claim rests on a small set of architectural free parameters (per-head damping, frequency, phase; coupling strength; local kernel size) that are learned or set by hand, plus standard deep-learning assumptions. No new physical particles or forces are postulated; the 'resonant field' is an engineered sequence operator, not an ontological claim about language.

free parameters (4)
  • per-head damping α_h, frequency ω_h, phase ϕ_h
    Unconstrained parameters mapped via softplus/sigmoid; initialized with log-spaced half-lives and frequencies; learned during training and directly determine the mixer kernels.
  • head-coupling strength λ_c
    Scalar that scales the learned coupling matrix; set to zero in ablation; free architectural choice.
  • local causal convolution kernel size K and channel scale s
    Optional short-range path whose presence and size are chosen by the authors; ablated but still a free design parameter.
  • model width / head dimension adjustments for parameter matching
    d_model=256 vs transformer 248 chosen to keep total parameters near 6M; affects capacity comparison.
axioms (3)
  • domain assumption Causal convolution with a damped complex exponential is a valid, stable sequence mixer that can replace attention while preserving parallel training and constant-memory decode.
    Stated as the core design premise in §3; justified by analogy to physics and SSMs but not derived from first principles of language.
  • ad hoc to paper A matched ~6M-parameter character-level WikiText-2 experiment is informative about long-context language-modeling quality and efficiency.
    Primary evaluation protocol in §4–5; the paper itself notes the limited scale in Limitations.
  • domain assumption Standard residual + RMSNorm + SwiGLU block structure remains appropriate when only the mixer is replaced.
    Adopted without ablation of the surrounding stack (§3.4).
invented entities (1)
  • Causal resonant field mixer (damped resonator kernels as attention replacement) no independent evidence
    purpose: Provide O(T log T) parallel mixing and fixed-size recurrent state for decode while injecting multi-timescale oscillatory dynamics.
    The concrete kernel family kh[t]=exp(−αht)cos(ωht+ϕh) together with the dual FFT/recurrent execution path is the paper’s central technical invention; independent evidence is limited to the reported small-scale experiments and a concurrent forum post.

pith-pipeline@v1.1.0-grok45 · 14169 in / 2924 out tokens · 25418 ms · 2026-07-11T05:27:12.943746+00:00 · methodology

0 comments
read the original abstract

Contemporary language models are dominated by the transformer architecture, which leverages self-attention mechanisms to enable more efficient, parallelized training across a wide set of documents and corpora. This has allowed transformers to effectively model data across a wide range of modalities and contexts. However, transformers, along with their conventional counterparts such as recurrent neural networks (RNNs) and convolutional neural networks (CNNs), often struggle to maintain efficiency when processing long contexts. We introduce ResonatorLM, a new mechanism that replaces attention with a physics-derived alternative. ResonatorLM treats token sequences as a single, driven one-dimensional latent field and replaces attention dot products with causal functions of damped resonators. We implement ResonatorLM on a traditional network architecture and test it on standard long-context modeling tasks. We find that in a small, 6M matched setting, training and prefill speedups increase with sequence length, decode speed reaches 6.47x compared to that of a standard, optimized transformer at 32K tokens, and accuracy reaches 61.31 percent (compared to 55.32 percent) on WikiText.

Figures

Figures reproduced from arXiv: 2607.05583 by Archie Chaudhury.

Figure 1
Figure 1. Figure 1: ResonatorLM block overview. A grouped input projection feeds a resonant mixing path that uses FFT convolution during training and prefill and fixed-size recurrent state during autoregressive decoding. A local causal convolution and head coupling complement the global resonant field before the output is passed to the MLP. 3.1 Resonant Kernel Parameterization For head h and discrete lag t ≥ 0, the mixer kern… view at source ↗
Figure 2
Figure 2. Figure 2: Practical block speedup trends versus sequence length. Train and prefill speedups increase with context length, while decode transitions from slower at short context to faster at long context. 5.3 Practical Efficiency and Kernel-Tail Scaling [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 12 linked inside Pith

  1. [1]

    Badaramoni, A.: Wave field llm — o(n log n) attention via wave equation dynamics, within 5% of standard transformer. https://discuss.huggingface.co/t/wave- field-llm-o-n-log-n-attention-via-wave-equation-dynamics-within-5-of-standard- transformer/173625 (2026), hugging Face Forums post, accessed 2026-03-31

    2026

  2. [2]

    arXiv preprint arXiv:2004.05150 (2020)

    Beltagy, I., Peters, M.E., Cohan, A.: Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020)

    Pith/arXiv arXiv 2004

  3. [3]

    In: Advances in Neural Information Processing Systems (2020)

    Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems (2020)

    2020

  4. [4]

    In: Advances in Neural Information Processing Systems (2018)

    Chen, R.T.Q., Rubanova, Y., Bettencourt, J., Duvenaud, D.: Neural ordinary differential equations. In: Advances in Neural Information Processing Systems (2018)

    2018

  5. [5]

    arXiv preprint arXiv:1904.10509 (2019) 12 A

    Child, R., Gray, S., Radford, A., Sutskever, I.: Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 (2019) 12 A. Chaudhury

    Pith/arXiv arXiv 1904

  6. [6]

    In: International Conference on Learning Representations (2021)

    Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al.: Rethinking attention with performers. In: International Conference on Learning Representations (2021)

    2021

  7. [7]

    arXiv preprint arXiv:2307.08691 (2023)

    Dao, T.: Flashattention-2: Faster attention with better parallelism and work parti- tioning. arXiv preprint arXiv:2307.08691 (2023)

    Pith/arXiv arXiv 2023

  8. [8]

    In: Advances in Neural Information Processing Systems (2022)

    Dao, T., Fu, D.Y., Ermon, S., Rudra, A., Re, C.: Flashattention: Fast and memory- efficient exact attention with io-awareness. In: Advances in Neural Information Processing Systems (2022)

    2022

  9. [9]

    Eldan, R., Li, Y.: Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759 (2023)

    Pith/arXiv arXiv 2023

  10. [10]

    In: Advances in Neural Information Processing Systems (2019)

    Greydanus, S., Dzamba, M., Yosinski, J.: Hamiltonian neural networks. In: Advances in Neural Information Processing Systems (2019)

    2019

  11. [11]

    arXiv preprint arXiv:2312.00752 (2023)

    Gu, A., Dao, T.: Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023)

    Pith/arXiv arXiv 2023

  12. [12]

    Advances in Neural Information Processing Systems (2020)

    Gu, A., Dao, T., Ermon, S., Rudra, A., Re, C.: Hippo: Recurrent memory with optimal polynomial projections. Advances in Neural Information Processing Systems (2020)

    2020

  13. [13]

    In: International Conference on Learning Representations (2022)

    Gu, A., Goel, K., Re, C.: Efficiently modeling long sequences with structured state spaces. In: International Conference on Learning Representations (2022)

    2022

  14. [14]

    arXiv preprint arXiv:2203.15556 (2022)

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L.A., Welbl, J., Clark, A., et al.: Training compute- optimal large language models. arXiv preprint arXiv:2203.15556 (2022)

    Pith/arXiv arXiv 2022

  15. [15]

    arXiv preprint arXiv:2001.08361 (2020)

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D.: Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

    Pith/arXiv arXiv 2001

  16. [16]

    In: International Conference on Machine Learning (2020)

    Katharopoulos, A., Vyas, A., Pappas, N., Fleuret, F.: Transformers are rnns: Fast autoregressive transformers with linear attention. In: International Conference on Machine Learning (2020)

    2020

  17. [17]

    In: Inter- national Conference on Learning Representations (2020)

    Kitaev, N., Kaiser, L., Levskaya, A.: Reformer: The efficient transformer. In: Inter- national Conference on Learning Representations (2020)

    2020

  18. [18]

    In: International Conference on Learning Representations (2017)

    Merity, S., Xiong, C., Bradbury, J., Socher, R.: Pointer sentinel mixture models. In: International Conference on Learning Representations (2017)

    2017

  19. [19]

    In: International Conference on Machine Learning (2023)

    Poli, M., Massaroli, S., Nguyen, E.Q., Fu, D.Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., Re, C.: Hyena hierarchy: Towards larger convolutional language models. In: International Conference on Machine Learning (2023)

    2023

  20. [20]

    arXiv preprint arXiv:2211.05102 (2022)

    Pope,R.,Douglas,S.,Craik,A.,Grisly,I.,Hall,M.,Salimbeni,F.,Wu,Y.:Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102 (2022)

    Pith/arXiv arXiv 2022

  21. [21]

    arXiv preprint arXiv:1911.02150 (2019)

    Shazeer, N.: Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150 (2019)

    Pith/arXiv arXiv 1911

  22. [22]

    arXiv preprint arXiv:2002.05202 (2020)

    Shazeer, N.: Glu variants improve transformer. arXiv preprint arXiv:2002.05202 (2020)

    Pith/arXiv arXiv 2002

  23. [23]

    In: Advances in Neural Information Processing Systems (2017)

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems (2017)

    2017

  24. [24]

    arXiv preprint arXiv:2006.04768 (2020)

    Wang, S., Li, B.Z., Khabsa, M., Fang, H., Ma, H.: Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 (2020)

    Pith/arXiv arXiv 2006

  25. [25]

    arXiv preprint arXiv:2007.14062 (2020)

    Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., Ahmed, A.: Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062 (2020)

    Pith/arXiv arXiv 2007

  26. [26]

    Advances in Neural Information Processing Systems (2019)

    Zhang, B., Sennrich, R.: Root mean square layer normalization. Advances in Neural Information Processing Systems (2019)

    2019