Pith. sign in

REVIEW 4 major objections 6 minor 28 references

Maglev: Sliding Recurrent Memory

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A sliding-window decoder, taught by a stronger prefiller's memory states, beats sliding-window and latent-recurrent baselines on language modeling.

desk verdict A well-run, honest study of a modest architecture trick; the closed-loop drift gap is real but mainly a longer-context question, and the paper deserves a serious referee. read the letter →

arxiv 2608.02870 v2 pith:IKB4PV6K submitted 2026-08-03 cs.LG

Bo Liu , Qiang Liu This is my paper · ORCID
classification cs.LG
keywords recurrenttransformersliding-windowattentionmemoryconsistencyparalleltraininglanguagemodelingfixed-sizeK/Vinjectionprefiller
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Maglev tries to give Transformers a fixed-size, token-wise recurrent memory without giving up parallel pretraining. The trick is to train two coupled models: a stronger prefiller that sees the full context and, in one parallel pass, writes a target memory at every token; and a decoder that sees only a sliding window plus the previous token's memory, and is trained to both predict the next token and reproduce the prefiller's memory. A consistency loss aligns the two memories, so at inference the prefiller is discarded and the decoder runs recurrently on its own memories at the same cache cost as sliding-window attention. The paper reports that this decoder, at 435M parameters and 43.52B training tokens, improves FineWeb-Edu validation bits-per-byte from 0.7413 to 0.7251 and average downstream accuracy from 54.1 to 56.4 over a matched sliding-window transformer, while also beating a latent recurrent transformer. Sharing parameters between prefiller and decoder preserves most of the gain.

What carries the argument

The central mechanism is lifted parallel training: a prefiller $Q$ with full-attention access constructs an auxiliary memory trajectory $m'_{1:T}$ in one causal, sequence-parallel pass; a sliding-window decoder $P$ consumes the shifted trajectory, predicts tokens, and emits its own memories $m_t$; and a consistency loss $\mathcal{L}_{\mathrm{cons}} = \lambda\|m_t - m'_t\|_2/\sqrt{d}$ teaches $P$ to reproduce the prefiller's memory so that at inference its own recurrent memories can substitute. The recurrence is carried by gated K/V injection: shifted memories are projected into key/value features, gated with local token features, and fed into the decoder's sliding-window attention, so the bounded KV cache and attention cost match ordinary sliding-window attention while the memory carries information from beyond the window.

What would settle it

Run the trained decoder in closed loop on a sequence much longer than the 2048-token training window, e.g., 10,000 tokens, and measure both the memory drift $\|m_t - m'_t\|$ (available by also running the prefiller as an oracle) and the language-modeling perplexity over the tail. If the drift grows without bound and the accuracy degrades toward the sliding-window baseline, the loop-closure claim fails.

Watch

Extended reading notes

Core claim

The central claim is that a decoder $P$ that uses only sliding-window attention with recurrent K/V injection—no full attention at inference—can outperform both a matched sliding-window transformer and a latent recurrent transformer on the same training budget. Training lifts the sequence-parallel problem by having a stronger prefiller $Q$ produce memory targets $m'_t$ for every position in one causal pass; $P$ then predicts tokens from the shifted targets while being penalized by $\lambda\|m_t - m'_t\|_2/\sqrt{d}$, aligning its own memories with the prefiller's. At inference $Q$ is removed and $P$ feeds its own memories back through the same K/V pathway, so the deployed model is a bounded-memory recurrent transformer. The best separate-parameter variant reaches $0.7251$ FineWeb-Edu BPB and $56.4$ average downstream accuracy, versus $0.7413$ and $54.1$ for the sliding-window baseline; the shared-parameter variant keeps most of the improvement.

Load-bearing premise

The load-bearing premise is training–inference loop closure: the consistency loss $\lambda\|m_t - m'_t\|_2/\sqrt{d}$ aligns decoder memories with prefiller memories only under teacher forcing, and the paper assumes this alignment keeps the decoder's own recurrent trajectory stable in closed-loop inference; it does not measure the actual drift or test beyond the 2048-token training context.

Editorial extensions

If this is right

  • Inference stays at fixed cost: the KV cache and attention pattern are those of ordinary sliding-window attention, independent of sequence length, because the recurrent memory is injected through existing K/V features.
  • The decoder trains with two parallel Transformer passes instead of a sequential unroll, preserving large-scale pretraining throughput while still yielding a recurrent model at deployment.
  • On the same token budget, the consistency-trained decoder improves both validation loss and several downstream commonsense benchmarks over matched sliding-window and latent-recurrent baselines.
  • Parameter sharing between the prefiller and decoder removes much of the extra parameter cost while keeping most of the gain, so the method can be implemented without doubling the model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the closed-loop alignment remains stable far beyond the 2048-token training context, the same scheme could offer a practical route to unbounded-context language modeling at fixed memory cost; a direct drift measurement would settle that.
  • The prefiller need not be trained from scratch: a pretrained model could supply memory targets, recasting Maglev as a distillation method that compresses a strong model's representations into a compact recurrent decoder.
  • Alternative memory-injection schemes (residual-stream injection, recurrent tokens, or cross-attention) and stronger consistency objectives (e.g., contrastive or adversarial) might reduce the teacher-forcing gap further; the paper leaves those comparisons open.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Maglev, a recurrent Transformer with fixed-size memory. A prefiller Q with full attention processes the whole sequence in parallel to produce memory targets m'_t; a decoder P with sliding-window attention and recurrent K/V injection consumes shifted targets, predicts next tokens, and produces its own memories m_t. Training minimizes a next-token loss plus a consistency loss aligning m_t with m'_t, so that at inference Q can be discarded and P runs recurrently. The authors evaluate on the nanochat d20 setup with 43.52B tokens, reporting FineWeb-Edu BPB and several downstream benchmarks. The best separate-parameter Maglev reaches 0.7251 BPB and 56.4 average downstream accuracy, improving over sliding-window (SSSS) and latent recurrent transformer (LRT) baselines. The paper is explicitly preliminary and identifies scaling and sharing patterns as future work.

Significance. If the closed-loop consistency assumption holds, Maglev offers a practical way to train a fixed-memory nonlinear recurrent transformer with parallel pretraining, with potential advantages in bounded inference cost. The experiments are unusually controlled for a pretraining paper: all models share the nanochat stack, tokenizer, data, sequence length, and optimizer, and the architectural delta is precisely specified. The paper also honestly discusses limitations and acknowledges it is preliminary. However, the central empirical claim depends on an unverified train/inference consistency, and no long-context evaluation is provided. The strengths (controlled comparison, explicit architecture) are significant, but the missing evidence is currently load-bearing.

major comments (4)
  1. [Section 2.1/2.3, Eqs. (3), (5), (10)] The training objective aligns decoder memories m_t with prefiller targets m'_t under teacher forcing, where P conditions on the shifted prefiller trajectory m'_{0:T-1}. At inference, P instead conditions on its own preceding memories m_{0:T-1}. This changes the input distribution at every recurrent K/V injection (Eqs. 6-8) and for the attention cache (Eq. 9). Pointwise L2 alignment is neither necessary nor sufficient to guarantee that the closed-loop trajectory remains close to the teacher-forced trajectory; small per-step mismatches can compound through the nonlinear recurrence. The paper reports no measurement of ||m_t - m'_t|| during inference, no open-loop versus closed-loop comparison, and no evaluation beyond the 2048-token training context. Because the headline numbers in Table 1 are claimed for P alone, this missing stability evidence is load-bearing.
  2. [Section 4, Evaluation] All evaluations operate on contexts within or below the 2048-token training sequence (LAMBADA, HellaSwag, PIQA, etc.). The architectural motivation of Maglev is to extend memory beyond the fixed window via recurrence, but no experiment shows behavior on sequences longer than 2048 tokens. Without such an experiment, the observed improvement over SSSS could be due solely to the added recurrent K/V pathway and consistency objective, not to genuine long-range memory. At minimum, the authors should add a long-context language modeling evaluation (e.g., 8k or 32k contexts) comparing Maglev against baselines.
  3. [Section 4, Experimental setup] The experimental section does not state the inference protocol used to obtain the numbers in Table 1. In particular, it is not specified whether the reported BPB and downstream accuracies are produced by P alone under the recurrent inference of Eq. (10) or by the two-pass training procedure of Eq. (3). This distinction is central to the paper's contribution, and the paper must report which procedure was used and, ideally, report both open-loop and closed-loop numbers.
  4. [Section 4, Training setup] Maglev training requires two sequence-parallel passes (one for Q, one for P) and a consistency loss, while the baselines are presumably trained with a single forward/backward per token. The paper states that all models use 'the same token budget,' but it does not report training FLOPs, wall-clock time, or parameter counts for the separate-parameter variant. Since the best result (0.7251 BPB, 56.4 average) comes from the separate-parameter model, the improvement may in part reflect additional training compute and capacity. The comparison should be placed on a compute-matched footing, or FLOPs and parameter counts should be reported so the reader can judge the trade-off.
minor comments (6)
  1. [Table 1] All configurations are single runs with no error bars or significance tests. Several adjacent scores (e.g., 56.2 vs 56.4, 56.0 vs 56.2) are within a few tenths of a percent, so the ranking among the better models may not be robust.
  2. [Section 2.2] The parameter-sharing description ('share Transformer blocks, with separate residual-scaling parameters') is vague; the number of shared layers and the resulting parameter count for each variant should be stated.
  3. [Section 4, Training setup] The paper does not report key training hyperparameters (learning rate, warmup, weight decay, dropout) beyond the optimizer name, and no code is linked. This reduces reproducibility.
  4. [Figure 2] Figure 2 is referenced with a caption describing training dynamics, but no figure image is included in the submitted text; the figure should be restored or removed.
  5. [References] The two Liu et al. entries marked 2024a and 2024b appear to refer to the same paper (Longhorn) and should be merged or disambiguated.
  6. [Section 2.1, Eq. (4)] The temperature-scaled softmax and tanh clipping in Eq. (4) are not explained; please state whether this follows nanochat and what effect it has.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are evaluated on external benchmarks and the consistency training is a genuine training objective, not a fitted prediction.

full rationale

Maglev's core derivation is self-contained in the sense required by the circularity test. The decoder P is trained with a next-token cross-entropy loss plus a consistency loss L_cons (Eq. 5) that aligns the decoder's own memory m_t with the prefiller target m'_t. The prefiller trajectory m'_t is produced by a trained causal model Q from the training sequence, not from any test-set quantity, and the consistency weight λ is varied (0.1 and 1.0) with both positive and negative effects reported in Table 1. The reported results—FineWeb-Edu validation BPB, LAMBADA, PIQA, HellaSwag, WinoGrande, ARC, SocialIQA, BoolQ—are standard external benchmarks evaluated after a fixed 43.52B-token training budget, with matched baselines and identical data and evaluation scripts. There is no equation in which a claimed improvement reduces by construction to a fitted parameter, and no load-bearing result is imported from the authors' own prior work: the citations to Longhorn (Liu et al. 2024a,b) appear only in related-work context and do not justify any architectural or empirical claim. The architectural choice of recurrent K/V injection follows the concurrently cited latent recurrent transformer (Huang et al. 2026), which is external to this paper, and the maglev contribution is the prefiller-consistency training scheme, not a renamed known result. The paper's acknowledged limitation—that inference closes the loop with the decoder's own memories while training uses prefiller memories—is a genuine generalization/robustness concern about closed-loop drift, but it is a correctness risk, not circularity; the paper does not claim the closed-loop trajectory is identical to the teacher-forced trajectory, and it does not use that assumption as a definitional substitute for its empirical results. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

Maglev introduces no new physical or mathematical entities; the prefiller and decoder are architectural components. The central claim rests on training assumptions: that a full-context teacher provides good memory targets and that an L2 consistency objective closes the train-inference loop. These are modeling premises, not verified facts, and the paper provides no dedicated experiment to validate the loop-closure premise.

free parameters (3)
  • consistency weight λ = 0.1 and 1.0
    Grid-selected consistency weight. The best value depends on whether Q and P share parameters: λ=1 helps the separate-parameter model but hurts the shared model on downstream tasks, so the method's benefit is sensitive to this tuning.
  • window size W = 512 tokens
    Chosen to match the nanochat d20 setup. It is not fitted to benchmarks but is also not systematically varied, so the effect of window size on the reported gains is unmeasured.
  • prefiller attention pattern = SLSL (interleaved full and sliding-window)
    The paper states that interleaved full and sliding-window attention for Q performs better than pure full attention (footnote 1). This is a design choice introduced by hand, with no ablation shown.
assumptions (4)
  • domain assumption The L2 consistency loss in Eq. (5) generalizes from teacher-forced training to the closed-loop trajectory where decoder memories replace prefiller memories.
    At training P consumes prefiller memories m'_{t-1}; at inference (Eq. 10) it consumes its own m_{t-1}. The paper relies on this transfer but provides no drift analysis or long-horizon stability check.
  • ad hoc to paper The prefiller Q is more expressive than P and its causal memory targets m' are suitable supervision for P.
    Footnote 1 asserts this as an essential requirement. No separate evidence establishes that L2-aligned final states preserve the semantic content needed for next-token prediction in a recurrent loop.
  • standard math Standard Transformer attention, backpropagation, and the nanochat pretraining pipeline are reliable.
    The method builds on ordinary transformer machinery, softmax attention, RMS normalization, and a public training stack; these are accepted background tools.
  • domain assumption Sliding-window attention with W=512 plus shifted memory K/V injection carries enough information for the downstream tasks at sequence length 2048.
    The window and memory mechanism are not ablated, and generalization to longer sequences is not tested, so the sufficiency of the 512-token window is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Maglev: Sliding Recurrent Memory." pith.science (2026). https://pith.science/paper/IKB4PV6K

@misc{pith2026260802870,
  author       = {Pith},
  title        = {Pith review of: Maglev: Sliding Recurrent Memory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IKB4PV6K}},
  note         = {Machine review of arXiv:2608.02870}
}
abstract

We introduce \ours{}, a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. \ours{} consists of two coupled models: a prefiller $Q$, which leverages full attention\footnote{In practice, we use interleaved full and sliding-window attention for $Q$, as this yields stronger performance. The essential requirement is that $Q$ be more expressive than $P$, with access to the full history.} to produce memory targets $m'_t$, and a decoder $P$, which uses only sliding-window attention and recurrent K/V injection to produce decoder memories $m_t$ for next-token prediction. We train \ours{} with a memory consistency loss that aligns $m_t$ with $m'_t$, allowing inference to use $P$ alone. Empirically, \ours{} improves validation loss and downstream pretraining benchmarks over sliding-window and latent recurrent transformer baselines. Moreover, sharing parameters between $P$ and $Q$ reduces parameter memory while preserving most of the gains.

Figures

Figures reproduced from arXiv: 2608.02870 by the authors.

Figure 1
Figure 1. Maglev: a prefiller Q produces memory targets m′ t from the observed sequence, and a decoder P consumes the shifted m′ t−1 while predicting xt+1 and producing its own memory mt. The consistency loss aligns mt with m′ t . At inference, the prefiller is discarded and the decoder runs recurrently using its own memories. In practice, we find that we could largely share parameters between P and Q without degrading much p… view at source ↗
Figure 2
Figure 2. Training dynamics for the Maglev consistency objective. The prefiller [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 2 canonical work pages

  1. [7]

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser

    URLhttps://arxiv.org/abs/2402.19427. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. InInternational Conference on Learning Representations,

    arXiv

  2. [10]

    Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein

    URLhttps://arxiv.org/abs/2212.14052. Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach.Advances in Neural Information Processing Systems, 38: 41340–41391,

    arXiv

  3. [13]

    David Herel and Tomas Mikolov

    URLhttps://arxiv.org/abs/2412.06769. David Herel and Tomas Mikolov. Thinking tokens for language modeling.arXiv preprint arXiv:2405.08644,

    arXiv

  4. [14]

    Sepp Hochreiter and Jürgen Schmidhuber

    URLhttps://arxiv.org/abs/2405.08644. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory.Neural Computation, 9(8): 1735–1780,

    arXiv

  5. [16]

    Zeyi Huang, Xuehai He, LiLiang Ren, Yiping Wang, Baolin Peng, Hao Cheng, Shuohang Wang, Pengcheng He, Jianfeng Gao, Yong Jae Lee, et al

    URL https://arxiv.org/abs/2203.15556. Zeyi Huang, Xuehai He, LiLiang Ren, Yiping Wang, Baolin Peng, Hao Cheng, Shuohang Wang, Pengcheng He, Jianfeng Gao, Yong Jae Lee, et al. Latent recurrent transformer: Architecture exploration, training strategies, and scaling behavior.arXiv preprint arXiv:2605.26797,

    arXiv

  6. [18]

    Albert Q

    URLhttps://arxiv.org/abs/2404.09173. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Saye...

    arXiv

  7. [19]

    Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein

    URLhttps://arxiv.org/abs/2310.06825. Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. https://kellerjordan. github.io/posts/muon/,

    arXiv

  8. [21]

    Pretraining Recurrent Networks without Recurrence

    URLhttps://arxiv.org/abs/2606.06479. Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer- mamba language model.arXiv preprint arXiv:2403.19887,

    work page Pith review arXiv

Show all 28 references
  1. [22]

    Yi Heng Lim, Qi Zhu, Joshua Selfridge, and Muhammad Firmansyah Kasim

    URL https://arxiv.org/ abs/2403.19887. Yi Heng Lim, Qi Zhu, Joshua Selfridge, and Muhammad Firmansyah Kasim. Parallelizing non- linear sequential models over the sequence length,

    arXiv

  2. [23]

    URL https://arxiv.org/abs/2309. 12252. Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, and Qian Liu. Longhorn: State space models are amortized online learners.ArXiv, abs/2407.14207, 2024a. URL https://api. semanticscholar.org/CorpusID:271310065. Bo Liu, Rui Wang, Lemeng...

    arXiv

  3. [25]

    9 Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal

    URL https://arxiv.org/ abs/2305.16300. 9 Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention,

    arXiv

  4. [27]

    Jacob Pfau, William Merrill, and Samuel R

    URLhttps://arxiv.org/abs/2305.13048. Jacob Pfau, William Merrill, and Samuel R. Bowman. Let’s think dot by dot: Hidden computation in transformer language models.arXiv preprint arXiv:2404.15758,

    arXiv

  5. [28]

    org/abs/2404.15758

    URL https://arxiv. org/abs/2404.15758. Jonathan Pilault, Mahan Fathi, Orhan Firat, Chris Pal, Pierre-Luc Bacon, and Ross Goroshin. Block- state transformers.Advances in Neural Information Processing Systems, 36:7311–7329,

    arXiv

  6. [29]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi

    URL https: //arxiv.org/abs/2406.07522. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale.Proceedings of the AAAI Conference on Artificial Intelligence, 35(10):8732–8740,

    arXiv

  7. [30]

    Social IQA: Com- monsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQA: Com- monsense reasoning about social interactions. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 4463–4473,

    2019

  8. [32]

    Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin

    URL https://arxiv.org/abs/ 2208.04933. Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (learn at test time): RNNs with expressive hidden states,

    arXiv

  9. [33]

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei

    URL https://arxiv.org/abs/2407.04620. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models,

    arXiv

  10. [34]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

    URL https://arxiv.org/abs/2307.08621. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems, volume 30,

    arXiv

  11. [36]

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh

    URLhttps://arxiv.org/abs/2203.08913. Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule,

    arXiv

  12. [37]

    Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D

    URLhttps://arxiv.org/abs/2412.06464. Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. Quiet-STaR: Language models can teach themselves to think before speaking.arXiv preprint arXiv:2403.09629,

    arXiv

  13. [38]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi

    URLhttps://arxiv.org/abs/2403.09629. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800,

    arXiv

  14. [2017]

    10 Yuhuai Wu, Markus N

    URLhttps://arxiv.org/abs/1706.03762. 10 Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. arXiv preprint arXiv:2203.08913,

    arXiv

  15. [2018]

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V

    URLhttps://arxiv.org/abs/1803.05457. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V . Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. InProceedings of the 57th Annual Meeting of the Association for Computati...

    arXiv

  16. [2020]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi

    URLhttps://arxiv.org/abs/2004.05150. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language.Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432–7439,

    2004 arXiv

  17. [2021]

    Daniel Y

    URLhttps://arxiv.org/abs/2002.09402. Daniel Y . Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models.arXiv preprint arXiv:2212.14052,

    2002 arXiv

  18. [2023]

    Albert Gu, Karan Goel, and Christopher Ré

    URLhttps://arxiv.org/abs/2312.00752. Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. InInternational Conference on Learning Representations,

    arXiv

  19. [2024]

    Iz Beltagy, Matthew E

    URLhttps://arxiv.org/abs/2501.00663. Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150,

    2004 arXiv

  20. [2026]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova

    URL https: //arxiv.org/abs/2602.11605. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. InProceedings of the 2019 Conference of the North American Ch...

    2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.