Pith. sign in

REVIEW 2 major objections 5 minor 68 references

LoopMTP: A looped transformer guided by latent multi-token prediction

T0 review · 2 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 260M-parameter looped transformer that aligns each loop's hidden state with a future token's embedding improves average downstream accuracy by up to 8.08% over a slightly larger non-looped baseline and trains stably through 15 loops.

desk verdict The core trick works: latent cosine MTP guidance gives a real, multi-seed-supported ~8% gain at 260M scale, but the abstract's 15-loop stability claim rests on single-seed runs and should be read skeptically. read the letter →

arxiv 2608.03624 v1 pith:A6TRULWO submitted 2026-08-04 cs.CL

Behzad Shomali , Markus Frey , David Berghaus , Joachim Koehler , Mehdi Ali This is my paper
classification cs.CL
keywords loopedtransformermulti-tokenpredictionlatentreasoninghidden-statealignmentgatedaggregationparameter-efficientsmalllanguagemodelsmath
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Looped transformers reuse one stack of layers several times to gain the reasoning depth of a bigger model without adding parameters, but intermediate passes receive no supervision, so later loops can overwrite useful work and repeat the same computation. LoopMTP claims that a model looping T times can be taught to anticipate T future tokens: the hidden state produced by loop t is softly aligned, via cosine similarity, with the embedding of the token t positions ahead. A learned per-token gate then aggregates all loop outputs instead of discarding them. On 6.8B training tokens with a 260M-parameter model, the paper reports up to 8.08% relative average-accuracy gain over a slightly larger non-looped baseline and stable training up to 15 loops; a math-focused version reaches 19.03% GSM8K accuracy versus 7.05% for the non-looped baseline. The claim matters because it offers a way to buy reasoning capacity at fixed parameter memory: same weights, more deliberate reuse.

What carries the argument

The load-bearing object is the tuple of hidden states produced by repeatedly applying the shared 12-layer transformer stack. Three mechanisms make those states useful: the MTP-guided looped block, which fuses the previous iteration's state with fixed token embeddings and an iteration-index signal, using per-iteration LayerNorms and a fixed 1/T Loop-LNS scaling; the soft MTP alignment loss, which steers each iteration's hidden state toward the output embedding of the token it should anticipate, leaving iteration 1 unconstrained; and the gated aggregation that combines all loop outputs with a shared content-conditioned gate normalized across iterations. The gate initialization biases the model

What would settle it

Retrain the Section 5 math expert at T=15 and T=11 with multiple seeds under the same hyperparameters; if 19.03% GSM8K accuracy does not reliably beat 7.05%, or the T=11 dip vanishes, the stability and expert-gain claims fail. Separately, permute the alignment targets across tokens so the loss value stays the same but the future-token correspondence is destroyed; if downstream accuracy does not drop, the specific lookahead alignment, not the extra loss, is not the operative mechanism.

Watch

Extended reading notes

Core claim

The paper's central discovery is a structural correspondence: a looped transformer that runs T iterations can be supervised like a multi-token predictor, with iteration t responsible for anticipating the token t steps ahead. LoopMTP realizes this without vocabulary-sized heads: it uses a cosine alignment between each loop's hidden representation and the output embedding of the future token, with the embedding held fixed via stop-gradient, so the cost is negligible even at high loop counts. The outputs of all loops are combined through a shared content-conditioned gate, and a ponder regularizer pulls the gate distribution toward uniform so no iteration is starved. The paper shows the alignmen

Load-bearing premise

The claim that a looped transformer can be guided by latent multi-token prediction assumes that cosine similarity to a fixed token embedding is a usable learning signal for a small model; the abstract's 15-loop stability claim and the Section 5 GSM8K numbers additionally rest on single-seed runs, as the paper's Limitations state.

Editorial extensions

If this is right

  • Parameter-efficient reasoning can be improved with a cheap auxiliary loss rather than more parameters, since the gains come from reusing the same weights more deliberately.
  • The small-scale MTP penalty reported by prior work can be overcome by replacing vocabulary-level predictions with latent cosine alignment.
  • Looped models can train stably through 15 unrollings without per-loop hyperparameter search, making deeper unrolling practical under a fixed memory budget.
  • Small on-premises domain experts become viable: a 260M math model reaches 19.03% GSM8K accuracy with no finetuning, versus 7.05% for the non-looped baseline.
  • Representation aggregation, not just looping, is what carries the benefit: learnable per-iteration gates outperform both overwriting and uniform averaging, especially on math and code.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the alignment is causal, the unconstrained first iteration may be learning a compressed bundle of future-token information; a testable extension is to probe whether removing the alignment loss at specific t values degrades only those lookahead distances.
  • The horizon-alignment conjecture in Appendix B could be tested directly by padding or truncating reasoning sentences to match a 15-token window at a fixed loop count; if the dip at T=11 is a horizon mismatch, the dip should move with the window length.
  • The gate's later iterations become more input-dependent, which suggests a natural adaptive-compute policy: stop looping when later gate weights are near zero, saving inference FLOPs while keeping accuracy.
  • Because the method is loss-level rather than architecture-level for the backbone, it should transfer to other decoder-only backbones and to fine-tuning setups, but that transfer is an inference, not a claim of the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. LoopMTP augments a vertically looped transformer with a cosine-similarity auxiliary loss that aligns the hidden state of loop t with the embedding of the token t steps ahead, and aggregates per-loop states with a learned per-token gate. On a 260M GPT-2-style model trained on 6.8B tokens, the method reports average general-task accuracy up to 50.02% vs 46.28% for a slightly larger non-looped baseline (8.08% relative improvement) at T=9, based on 3-seed averages with small standard deviations. A math-expert variant reaches 19.03% GSM8K accuracy vs 7.05% at matched parameter count, and the paper claims stable training up to 15 loops. The paper also compares favorably to LoopFormer in 27 of 28 matched loop-count/benchmark settings.

Significance. If the main result holds, LoopMTP is a useful contribution: it provides a cheap latent MTP-style regularizer for looped transformers and a gated aggregation mechanism that mitigates representation overwriting, at negligible parameter overhead. The paper's strengths include three-seed averages with low standard deviations for the core Table 1 results, a direct w/ vs w/o MTP ablation that supports the mechanism, detailed appendices on hyperparameters, FLOPs accounting, and an unusually candid limitations section. The principal weakness is that the most headline-ready claims—stable training up to 15 loops and the GSM8K expert advantage—rest on single-seed runs, as the limitations explicitly concede. The central T<=9 result is credible, but the abstract and conclusion currently overstate the evidence beyond that range.

major comments (2)
  1. [Section 5 / Figure 5 / Limitations] The abstract's 'training remains stable for up to 15 loops' and the conclusion's '11.98 p.p. improvement on GSM8K' are supported only by single-seed expert runs, as the Limitations paragraph concedes. Figure 5 is non-monotonic (12.81 at T=3, 15.92 at T=11, 16.60 at T=13, 19.03 at T=15), so the plotted trend by itself cannot establish seed-insensitive stability; at most it shows that the particular seeds did not diverge. This is load-bearing because the abstract's most quotable claim goes beyond the multi-seed T<=9 evidence. Please add multi-seed runs for the T=11,13,15 configurations (particularly T=15) and report standard deviations, or qualify the abstract and conclusion to state that multi-seed stability evidence covers up to 9 loops and that the Section 5 results are single-seed exploratory.
  2. [Appendix A / Table 2 / Section 4.2] lambda_align is swept per loop count over {0.01,0.05,0.1,0.15,0.3,0.4} and selected for 'optimal performance'. Because lambda_align controls the strength of the very MTP signal that the paper claims is responsible for the gains, the headline 8.08% relative improvement over the non-looped baseline could partly reflect per-loop hyperparameter selection rather than a pure architectural effect. Please report the full lambda_align sweep or a fixed-lambda_align variant for the main results, and state explicitly whether the sweep was performed on the same evaluation benchmarks as Table 1. If it was, the reported numbers should be framed as selected rather than as a single-configuration comparison.
minor comments (5)
  1. [Section 4.3 / Figure 3] The median ground-truth rank at iteration t is evaluated for exactly the tokens u_{i+t} that Eq. (13) aligns to via the embedding E. The improvement in rank is therefore partly a measure of fitting the auxiliary objective, not an independent demonstration that the alignment 'transfers to the language-model head's coordinate system.' Please rephrase the claim or add a control, e.g., rank at offsets not used in L_align or on a held-out domain.
  2. [Abstract / Section 3.4] The abstract states that 'a model that loops T times can anticipate T future tokens,' but Eq. (13) aligns only iterations t=2..T, i.e., T-1 future tokens. Either adjust the wording or the loss to make the correspondence exact.
  3. [Table 1 / Table 5] The LoopFormer Loops=3 row is a 2-seed average because one run diverged. The '27 of 28' statement should note this asymmetry or exclude that row from matched comparisons.
  4. [Appendix B] The 'horizon alignment conjecture' is admittedly speculative and is contradicted by the strong results at Loop7 and Loop9, as the authors note. Consider moving it to a clearly labeled exploratory discussion or removing it; in its current placement it may distract from the main evidence.
  5. [Appendix C / Table 3] The weight-decay GSM8K results report no standard deviations or number of seeds. If these are single-seed exploratory runs, please state so in the table caption.

Circularity Check

1 steps flagged · score 3.0 of 10

Main downstream results are external and multi-seed; Figure 3's 'alignment transfers to the LM head' claim is largely definitional because the alignment loss is defined in the head's own unembedding coordinates.

  1. self definitional [Section 3.4 (Eq. 13) and Section 4.3 (Figure 3)]
    "Let E∈R^{|V|×d} denote the output (unembedding) matrix, treated as a fixed target via stop-gradient sg[·]. For t= 2, . . . , T, the per-step hidden-state alignment loss is: L(t)_align = 1/(S−t) Σ_{i=1}^{S−t} (1−cos(x_i^(t), sg[E_{u_{i+t}}])). ... This was obtained by feeding the output of each iteration to the language model head and reading off the logit distribution... This confirms the alignment is actually achieved and that it transfers to the language-model head’s coordinate system, rather than being satisfied in a subspace the head ignores."

    The alignment loss (Eq. 13) is defined as 1 − cos(x_i^(t), sg[E_{u_{i+t}}]) for exactly the same offset t and token u_{i+t} used in Figure 3, using the output/unembedding matrix E that defines the LM head's coordinate system. Measuring the ground-truth rank of u_{i+t} at iteration t therefore measures the quantity the training objective directly encourages; the 'transfer to the LM head' is built into the loss by construction, because the target embedding is taken from the head's own unembedding matrix. Figure 3 is thus a sanity check that the objective was optimized, not an independent confirmation of MTP's effect. The main downstream Table 1 result, however, is evaluated on external benchmarks and does not reduce to this objective.

full rationale

The paper's central claim—that LOOPMTP improves average downstream accuracy by up to 8.08% over a parameter-matched non-looped baseline—rests on Table 1, which is evaluated on external OLMES benchmarks, averaged over three seeds, and reported with small standard deviations. This is not circular. The comparison against LoopFormer is also external, and sweeping λalign does not turn the result into a fitted prediction of the same quantity. The self-citations (Frey et al. 2026a for iteration-index embedding and gate initialization; Frey et al. 2026b for capacity; Shomali et al. 2026 for math parameters) are implementation details or interpretive asides, not load-bearing derivations. The single-seed nature of Section 5 is an evidence limitation, not circularity. The only circular-adjacent element is Figure 3: the paper defines the alignment loss (Eq. 13) using the LM head's own unembedding matrix E and the exact future tokens u_{i+t}, then 'confirms' that the alignment transfers to the LM head by measuring the rank of u_{i+t} at iteration t. That measurement substantially reduces to the training objective. Because this is a supporting interpretability check rather than the central downstream claim, the overall circularity is mild.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on a small set of tuned hyperparameters (lambda_align per loop count, gate bias init, Loop-LNS factor, expert WD/LR) and on domain assumptions about metric reliability and data representativeness. The method introduces no new physical or formal entities; its architectural components (gate, loop index embedding, cosine target) are design elements, not invented entities of the graviton type.

free parameters (4)
  • lambda_align per loop count = T=3: 0.01; T=5: 0.01; T=7: 0.05; T=9: 0.15
    Auxiliary alignment loss weight, swept per loop count to optimize downstream performance (Appendix A). The choice affects the headline gains and is not fixed a priori.
  • gate bias initialization = first gate: 0.55; later gates: -3.0, then decreasing by 0.5 for expert runs
    Initial per-iteration gate biases are hand-set to favor the first iteration; the paper states initialization matters for training behavior (Section 3.3, Appendix A).
  • Loop-LNS scaling factor = 1/T
    Fixed per-iteration residual scaling inside the looped block, chosen over running-depth alternatives based on empirical gradient behavior (Section 3.2).
  • expert weight decay and learning rate = WD=0.132, LR=1.889e-3
    For Section 5 domain-expert models, these values differ from the main run and are motivated by the Appendix C weight decay study; they affect GSM8K results.
assumptions (4)
  • domain assumption BPB is a reliable proxy for downstream task accuracy at this scale.
    The paper uses bits-per-byte as the primary metric for math, code, and QA (Section 4.1), citing Gadre et al. (2025) and Heineman et al. (2026). If BPB improvements do not track task accuracy at 260M scale, several reported gains are weakened.
  • domain assumption The 'high-quality subset' of Nemotron-CC-v2 is a representative training corpus.
    Training uses a filtered subset whose exact filtering procedure is not fully specified (Section 4.1). Reproducibility and generalization depend on this choice.
  • ad hoc to paper The structural correspondence between loop count and future-token offset is a valid inductive bias.
    The paper states 'a model that loops T times can anticipate T future tokens' (Abstract) and pairs loop t with token u_{i+t} in Equation 13. No derivation or comparison to alternative offsets is provided; it is the central design assumption.
  • domain assumption Cosine similarity against the output unembedding matrix is a meaningful alignment target.
    Equation 13 uses sg[E_{u_{i+t}}] with the unembedding matrix as target. This assumes that direction in unembedding space carries semantic structure and that aligning hidden states to it helps, not hurts, downstream reasoning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LoopMTP: A looped transformer guided by latent multi-token prediction." pith.science (2026). https://pith.science/paper/A6TRULWO

@misc{pith2026260803624,
  author       = {Pith},
  title        = {Pith review of: LoopMTP: A looped transformer guided by latent multi-token prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A6TRULWO}},
  note         = {Machine review of arXiv:2608.03624}
}
abstract

Looped transformers have emerged as a parameter-efficient alternative to scaling depth for strong reasoning. By reusing one stack of layers across $T$ iterations, they attain the effective depth and reasoning capabilities of larger models at a fixed parameter count. Yet existing approaches suffer from latent overthinking and undifferentiated computation, largely because intermediate representations receive no guidance across loops. Multi-token prediction (MTP) supplies exactly the dense, forward-looking supervision the loop is missing. We propose \textsc{LoopMTP}, which links the two through a structural correspondence in latent space: a model that loops $T$ times can anticipate $T$ future tokens. \textsc{LoopMTP} realizes this by softly aligning the hidden state of loop $t$ with the embedding of the token $t$ steps ahead, while a lightweight gate preserves useful information across iterations. \textsc{LoopMTP} improves average accuracy by up to 8.1\% (relative) over the non-looped baseline, with training remaining stable for up to 15 loops.

Figures

Figures reproduced from arXiv: 2608.03624 by the authors.

Figure 1
Figure 1. Architecture and mechanism overview. (a) A standard looped transformer block applied T times where next-token supervision is applied only at the very end. (b) A standard multi-token prediction (MTP) model requiring full-vocabulary projections. (c) LOOPMTP (ours): per-iteration outputs are aligned in the latent space with future token embeddings (soft MTP) and combined via a gating mechanism. Illustrated for a model … view at source ↗
Figure 2
Figure 2. Impact of MTP auxiliary training signal on performance and per-loop representations. (Top) Aligning per-loop representations with the MTP sig￾nal improves performance, particularly at higher loop counts. (Bottom) With MTP, at most iterations yield a more distinct representation. Effect of MTP on hidden representations. Fig￾ure 2 (Bottom) reports cosine similarities for T=5 models trained with and without the MTP sig… view at source ↗
Figure 3
Figure 3. MTP training sharpens ground-truth re￾trieval. Despite the lightweight nature of the latent MTP guidance, the w/ MTP model achieves a ground￾truth rank up to 35.6× better than the w/o MTP model. produces a more distinct representation than its non-MTP counterpart, increasing expressiveness. These results confirm the existence of undifferen￾tiated computation (Yu et al., 2025), visible in the consistently high cross-… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Gating mechanism effects. (Left) When all gates are learnable we consistently achieve the best performance, particularly as the number of loops increases. (Right) Later iterations exhibit higher and more variable gate values, indicating increasingly input-dependent ble…
Figure 5
Figure 5. Figure 5: GSM8K accuracy for expert LOOPMTP and BPB–accuracy sensitivity. (Left) LOOPMTP achieves 19.03% vs. 7.05% at equal parameter count. (Right) The steep slope shows that even small BPB re￾ductions yield meaningful accuracy gains. reasoning or deployment on memory-constrain…
Figure 6
Figure 6. Figure 6: Average number of tokens per annotated tag. The reasoning traces are annotated following Bog￾dan et al. (2025). An unexplained dip at Loop11 and Loop13. When the lookahead window neither spans a full execution step nor coincides with a stable planning boundary, perform…
Figure 7
Figure 7. Figure 7: Effect of WD on a looped model. (Left) Gradients during training for T=9 model (Right) Per-layer spectral norms of the trained model. Inference (FLOPs/token, ×109 ) Training (total FLOPs, ×1019) T LOOPMTP LoopFormer ∆ LOOPMTP LoopFormer ∆ Non-looped 0.481 – 0.988 – 3 1…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 17 canonical work pages

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  2. [2]

    Publications Manual , year = "1983", publisher =

    1983

  3. [3]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    arXiv 1981

  4. [4]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  5. [5]

    Dan Gusfield , title =. 1997

    1997

  6. [6]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  7. [7]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  8. [8]

    International Conference on Learning Representations , volume=

    Reasoning with latent thoughts: On the power of looped transformers , author=. International Conference on Learning Representations , volume=

    work page

Show all 68 references
  1. [9]

    Advances in Neural Information Processing Systems , volume=

    The curse of depth in large language models , author=. Advances in Neural Information Processing Systems , volume=

  2. [10]

    arXiv preprint arXiv:2603.08391 , year=

    Adaptive Loops and Memory in Transformers: Think Harder or Know More? , author=. arXiv preprint arXiv:2603.08391 , year=

  3. [11]

    arXiv preprint arXiv:2510.25741 , year=

    Scaling latent reasoning via looped language models , author=. arXiv preprint arXiv:2510.25741 , year=

    arXiv

  4. [12]

    arXiv preprint arXiv:2404.19737 , year=

    Better & faster large language models via multi-token prediction , author=. arXiv preprint arXiv:2404.19737 , year=

    arXiv

  5. [13]

    arXiv preprint arXiv:2412.19437 , year=

    Deepseek-v3 technical report , author=. arXiv preprint arXiv:2412.19437 , year=

    arXiv

  6. [14]

    arXiv preprint arXiv:2404.00614 , year=

    Learning to plan for language modeling from unlabeled data , author=. arXiv preprint arXiv:2404.00614 , year=

    arXiv

  7. [15]

    arXiv preprint arXiv:2603.20219 , year=

    Thinking into the Future: Latent Lookahead Training for Transformers , author=. arXiv preprint arXiv:2603.20219 , year=

  8. [16]

    arxiv , year=

    A Survey on Latent Reasoning , author=. arxiv , year=

  9. [17]

    arXiv preprint arXiv:1807.03819 , year=

    Universal transformers , author=. arXiv preprint arXiv:1807.03819 , year=

    arXiv

  10. [18]

    arXiv preprint arXiv:2412.06769 , year=

    Training large language models to reason in a continuous latent space , author=. arXiv preprint arXiv:2412.06769 , year=

    arXiv

  11. [19]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Codi: Compressing chain-of-thought into continuous space via self-distillation , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  12. [20]

    arXiv preprint arXiv:2602.11451 , year=

    Loopformer: Elastic-depth looped transformers for latent reasoning via shortcut modulation , author=. arXiv preprint arXiv:2602.11451 , year=

  13. [21]

    arXiv preprint arXiv:2510.07358 , year=

    Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts , author=. arXiv preprint arXiv:2510.07358 , year=

  14. [22]

    Advances in Neural Information Processing Systems , volume=

    Scaling up test-time compute with latent reasoning: A recurrent depth approach , author=. Advances in Neural Information Processing Systems , volume=

  15. [23]

    Advances in Neural Information Processing Systems , volume=

    Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation , author=. Advances in Neural Information Processing Systems , volume=

  16. [24]

    The Fourteenth International Conference on Learning Representations , year=

    PonderLM: Pretraining Language Models to Ponder in Continuous Space , author=. The Fourteenth International Conference on Learning Representations , year=

  17. [25]

    arXiv preprint arXiv:2107.05407 , year=

    Pondernet: Learning to ponder , author=. arXiv preprint arXiv:2107.05407 , year=

    arXiv

  18. [26]

    arXiv preprint arXiv:2511.07384 , year=

    Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence , author=. arXiv preprint arXiv:2511.07384 , year=

  19. [27]

    Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    Olmes: A standard for language model evaluations , author=. Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    2025

  20. [28]

    arXiv preprint arXiv:2101.00027 , year=

    The pile: An 800gb dataset of diverse text for language modeling , author=. arXiv preprint arXiv:2101.00027 , year=

    arXiv

  21. [29]

    arXiv preprint arXiv:2510.07739 , year=

    MeSH: Memory-as-State-Highways for Recursive Transformers , author=. arXiv preprint arXiv:2510.07739 , year=

    arXiv

  22. [30]

    arXiv preprint arXiv:2605.18797 , year=

    Simply Stabilizing the Loop via Fully Looped Transformer , author=. arXiv preprint arXiv:2605.18797 , year=

    arXiv

  23. [31]

    arXiv preprint arXiv:2502.16982 , year=

    Muon is scalable for llm training , author=. arXiv preprint arXiv:2502.16982 , year=

    arXiv

  24. [32]

    arXiv preprint arXiv:1711.05101 , year=

    Decoupled weight decay regularization , author=. arXiv preprint arXiv:1711.05101 , year=

    arXiv

  25. [33]

    Language Models are Unsupervised Multitask Learners , author=

  26. [34]

    arXiv preprint arXiv:2508.14444 , year=

    Nvidia nemotron nano 2: An accurate and efficient hybrid mamba-transformer reasoning model , author=. arXiv preprint arXiv:2508.14444 , year=

    arXiv

  27. [35]

    Nemotron-cc-math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset , author =

  28. [36]

    arXiv preprint arXiv:2602.08387 , year=

    Modalities, a PyTorch-native Framework For Large-scale LLM Training and Research , author=. arXiv preprint arXiv:2602.08387 , year=

  29. [37]

    Neurocomputing , volume=

    Roformer: Enhanced transformer with rotary position embedding , author=. Neurocomputing , volume=. 2024 , publisher=

    2024

  30. [38]

    Findings of the Association for Computational Linguistics: EMNLP 2020 , pages=

    Query-key normalization for transformers , author=. Findings of the Association for Computational Linguistics: EMNLP 2020 , pages=

    2020

  31. [39]

    International conference on machine learning , pages=

    Scaling vision transformers to 22 billion parameters , author=. International conference on machine learning , pages=. 2023 , organization=

    2023

  32. [40]

    arXiv preprint arXiv:2002.05202 , year=

    Glu variants improve transformer , author=. arXiv preprint arXiv:2002.05202 , year=

    2002 arXiv

  33. [41]

    Neural computation , volume=

    Flat minima , author=. Neural computation , volume=. 1997 , publisher=

    1997

  34. [42]

    arXiv preprint arXiv:1609.04836 , year=

    On large-batch training for deep learning: Generalization gap and sharp minima , author=. arXiv preprint arXiv:1609.04836 , year=

    arXiv

  35. [43]

    Advances in Neural Information Processing Systems , volume=

    Why do we need weight decay in modern deep learning? , author=. Advances in Neural Information Processing Systems , volume=

  36. [44]

    Advances in neural information processing systems , volume=

    Spectrally-normalized margin bounds for neural networks , author=. Advances in neural information processing systems , volume=

  37. [45]

    Advances in Neural Information Processing Systems , volume=

    A little depth goes a long way: The expressive power of log-depth transformers , author=. Advances in Neural Information Processing Systems , volume=

  38. [46]

    Advances in neural information processing systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=

  39. [47]

    arXiv preprint arXiv:2511.08577 , year=

    Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models , author=. arXiv preprint arXiv:2511.08577 , year=

    arXiv

  40. [48]

    arXiv:1803.05457v1 , year =

    Peter Clark and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord , title =. arXiv:1803.05457v1 , year =

    arXiv

  41. [49]

    Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

    Hellaswag: Can a machine really finish your sentence? , author=. Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

  42. [50]

    Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    The LAMBADA dataset: Word prediction requiring a broad discourse context , author=. Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

  43. [51]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Piqa: Reasoning about physical commonsense in natural language , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  44. [52]

    Social IQa: Commonsense reasoning about social interactions , author=. Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) , pages=

    2019

  45. [53]

    Communications of the ACM , volume=

    Winogrande: An adversarial winograd schema challenge at scale , author=. Communications of the ACM , volume=. 2021 , publisher=

    2021

  46. [54]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

    arXiv

  47. [55]

    arXiv preprint arXiv:2108.07732 , year=

    Program synthesis with large language models , author=. arXiv preprint arXiv:2108.07732 , year=

    arXiv

  48. [56]

    doi:10.57967/hf/2497 , publisher =

    Lozhkov, Anton and Ben Allal, Loubna and von Werra, Leandro and Wolf, Thomas , title =. doi:10.57967/hf/2497 , publisher =

    doi

  49. [57]

    OpenWebText Corpus , author=

  50. [58]

    arXiv preprint arXiv:2110.14168 , year=

    Training Verifiers to Solve Math Word Problems , author=. arXiv preprint arXiv:2110.14168 , year=

    arXiv

  51. [59]

    International Conference on Learning Representations , volume=

    Language models scale reliably with over-training and on downstream tasks , author=. International Conference on Learning Representations , volume=

  52. [60]

    Advances in Neural Information Processing Systems , volume=

    Signal and noise: A framework for reducing uncertainty in language model evaluation , author=. Advances in Neural Information Processing Systems , volume=

  53. [61]

    arXiv preprint arXiv:2506.19143 , year=

    Thought Anchors: Which LLM Reasoning Steps Matter? , author=. arXiv preprint arXiv:2506.19143 , year=

  54. [62]

    arXiv preprint arXiv:2605.26494 , year=

    The minimax-m2 series: Mini activations unleashing max real-world intelligence , author=. arXiv preprint arXiv:2605.26494 , year=

    arXiv

  55. [63]

    arXiv preprint arXiv:2508.15096 , year=

    Nemotron-cc-math: A 133 billion-token-scale high quality math pretraining dataset , author=. arXiv preprint arXiv:2508.15096 , year=

    arXiv

  56. [64]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    arXiv

  57. [65]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    arXiv

  58. [66]

    Computer science review , volume=

    The rise of small language models in healthcare: A comprehensive survey , author=. Computer science review , volume=. 2026 , publisher=

    2026

  59. [67]

    arXiv preprint arXiv:2605.30202 , year=

    A Dual-Path Architecture for Scaling Compute and Capacity in LLMs , author=. arXiv preprint arXiv:2605.30202 , year=

    arXiv

  60. [68]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026) , pages=

    LLM Parameters for Math Across Languages: Shared or Separate? , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026) , pages=

    2026

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.