REVIEW 5 major objections 9 minor 2 cited by
Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from $k$-Parity
T0 review · 5 major / 9 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The masked diffusion loss splits into signal and noise, and tuning the mask schedule to the signal-rich window speeds up parity learning and 8B-scale language training.
desk verdict Parity theory is solid and worth refereeing; the language-scale claims need tempering. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Signal/Noise decomposition of the masked diffusion objective, paired with the reduction of the transformer to a two-layer MLP acting on the aggregated embedding $\tilde z$ after attention is fixed to uniform. In that reduced model, Theorem 4.3 rewrites the loss as a signal term that fits $f^*(\tilde z)$ on identifiable examples plus a noise term that drives outputs to zero on unidentifiable examples. Theorem 4.4 turns gradient flow on this loss into ascent on the energy $E(W)=c(W)^\top \Sigma(W)^\dagger c(W)$, where $c$ is the correlation with the target and $\Sigma$ is the feature covariance; the signal probability $P_S$ multiplies this energy and therefore sets the effective learning-rate scale for the hidden weights. Maximizing $P_S$ over a uniform mask distribution yields the single-point optimum $t_0=t_1=1/(k+1)$, which is the theoretical anchor for the language-model schedule. The same decomposition also produces the sample-complexity bound of Theorem 4.2, which the paper uses to derive a separate sample-complexity-optimal schedule.
What would settle it
Train a masked-diffusion transformer on $k$-parity with learnable attention and compare its generalization curve and optimal mask rate with the uniform-attention MLP; a grokking plateau in the learnable-attention run, or an optimal rate far from $1/(k+1)$, would show the attention reduction hides the real mechanism. A language-scale test would hold compute fixed and swap the mask schedule to, say, high-noise intervals during pretraining; if a high-noise schedule matches or beats the mid-range window on knowledge-heavy benchmarks, the claim that $t\approx0.5$ is the general signal-optimal window would be weakened.
Extended reading notes
Core claim
On the authors' own terms, the discovery is that the masked diffusion loss for $k$-parity separates exactly (up to constants) into $$L_{\mathrm{eff}}(\$\theta$)=P_S\,\mathbb{E}_{\tilde z\mid R_S}\!\left[\frac{|M_m|}{2t}(f_\$\theta$(\tilde z)-f^*(\tilde z))^2\right]+P_N\,\mathbb{E}_{\tilde z\mid R_N}\!\left[\frac{|M_m|}{2t}f_\$\theta$(\tilde z)^2\right],$$ with $P_S=(k+1)\mathbb{E}_t[t(1-t)^k]$ and $P_N=1-P_S$. Signal examples leave exactly one member of the secret set unmasked and therefore determine the target; noise examples either hide too much or hide too little and make the target unidentifiable. The noise term acts as an implicit regularizer that keeps the model from memorizing, and the landscape analysis in Theorem 4.4 shows that feature learning is driven by an energy $E(W)=c(W)^\top \Sigma(W)^\dagger c(W)$ whose ascent speed is set by $P_S$. From this, the paper derives $t^*=1/(k+1)$ as the signal-optimal mask rate and verifies on $(20,6)$-parity that masked diffusion reaches train-and-test accuracy together, without a grokking plateau, and fastest near the predicted range. At language scale, the paper reports that restricting mask sampling to $[0.45,0.55]$ improves pretraining and fine-tuning results for 50M and 8B models relative to uniform $[0,1]$ sampling.
Load-bearing premise
The theory assumes attention can be fixed to uniform without changing the learning dynamics, and all theorems are proved for the resulting two-layer MLP; if attention itself is doing load-bearing work in real masked diffusion transformers, the mechanism would not transfer to the large models.
Editorial extensions
If this is right
- Masked diffusion training on $k$-parity should reach train and test accuracy at the same time, with no grokking plateau, whenever the mask schedule keeps $P_S$ away from zero.
- For a known dependency order $k$, the signal-optimal schedule is concentrated at $t=1/(k+1)$; for $k=6$ this predicts the fast range observed near $t\approx0.246$.
- The standard uniform schedule $t\sim U[0,1]$ spends a large fraction of its compute on low-signal regimes, so restricting training to the empirically located signal-rich window should improve convergence speed and final perplexity.
- At 8B scale, the restricted schedule improves zero-shot HellaSwag by 4.6 absolute points and ARC-Easy by 8.8 absolute points after 15,000 pretraining steps relative to uniform sampling.
- For supervised fine-tuning, mid-range masking helps discriminative benchmarks (GPQA 0.402 vs 0.344 uniform), while shifting toward high-noise intervals improves generative reasoning (GSM8K 0.785 on $[0.5,1.0]$ vs 0.768 on $[0,1]$).
Reading between the lines
- The paper does not say this, but the optimal window should depend on the effective dependency structure of the task; longer-range dependencies would shift the signal maximum, so a fixed $[0.45,0.55]$ window is unlikely to be optimal for every benchmark.
- An extension the authors leave implicit is a curriculum: pretrain on the signal-rich window first, then anneal the schedule toward higher mask rates for reasoning tasks, which their range ablations suggest would combine the knowledge and reasoning gains.
- The uniform-attention reduction implies a sharper test than the paper runs: an MLP and a transformer with identical schedules should show the same optimal mask rate, and the appendix only checks that uniform attention is trainable, not that the rate is identical.
- The same decomposition should apply to any reconstruction loss over independently masked coordinates, so the schedule-tuning idea could be tested in masked autoencoders outside language modeling.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper studies the generalization properties of masked diffusion language models (MDLMs) through the lens of the k-parity problem. The authors decompose the masked diffusion objective into a Signal regime (masks that leave the parity bit identifiable) and a Noise regime (masks that do not), and prove for a reduced two-layer MLP with uniform attention that the noise term acts as an implicit regularizer penalizing confident outputs on unidentifiable inputs. They derive an optimal masking rate t=1/(k+1) for signal maximization and a sample-complexity-optimal schedule. Experiments on (20,6) parity with a nanoGPT-style transformer show that masked diffusion avoids the grokking plateau, and the best interval U[0,0.246] roughly matches the computed signal-optimal interval for t0=0. The authors then transfer the idea to language modeling: a 50M-parameter ablation on WikiText locates the best masking interval near t in [0.45,0.55], and using this interval for LLaDA-8B pretraining and SFT yields downstream gains up to 8.8 points on ARC-Easy and 5.8 points on GPQA, with the caveat that generative math reasoning favors wider intervals extending to t=1.
Significance. If the decomposition is taken at face value for the reduced model, the paper provides a clean explanation for the absence of grokking under masked diffusion and a falsifiable prediction for the optimal mask rate, which is qualitatively confirmed by the parity experiment. The core loss decomposition (Theorem 4.3) is derived rather than fitted, and the 8B pretraining and SFT gains are substantial and practically useful. The paper is honest about several limitations but overstates the theoretical scope: the theory applies to a uniform-attention MLP, while the language-model window is selected empirically. The contribution is therefore a mechanism-inspired schedule with partial theoretical backing, and the manuscript would benefit from a more precise separation of what is proven, what is empirically established, and what is conjecture.
major comments (5)
- [Section 4.1 and Appendix F.1] The theoretical claims (Theorems 4.3, 4.4, Corollaries 4.5-4.7) are all proved for the aggregated-input two-layer MLP after fixing attention to uniform. The only justification for this reduction is the ablation in Appendix F.1, which demonstrates sufficiency (uniform attention can learn parity without grokking) but not necessity. The nanoGPT experiments in Section 4.5 and the LLaDA-8B experiments in Section 5 train learned attention jointly, and the paper contains no analysis of how attention interacts with the noise-regime regularizer. The statement in the introduction that the noise regime 'prove[s]' the mechanism for masked diffusion generally therefore overstates the theoretical scope; the language-model results should be presented as an empirically motivated schedule, with the theory applying strictly to the reduced MLP.
- [Section 4.3, after Theorem 4.4] The sentence 'Since E(W) is proportional to P_S^2' is not a consequence of the theorem as stated. With Sigma(W) = P_S E_S[|Mm|/(2t) hh^T] + P_N E_N[|Mm|/(2t) hh^T] and c(W) = P_S E_S[|Mm|/(2t) f* h], the energy is E(W) = P_S^2 a^T (P_S A + P_N B)^dagger a, which is not proportional to P_S^2 in general. This invalidates the 'dynamic gain factor' interpretation as written, although the qualitative statement that the signal term scales c(W) remains true. Please correct the proportionality claim and re-derive any subsequent conclusions that rely on it.
- [Sections 5.1 and 5.2] The optimal language window t in [0.45,0.55] is selected from the 50M WikiText ablation (Figure 3), not derived from the parity theory. The parity theory gives t=1/(k+1) for a known k, which has no direct analogue for natural language. The 8B results therefore validate an empirical schedule heuristic, not the theoretical prediction. This should be stated explicitly, and the abstract's phrasing 'leveraging our theoretical insights' should be softened accordingly.
- [Section 5.2.2 and Table 3] The generative reasoning results show the opposite of the signal-optimal prescription: shifting the sampling interval toward t=1 improves GSM8K and MATH, and the [0.5,1.0] interval is best. The paper's post hoc explanation is plausible, but it means the proposed 'signal-optimal window' is task-dependent, not a universal principle. The conclusion and abstract should present the discriminative/generative distinction as part of the empirical finding rather than as a corollary of the parity theory.
- [Section 5, objective with 1/t weighting] The training objective uses a 1/t factor on the cross-entropy loss. Under U[0,1], samples with t near 0 receive extremely large weights; restricting to [0.45,0.55] removes this instability. The 50M and 8B improvements could be due in part to this reweighting artifact rather than to the signal-noise mechanism. To support the theoretical attribution, add a control schedule that excludes low t but includes high t (e.g., U[0.1,1] or U[0.05,0.95]) and show that the gains are not explained by the 1/t weighting alone.
minor comments (9)
- [Section 1] 'quesiton' is a typo for 'question'.
- [Definition 3.3] 'colume-wise' should be 'column-wise'.
- [Theorem 4.2] The theorem statement refers to 'theorem 3.4'; it should cite Definition 3.4.
- [After Corollary 4.5] The text refers to 'theorem 4.5' when it means 'Corollary 4.5'.
- [Figure 3 and Section 5.1] The figure caption says the baseline is a 'dashed blue line' while the text refers to a 'dashed black line'; make the references consistent.
- [Table 3] The table includes a 'Base' column but the caption says 'Evaluation of SFT models'; clarify whether 'Base' is the pretrained model without SFT.
- [Section 5.1] The choice t in [0.45,0.55] is said to be suggested by Figure 3, but Figure 3's best intervals are wider ([0.4,0.5] and [0.5,0.6]); justify the narrower choice.
- [Appendix F.5] 'Experiments was conducted' should be 'Experiments were conducted'.
- [Theorem 4.4 statement vs Appendix D.1] The main text defines Sigma(W) without the explicit P_S/P_N weighting that appears in the proof; align the two definitions.
Circularity Check
No significant circularity in the theoretical decomposition; only the language mask window is empirically fitted and then labeled 'signal-optimal'.
-
fitted input called prediction
[Section 5.1, 'Locating the Signal-Optimal Window' and Section 5.2.1, Table 1]
"The U-shaped curve demonstrates that training exclusively within the ”Signal-Optimal” window (t∈[0.4,0.5] or [0.5,0.6]) significantly outperforms the standard practice of sampling the full range. As a result, we use the range t∈[0.45,0.55] for the following 8B experiments."
The language 'signal-optimal window' is not derived from the k-parity theory (which yields t=1/(k+1) for known k); it is the empirical argmin of a 50M-parameter test-loss scan over ten intervals. Calling this window 'Signal-Optimal' and reporting that it 'significantly outperforms' the baseline at 50M is therefore a restatement of the selection criterion: the chosen interval is, by construction, the best in that scan. The 8B experiments are out-of-sample and provide independent support, so the circularity is partial and method-level rather than a forced mathematical equivalence.
full rationale
The core derivation is not circular. Theorem 4.3 partitions mask configurations by |M_m ∩ S'| and algebraically reduces the expected MD loss to a signal-squared term plus a noise-norm term, with P_S computed from the Bernoulli masking process; it assumes the conclusion nowhere. Corollary 4.7's t=1/(k+1) is a calculus consequence of maximizing that P_S, and the parity experiment independently confirms the predicted U[0,0.246] range, so the parity 'prediction' is a genuine out-of-sample result. The only mild circularity is in the language application: the [0.45,0.55] window is selected as the empirical minimum of a 50M-parameter scan (Section 5.1), then labeled 'Signal-Optimal' and presented as consistent with the theory; at 50M the superiority is by construction of the selection, though the 8B results are out-of-sample and supply real evidence. The transformer-to-MLP reduction (Section 4.1) rests on the F.1 ablation, which is sufficiency evidence, not a theorem, and is therefore a correctness/scope concern rather than a circular step. The self-citation (Huang et al. 2025a) supports only a standard matrix-consolidation convention and is not load-bearing.
Assumptions & free parameters
free parameters (1)
- Language mask interval [t_min, t_max] =
[0.45, 0.55] (and [0.5, 1.0] for generative SFT)
assumptions (4)
- domain assumption Lazy readout: linear readout v reaches optimality v*(W) much faster than hidden weights W evolve.
- domain assumption Attention can be replaced by uniform attention; the transformer reduces to a 2-layer MLP on the aggregated input.
- ad hoc to paper In the pure-signal limit, the network is over-parameterized so the target y lies in the span of features H for the relevant W.
- standard math Input bits and parity labels are drawn uniformly; noise bits are statistically independent of visible bits.
Cite this review
Pith. "Pith review of Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from $k$-Parity." pith.science (2026). https://pith.science/paper/OZTHJLTS
@misc{pith2026260122450,
author = {Pith},
title = {Pith review of: Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from $k$-Parity},
year = {2026},
howpublished = {\url{https://pith.science/paper/OZTHJLTS}},
note = {Machine review of arXiv:2601.22450}
}
abstract
Masked Diffusion Language Models have recently emerged as a powerful generative paradigm, yet their generalization properties remain understudied compared to their auto-regressive counterparts. In this work, we investigate these properties within the setting of the $k$-parity problem (computing the XOR sum of $k$ relevant bits), where neural networks typically exhibit grokking -- a prolonged plateau of chance-level performance followed by sudden generalization. We theoretically decompose the Masked Diffusion (MD) objective into a Signal regime which drives feature learning, and a Noise regime which serves as an implicit regularizer. By training nanoGPT using MD objective on the $k$-parity problem, we demonstrate that MD objective fundamentally alters the learning landscape, enabling rapid and simultaneous generalization without experiencing grokking. Furthermore, we leverage our theoretical insights to optimize the distribution of the mask probability in the MD objective. Our method significantly improves perplexity for 50M-parameter models and achieves superior results across both pre-training from scratch and supervised fine-tuning. Specifically, we observe performance gains peaking at $8.8\%$ and $5.8\%$, respectively, on 8B-parameter models, confirming the scalability and effectiveness of our framework in large-scale masked diffusion language model regimes.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 2 Pith papers
-
Discrete Diffusion Models: A Unified Framework from Tokenization to Generation
Discrete diffusion models are re-framed as instances of a tokenization-centric, four-component design space (corruption, denoiser, objective, sampler) in a broad survey with no new experimental or theoretical results.
-
When Top-1 Fails: Calibrating LoRA Monitors for Masked Diffusion LMs
Empirical test shows top-1 argmax concentration has zero precision as collapse warning in DLM LoRA training due to pre-equilibrium saturation while max gradient norm provides usable but family-specific detection on sh...
Reference graph
Works this paper leans on
-
[1]
B., and Misiakiewicz, T
Abbe, E., Adsera, E. B., and Misiakiewicz, T. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory, pp.\ 4782--4887. PMLR, 2022
2022
-
[2]
Abbe, E., Adsera, E. B., and Misiakiewicz, T. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. In The Thirty Sixth Annual Conference on Learning Theory, pp.\ 2552--2623. PMLR, 2023
work page 2023
-
[3]
T., Yang, Z., Qi, Z., Han, J., Sahoo, S
Arriola, M., Gokaslan, A., Chiu, J. T., Yang, Z., Qi, Z., Han, J., Sahoo, S. S., and Kuleshov, V. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503.09573, 2025
arXiv 2025
-
[4]
Parallel sampling from masked diffusion models via conditional independence testing
Azangulov, I., Pandeva, T., Prasad, N., Zazo, J., and Karmalkar, S. Parallel sampling from masked diffusion models via conditional independence testing. arXiv preprint arXiv:2510.21961, 2025
arXiv 2025
-
[5]
Hidden progress in deep learning: Sgd learns parities near the computational limit
Barak, B., Edelman, B., Goel, S., Kakade, S., Malach, E., and Zhang, C. Hidden progress in deep learning: Sgd learns parities near the computational limit. Advances in Neural Information Processing Systems, 35: 0 21750--21764, 2022
2022
-
[6]
S., Bachmann, G., He, B., Noci, L., and Hofmann, T
Bautiste, J. S., Bachmann, G., He, B., Noci, L., and Hofmann, T. Unveiling grokking: Analyzing feature learning dynamics during training. In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024. URL https://openreview.net/forum?id=gciHssAM8A
work page 2024
-
[7]
Learning time-scales in two-layers neural networks
Berthier, R., Montanari, A., and Zhou, K. Learning time-scales in two-layers neural networks. Foundations of Computational Mathematics, 25 0 (5): 0 1627--1710, 2025
work page 2025
-
[8]
Think you have solved question answering? try arc, the ai2 reasoning challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018
arXiv 2018
Show all 67 references
-
[9]
Training verifiers to solve math word problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021
2021 arXiv
-
[10]
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities
Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv prepri...
2025 arXiv
-
[11]
and Malach, E
Daniely, A. and Malach, E. Learning parities with neural networks. Advances in Neural Information Processing Systems, 33: 0 20356--20365, 2020
2020
-
[12]
Bert: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolog...
2019
-
[13]
Domin \'e , C. C. J., Anguita, N., Proca, A. M., Braun, L., Kunin, D., Mediano, P. A. M., and Saxe, A. M. From lazy to rich: Exact learning dynamics in deep linear networks. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.ne...
2025
-
[14]
Saber: An efficient sampling with adaptive acceleration and backtracking enhanced remasking for diffusion language model
Dong, Y., Ma, Z., Jiang, X., Fan, Z., Qian, J., Li, Y., Xiao, J., Jin, Z., Cao, R., Li, B., et al. Saber: An efficient sampling with adaptive acceleration and backtracking enhanced remasking for diffusion language model. arXiv preprint arXiv:2510.18165, 2025
-
[15]
L., Goel, S., Kakade, S
Edelman, B. L., Goel, S., Kakade, S. M., eran malach, and Zhang, C. Pareto frontiers in deep feature learning: Data, compute, width, and luck. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=Ypbke6biDm
2023
-
[16]
Scaling diffusion language models via adaptation from autoregressive models
Gong, S., Agarwal, S., Zhang, Y., Ye, J., Zheng, L., Li, M., An, C., Zhao, P., Bi, W., Han, J., Peng, H., and Kong, L. Scaling diffusion language models via adaptation from autoregressive models. In The Thirteenth International Conference on Learning Representations, 2025 a . ...
2025
-
[17]
Diffucoder: Understanding and improving masked diffusion models for code generation
Gong, S., Zhang, R., Zheng, H., Gu, J., Jaitly, N., Kong, L., and Zhang, Y. Diffucoder: Understanding and improving masked diffusion models for code generation. arXiv preprint arXiv:2506.20639, 2025 b
2025 arXiv
-
[18]
Mdpo: Overcoming the training-inference divide of masked diffusion language models
He, H., Renz, K., Cao, Y., and Geiger, A. Mdpo: Overcoming the training-inference divide of masked diffusion language models. arXiv preprint arXiv:2508.13148, 2025
2025
-
[19]
Measuring massive multitask language understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021 a . URL https://openreview.net/forum?id=d7KBjmI3GmQ
2021
-
[20]
Measuring mathematical problem solving with the MATH dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021...
2021
-
[21]
Hong, C., An, S., Kim, M.-S., and Ye, J. C. Improving discrete diffusion unmasking policies beyond explicit reference policies. arXiv preprint arXiv:2510.05725, 2025 a
2025
-
[22]
Wide-in, narrow-out: Revokable decoding for efficient and effective dllms
Hong, F., Yu, G., Ye, Y., Huang, H., Zheng, H., Zhang, Y., Wang, Y., and Yao, J. Wide-in, narrow-out: Revokable decoding for efficient and effective dllms. arXiv preprint arXiv:2507.18578, 2025 b
2025
-
[23]
Huang, J., Wang, Z., and Lee, J. D. Transformers learn to implement multi-step gradient descent with chain of thought. In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openreview.net/forum?id=r3DF5sOo5B
2025
-
[24]
Pc-sampler: Position-aware calibration of decoding bias in masked diffusion models
Huang, P., Liu, S., Liu, Z., Yan, Y., Wang, S., Chen, Z., and Xiao, T. Pc-sampler: Position-aware calibration of decoding bias in masked diffusion models. arXiv preprint arXiv:2508.13021, 2025 b
2025
-
[25]
X., B \'e thune, L., Ablin, P., Kirchhof, M., Monterio, J., Turrisi, V., Ramapuram, J., and Cuturi, M
Jazbec, M., Olausson, T. X., B \'e thune, L., Ablin, P., Kirchhof, M., Monterio, J., Turrisi, V., Ramapuram, J., and Cuturi, M. Learning unmasking policies for diffusion language models. arXiv preprint arXiv:2512.09106, 2025
2025 arXiv
-
[26]
and Suzuki, T
Kim, J. and Suzuki, T. Transformers provably solve parity efficiently with chain of thought. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=n2NidsYDop
2025
-
[27]
Train for the worst, plan for the best: Understanding token ordering in masked diffusions
Kim, J., Shah, K., Kontonis, V., Kakade, S., and Chen, S. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arXiv preprint arXiv:2502.06768, 2025
2025 arXiv
-
[28]
Masked diffusion language models with frequency-informed training
Kosmopoulou, D., Georgiou, E., Dorovatas, V., Paraskevopoulos, G., and Potamianos, A. Masked diffusion language models with frequency-informed training. In Proceedings of the First BabyLM Workshop, pp.\ 531--539, 2025
2025
-
[29]
Matching the statistical query lower bound for k -sparse parity problems with sign stochastic gradient descent
Kou, Y., Chen, Z., Gu, Q., and Kakade, S. Matching the statistical query lower bound for k -sparse parity problems with sign stochastic gradient descent. Advances in Neural Information Processing Systems, 37: 0 113001--113037, 2024
2024
-
[30]
J., and Pehlevan, C
Kumar, T., Bordelon, B., Gershman, S. J., and Pehlevan, C. Grokking as the transition from lazy to rich training dynamics. In The twelfth international conference on learning representations, 2023
2023
-
[31]
Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, X., Gu, Y., Malik, S., Graf, V., Hwang, J. D., Yang, J., Bras, R. L., Tafjord, O., Wilhelm, C., Soldaini, L., Smith, N. A., Wang, Y., Dasigi, P., and Hajishi...
2025
-
[32]
Y., Bansal, H., Guha, E., Keh, S
Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S. Y., Bansal, H., Guha, E., Keh, S. S., Arora, K., et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems, 37: 0 14200--14282, 2024
2024
-
[33]
Deepseek-v3 technical report
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024
2024 arXiv
-
[34]
Tidar: Think in diffusion, talk in autoregression
Liu, J., Dong, X., Ye, Z., Mehta, R., Fu, Y., Singh, V., Kautz, J., Zhang, C., and Molchanov, P. Tidar: Think in diffusion, talk in autoregression. arXiv preprint arXiv:2511.08923, 2025
2025
-
[35]
Discrete diffusion modeling by estimating the ratios of the data distribution
Lou, A., Meng, C., and Ermon, S. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=CNicRIVIPA
2024
-
[36]
and Berthier, R
Marion, P. and Berthier, R. Leveraging the two-timescale regime to demonstrate convergence of neural networks. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ 64996--65029. ...
2023
-
[37]
Pointer sentinel mixture models
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017
2017
-
[38]
A tale of two circuits: Grokking as competition of sparse and dense subnetworks
Merrill, W., Tsilivis, N., and Shukla, A. A tale of two circuits: Grokking as competition of sparse and dense subnetworks. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023. URL https://openreview.net/forum?id=8GZxtu46Kx
2023
-
[39]
Ni, J., Liu, Q., Dou, L., Du, C., Wang, Z., Yan, H., Pang, T., and Shieh, M. Q. Diffusion language models are super data learners. arXiv preprint arXiv:2511.03276, 2025 a
2025
-
[40]
Ni, J., Liu, Q., Du, C., Dou, L., Yan, H., Wang, Z., Pang, T., and Shieh, M. Q. Training optimal large diffusion language models. arXiv preprint arXiv:2510.03280, 2025 b
2025
-
[41]
Nichani, E., Damian, A., and Lee, J. D. How transformers learn causal structure with gradient descent. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=jNM4imlHZv
2024
-
[42]
Scaling up masked diffusion models on text
Nie, S., Zhu, F., Du, C., Pang, T., Liu, Q., Zeng, G., Lin, M., and Li, C. Scaling up masked diffusion models on text. In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openreview.net/forum?id=WNvvwK0tut
2025
-
[43]
Large language diffusion models
Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., and Li, C. Large language diffusion models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025 b . URL https://openreview.net/forum?id=KnqiC0znVF
2025
-
[44]
Gpt-4 technical report
OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023
2023 arXiv
-
[45]
Your absorbing discrete diffusion secretly models the conditional distributions of clean data
Ou, J., Nie, S., Xue, K., Zhu, F., Sun, J., Li, Z., and Li, C. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=sMyXP8Tanm
2025
-
[46]
Pasand, A. S. and Dohmatob, E. Egalitarian gradient descent: A simple approach to accelerated grokking, 2025. URL https://arxiv.org/abs/2510.04930
2025 arXiv
-
[47]
Masks can be distracting: On context comprehension in diffusion language models
Piskorz, J., Pinneri, C., Correia, A., Alfarra, M., Garrepalli, R., and Louizos, C. Masks can be distracting: On context comprehension in diffusion language models. arXiv preprint arXiv:2511.21338, 2025
2025 arXiv
-
[48]
Grokking: Generalization beyond overfitting on small algorithmic datasets
Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022
2022 arXiv
-
[49]
Prieto, L., Barsbey, M., Mediano, P. A. M., and Birdal, T. Grokking at the edge of numerical stability. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=TvfkSyHZRA
2025
-
[50]
L., Stickland, A
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024
2024
-
[51]
S., Arriola, M., Gokaslan, A., Marroquin, E
Sahoo, S. S., Arriola, M., Gokaslan, A., Marroquin, E. M., Rush, A. M., Schiff, Y., Chiu, J. T., and Kuleshov, V. Simple and effective masked diffusion language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openrevie...
2024
-
[52]
Simplified and generalized masked diffusion for discrete data
Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37: 0 103131--103167, 2024
2024
-
[53]
Provable scaling laws of feature emergence from learning dynamics of grokking
Tian, Y. Provable scaling laws of feature emergence from learning dynamics of grokking. arXiv preprint arXiv:2509.21519, 2025
2025
-
[54]
Tian, Y., Wang, Y., Chen, B., and Du, S. S. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volu...
2023
-
[55]
Time is a feature: Exploiting temporal dynamics in diffusion language models
Wang, W., Fang, B., Jing, C., Shen, Y., Shen, Y., Wang, Q., Ouyang, H., Chen, H., and Shen, C. Time is a feature: Exploiting temporal dynamics in diffusion language models. arXiv preprint arXiv:2508.09138, 2025 a
2025
-
[56]
Diffusion llms can do faster-than-ar inference via discrete diffusion forcing
Wang, X., Xu, C., Jin, Y., Jin, J., Zhang, H., and Deng, Z. Diffusion llms can do faster-than-ar inference via discrete diffusion forcing. arXiv preprint arXiv:2508.09192, 2025 b
2025 arXiv
-
[57]
From sparse dependence to sparse attention: Unveiling how chain-of-thought enhances transformer sample efficiency
Wen, K., Zhang, H., Lin, H., and Zhang, J. From sparse dependence to sparse attention: Unveiling how chain-of-thought enhances transformer sample efficiency. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=AmEgWDhmTr
2025
-
[58]
Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding
Wu, C., Zhang, H., Xue, S., Liu, Z., Diao, S., Zhu, L., Luo, P., Han, S., and Xie, E. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.22618, 2025
2025 arXiv
-
[59]
Benign overfitting and grokking in re LU networks for XOR cluster data
Xu, Z., Wang, Y., Frei, S., Vardi, G., and Hu, W. Benign overfitting and grokking in re LU networks for XOR cluster data. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=BxHgpC6FNv
2024
-
[60]
Qwen3 technical report
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025
2025 arXiv
-
[61]
Dream 7b: Diffusion large language models
Ye, J., Xie, Z., Zheng, L., Gao, J., Wu, Z., Jiang, X., Li, Z., and Kong, L. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487, 2025
2025 arXiv
-
[62]
Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4800, 2019
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4800, 2019
2019
-
[63]
Zhang, R., Frei, S., and Bartlett, P. L. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25 0 (49): 0 1--55, 2024
2024
-
[64]
Diffpo: Training diffusion llms to reason fast and furious via reinforcement learning
Zhao, H., Liang, D., Tang, W., Yao, D., and Kallus, N. Diffpo: Training diffusion llms to reason fast and furious via reinforcement learning. arXiv preprint arXiv:2510.02212, 2025 a
2025
-
[65]
d1: Scaling reasoning in diffusion large language models via reinforcement learning
Zhao, S., Gupta, D., Zheng, Q., and Grover, A. d1: Scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint arXiv:2504.12216, 2025 b
2025 arXiv
-
[66]
Inpainting-guided policy optimization for diffusion large language models
Zhao, S., Liu, M., Huang, J., Liu, M., Wang, C., Liu, B., Tian, Y., Pang, G., Bell, S., Grover, A., et al. Inpainting-guided policy optimization for diffusion large language models. arXiv preprint arXiv:2509.10396, 2025 c
2025 arXiv
-
[67]
dllm: Simple diffusion language modeling
Zhou, Z., Chen, L., Tong, H., and Song, D. dllm: Simple diffusion language modeling. https://github.com/ZHZisZZ/dllm, 2025
2025
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.