Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Masked diffusion models and any-order autoregressive models are the same formulation, and a decoder-only implementation of that formulation generates about 25 times faster than an encoder-only one while reaching comparable perplexity…

desk verdict Useful empirical findings (10% L2R mix, ~25x speedup, order-ensemble) paired with an over-sold equivalence claim that breaks for the order-dependent decoder model the paper actually trains. read the letter →

arxiv 2506.19935 v1 pith:NZ7T6YEF submitted 2025-06-24 cs.LG cs.CVstat.ML

Shuchen Xue , Tianyu Xie , Tianyang Hu , Zijin Feng , Jiacheng Sun , Kenji Kawaguchi , Zhenguo Li , Zhi-Ming Ma This is my paper · ORCID
classification cs.LGcs.CVstat.ML
keywords maskeddiffusionmodelsany-orderautoregressivedecoder-onlyarchitecturetoken-orderdistributiongenerationspeedupperplexitydiscretelanguagetemperatureannealing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes that masked diffusion models and any-order autoregressive models optimize the same objective, so the paradigm difference between them collapses to a choice of token-order distribution. Using a decoder-only architecture that can predict any token given any ordered prefix, the paper shows that the uniform any-order objective converges more slowly than left-to-right and that adding a small fraction of left-to-right data improves both left-to-right and any-order performance. Within the masked-diffusion formulation, encoder-only models represent far fewer conditional probabilities than decoder-only models, but decoder-only models reduce generation complexity from quadratic to linear and achieve a measured speedup of roughly 25x at comparable generation perplexity after temperature annealing. The reason to care is that these results separate what objective is being optimized from what network shape is used, which is the only way to compare autoregressive and diffusion language models fairly.

What carries the argument

The load-bearing identity is Eq. (5), which chains two existing derivations: the RADD derivation showing that the masked-diffusion ELBO equals a nested expectation over mask levels, and the NADE derivation showing that nested expectation equals the uniform average over all $n!$ generation orders. The supporting mechanism is AO-GPT's decoder-only any-order architecture: causal attention plus adaptive LayerNorm conditioned on the target token's original position, so the network can predict any position conditioned on any ordered prefix. The speed mechanism is the pairing of a KV-cache with Lemma 1's two-stage sampling, which replaces a per-token model evaluation by a Bernoulli draw that often leaves the token masked, and a specialized parallel attention mask that predicts all simultaneously unmasked tokens in one forward pass.

What would settle it

Enumerate all $n!$ orders for short sequences and compare the exact AO-AR loss with a high-precision numerical evaluation of the MDM ELBO for the same trained decoder-only network; any mismatch would show Eq. (5) fails for finite models. Alternatively, train AO-GPT and an encoder-only MDM at the same scale with the same token budget and compare zero-shot perplexity after context-order ensembling: if the decoder-only gap persists, the cause is not the larger order-dependent conditional space.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is captured by Eq. (5): $\mathcal{L}_{\mathrm{MDM}} = \mathcal{L}_{\mathrm{AO-AR}}$, so a masked diffusion language model trained with causal attention is exactly an autoregressive model trained on all $n!$ token permutations, and any empirical difference from standard left-to-right AR in the same architecture comes from the distribution over orders. With architecture held fixed, left-to-right order converges faster than uniform any-order, and mixing 10% left-to-right data into any-order training improves both objectives. With formulation held fixed, an encoder-only MDM parameterizes $n \cdot 2^{n-1}$ order-invariant univariate conditionals while a decoder-only MDM parameterizes roughly $e \cdot n!$ order-dependent conditionals; ensembling over context orders largely closes the perplexity gap, showing the decoder's harder task is the main source of the gap. The practical consequence is that a decoder-only MDM with KV-caching and the Lemma 1 sampling shortcut generates a sequence of length $n$ in $O(n)$ time instead of $O(n^2)$, yielding the observed ~25x wall-clock speedup and comparable generation perplexity ($4.611$ vs. $5.051$ at 1024 steps with temperature 0.7).

Load-bearing premise

The load-bearing premise is that Eq. (5) holds for the actual trained network, so the slower convergence and higher perplexity of decoder-only AO-GPT are caused by the uniform token-order distribution rather than by optimization failure, Monte Carlo noise, or approximation error in the objective.

Editorial extensions

If this is right

  • Any architectural or training improvement to any-order autoregressive models transfers directly to masked diffusion models, because their objectives are the same expectation.
  • Comparisons between AR and MDM that change architecture at the same time as formulation are uninformative; the paper's paired experiments provide the clean separation.
  • Uniform order sampling is a poor default for language; non-uniform or left-to-right-biased order distributions are a promising axis for improving MDM training.
  • Decoder-only MDMs offer a practical efficiency-accuracy trade-off: linear-time generation with comparable perplexity after annealing, at the cost of a larger conditional probability space.
  • Adding a small fraction of left-to-right data can serve as a cheap regularizer that improves even any-order performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test this, but the identity in Eq. (5) suggests that any token-order curriculum for AR models (block-wise order, learned orders, insertion orders) can be read as a masked-diffusion schedule, making order-distribution design a shared lever for both families.
  • The context-order ensembling result is evaluated at inference, but it points to a training-side extension: augmenting decoder-only any-order training with random context permutations could close the remaining perplexity gap without switching to full attention.
  • The equal-loss-lower-bound argument implies the left-to-right advantage is an optimization and inductive-bias effect, so at larger scale or with much longer training the gap may shrink; this is an inference, since the paper only reports models up to about 350M parameters.
  • For discrete modalities without a natural left-to-right order, the uniform any-order objective may fit better than it does for language, but the paper does not test this.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces AO-GPT, a decoder-only transformer trained with an any-order autoregressive (AO-AR) objective, and argues that because masked diffusion models (MDMs) and AO-AR models are equivalent formulations (Eq. 5), this setup permits a fair comparison of AR versus MDM paradigms within a decoder-only architecture and an architectural comparison of encoder-only versus decoder-only MDMs. Experiments on OpenWebText at 125M and 350M scales report that uniform-order training converges more slowly than left-to-right training, that mixing 10% left-to-right data improves both left-to-right and any-order perplexity, and that AO-GPT-Medium achieves roughly a 25x generation speedup over SEDD-Medium with comparable generation perplexity after temperature annealing. The paper concludes that MDM's uniform order-agnosticism is suboptimal for natural language and that decoder-only MDMs offer a favorable speed/quality trade-off.

Significance. If the central equivalence held, this would be a valuable clean separation of generative formulation from architectural choice, plus a practical decoder-only masked diffusion model with a large inference speedup. The paper has genuine strengths: the combinatorial count in Finding 4 is correct, the proof of Lemma 1 is sound, the ablation study of target-position injection and EMA is informative, and the code is released. However, the central theoretical identification is not established for the actually trained decoder-only model. Because AO-GPT's conditionals are order-dependent (Finding 4 and Eq. 6) and ensembling over context order is needed to approach encoder-only performance (Finding 5), the right-hand side of Eq. (5) is not the MDM ELBO for this network. The empirical findings remain potentially useful, but the paper's main interpretive claim and its comparison of AO-GPT with SEDD need substantial reframing and additional uncertainty quantification.

major comments (3)
  1. [Section 3, Eq. (5), and Section 4.1, Finding 4] The equivalence L_MDM = L_AO-AR is derived for conditionals p_theta(x_i|x_S) that depend on the context only as an unordered set. The paper itself shows that decoder-only AO-GPT conditionals are order-dependent, p_theta(x_j|x_E, sigma_E), and counts approximately e*n! such conditionals versus n*2^(n-1) for the encoder. Finding 5 then shows that averaging over M context permutations is required to bring the decoder's perplexity in line with encoder-only models. Therefore, for the actual AO-GPT network, the right-hand side of Eq. (5) is not equal to the left-hand side: the trained model is not optimizing the MDM ELBO. This undermines the claim that comparing AO-GPT with SEDD isolates the architectural variable, and also undermines the title's identification of any-order GPT with a masked diffusion model. The authors should either prove the equivalence for order-dependent conditionals or explicitly reframe the paper's contribution as a comparison of decoder-only AO-AR with encoder-only MDM, adjusting the abstract and title accordingly.
  2. [Section 4.3, Table 2 and Figure 4] The headline quantitative claims of a ~25x speedup and comparable perplexity under annealing rest on single training runs and point estimates without error bars or multiple seeds. Given the confound described in the previous comment, it is especially important to know whether the observed differences are stable. Please report means and standard deviations over at least three seeds for the generation perplexity measurements, and state the hardware and measurement conditions for the timing results. If multi-seed experiments are not feasible at this scale, the claims should be softened to single-run observations rather than presented as general findings.
  3. [Section 3, Remark 1 and Findings 1-3] Remark 1 correctly notes that the optimal loss is the same for every permutation only for a perfect model. With finite capacity and stochastic optimization, the observed convergence gap between left-to-right and any-order training may reflect optimization difficulty rather than the intrinsic informativeness of the order distribution. The claim that many permutations are 'less informative' would be strengthened by a controlled comparison that matches the number of gradient steps per token and by ablating the 10% L2R fraction to show that the benefit is not simply a data-level trade-off. As written, the causal interpretation in Findings 1-3 is plausible but not fully established.
minor comments (5)
  1. [Section 4.3] The findings numbering is inconsistent: the text defines Finding 6 and Finding 7.1/7.2, but later refers to 'Findings 8.1 and 8.2'. Please renumber to avoid confusion.
  2. [Figures 9-11] Several figure captions use 'OA-GPT' instead of 'AO-GPT'. Please correct these typos.
  3. [Section 4.2, Eq. (7)] The ensemble definition always includes the identity permutation, so for M=1 the 'ensemble' is not an average. The text should clarify this point, as it affects the interpretation of the leftmost point in Figure 3.
  4. [Section 2.3] The statement that data processing 'exactly follow[s] SEDD' makes reproducibility difficult. Please include a brief description of tokenization, packing, and the exact train/evaluation split in the appendix.
  5. [Table 2] Please specify whether the generated perplexity is computed on the same output samples for both models and whether any decoding randomness (e.g., Gumbel noise) is controlled across runs.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular step found; Eq. (5) is an imported, independently supported equivalence and the empirical claims are external measurements.

full rationale

I find no circular step that reduces a predicted quantity to a fitted input or to the paper's own definitions. Equation (5), which identifies the MDM ELBO with the AO-AR loss, is imported from prior derivations (NADE [11], MDLM [16-18], RADD [19]); although RADD shares authors with the present paper, the equivalence is independently supported by NADE and by the cited MDLM line, and the paper's own contributions (Findings 1-7, Tables 1-3, Figure 4) are empirical measurements against external benchmarks. The temperature and top-p settings in Table 2 are post hoc choices, not fitted parameters relabeled as predictions, and the paper also reports the un-annealed results, so the annealing comparison is transparent. The order-invariance limitation of Eq. (5) for decoder-only models is explicitly acknowledged and studied in Section 4 (Findings 4-5, Eq. 7); it is a correctness or assumption concern, not a circularity. The paper's stated limitations about scale and language-only experiments do not reveal any reduction of conclusions to their inputs.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its central assumptions are the cited MDM/AO-AR equivalence, the capacity-counting interpretation of the perplexity gap, and the standard independent factorization of the reverse process. The free parameters are training and sampling choices, none of which are fitted constants in a derivation.

free parameters (4)
  • L2R data fraction = 10%
    All reported AO-GPT models are trained with 10% left-to-right ordered data; this is an ad hoc choice that improves both L2R and any-order perplexity, but it is not swept or justified by a principled criterion.
  • EMA decay = 0.9999
    Selected from ablation over 0.99, 0.999, 0.9999 in Section 5.3.2; the best value is used in the final model.
  • Annealing temperature = 0.7
    Chosen as 'appropriate annealing' to make AO-GPT's generation perplexity comparable to SEDD's; the comparison at temperature 1.0 shows AO-GPT worse.
  • Target PE hidden dim = 128
    Chosen to minimize parameter increase from adaLN; no ablation reported.
assumptions (3)
  • domain assumption MDM and AO-AR training objectives are equivalent (L_MDM = L_AO-AR)
    Eq. (5) in Section 3 states the equivalence citing RADD [19] and NADE [11]; the paper's entire framing depends on this. If the equivalence fails for finite networks or finite Monte Carlo samples, the interpretation of the experiments changes.
  • ad hoc to paper Decoder-only and encoder-only architectures differ in the number of distinct univariate conditionals they can represent (e*n! vs n*2^(n-1)), and this capacity difference is the dominant cause of the observed perplexity gap
    Finding 4/5 in Section 4.1-4.2. The ensembling experiment supports this, but it assumes the model actually learns all these distinct conditionals and that context-order sensitivity is not confounded with other architectural effects.
  • domain assumption The reverse process factorizes as a product of independent univariate conditionals q_{s|t}(x_s|xt)
    This is the standard MDM reverse-process assumption used in Eq. (8) and Lemma 1; it underlies the parallel generation mask and the speedup claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture." pith.science (2026). https://pith.science/paper/NZ7T6YEF

@misc{pith2026250619935,
  author       = {Pith},
  title        = {Pith review of: Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NZ7T6YEF}},
  note         = {Machine review of arXiv:2506.19935}
}
abstract

Large language models (LLMs) predominantly use autoregressive (AR) approaches, but masked diffusion models (MDMs) are emerging as viable alternatives. A key challenge in comparing AR and MDM paradigms is their typical architectural difference: AR models are often decoder-only, while MDMs have largely been encoder-only. This practice of changing both the modeling paradigm and architecture simultaneously makes direct comparisons unfair, as it's hard to distinguish whether observed differences stem from the paradigm itself or the architectural shift. This research evaluates MDMs within a decoder-only framework to: (1) equitably compare MDM (as Any-Order AR, or AO-AR) and standard AR paradigms. Our investigation suggests that the standard AO-AR objective, which averages over all token permutations, may benefit from refinement, as many permutations appear less informative compared to the language's inherent left-to-right structure. (2) Investigate architectural influences (decoder-only vs. encoder-only) within MDMs. We demonstrate that while encoder-only MDMs model a simpler conditional probability space, decoder-only MDMs can achieve dramatic generation speedups ($\sim25\times$) and comparable perplexity with temperature annealing despite modeling a vastly larger space, highlighting key trade-offs. This work thus decouples core paradigm differences from architectural influences, offering insights for future model design. Code is available at https://github.com/scxue/AO-GPT-MDM.

Figures

Figures reproduced from arXiv: 2506.19935 by the authors.

Figure 1
Figure 1. Training loss curves comparing a standard AR GPT against an AO-GPT. Both models em [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. (a) Convergence speed with different fixed prediction orders: left-to-right, fixed random, [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Zero-shot unconditional perplexity (↓) for varying ensemble sizes. An ensemble size of 1 represents the baseline model without ensembling. Finding 5: Decoder-only AO-AR falls short of their Encoder-only counterpart, while ensemble on order context fills the gap. This observation validates our initial hypothesis. The dramatic reduction in perplexity, which brings the decoder’s performance nearly in line with the enco… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Generation time versus number of generation steps with sequence length [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Target position injection strategies for decoder-only AO-AR model. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Ablation studies for AO-GPT: target positional encoding and exponential moving average. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Combined impact of adaptive layerNorm (AdaLN) and exponential moving average (EMA) [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Attention mask for simultaneous prediction of multiple tokens in AO-GPT. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Unconditional generation result of OA-GPT Medium (top-p [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Unconditional generation result of OA-GPT Medium (top-p [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Unconditional generation result of OA-GPT Medium (top-p [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hierarchical Domain Generalization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Over infinite domains, hierarchy-uniform domain generalization is impossible for every nontrivial hypothesis class; a length-generalization bound is a property of the length hierarchy, not a hierarchy-free guarantee.

  2. Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

    cs.CL 2026-07 accept novelty 6.0 of 10

    Joint AR–diffusion training yields one tri-mode LM that switches AR, diffusion, and self-speculation, beating open AR/diffusion models on accuracy and tokens-per-forward.

  3. Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMs

    cs.CL 2025-10 unverdicted novelty 6.0 of 10

    Masked fine-tuning enables autoregressive LLMs to inject new factual knowledge without paraphrases and with reversal-curse resistance, matching diffusion LLM advantages on QA tasks.

  4. Discrete Diffusion Models: A Unified Framework from Tokenization to Generation

    cs.LG 2026-07 unverdicted novelty 4.0 of 10

    Discrete diffusion models are re-framed as instances of a tokenization-centric, four-component design space (corruption, denoiser, objective, sampler) in a broad survey with no new experimental or theoretical results.

Reference graph

Works this paper leans on

38 extracted references · 17 canonical work pages · cited by 4 Pith papers

  1. [1]

    Deep unsupervised learning using nonequilibrium thermodynamics,

    J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” inInternational conference on machine learning, pp. 2256–2265, pmlr, 2015

    2015

  2. [2]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,”Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020. 14

    work page 2020

  3. [3]

    Score-based gen- erative modeling through stochastic differential equations,

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based gen- erative modeling through stochastic differential equations,”arXiv preprint arXiv:2011.13456, 2020

    arXiv 2011

  4. [4]

    Structured denoising diffusion models in discrete state-spaces,

    J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. Van Den Berg, “Structured denoising diffusion models in discrete state-spaces,”Advances in neural information processing systems, vol. 34, pp. 17981–17993, 2021

    work page 2021

  5. [5]

    A continuous time framework for discrete denoising models,

    A. Campbell, J. Benton, V . De Bortoli, T. Rainforth, G. Deligiannidis, and A. Doucet, “A continuous time framework for discrete denoising models,”Advances in Neural Information Processing Systems, vol. 35, pp. 28266–28279, 2022

    work page 2022

  6. [6]

    Discrete diffusion modeling by estimating the ratios of the data distribution,

    A. Lou, C. Meng, and S. Ermon, “Discrete diffusion modeling by estimating the ratios of the data distribution,”arXiv preprint arXiv:2310.16834, 2023

    arXiv 2023

  7. [7]

    Large language diffusion models,

    S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y . Lin, J.-R. Wen, and C. Li, “Large language diffusion models,”arXiv preprint arXiv:2502.09992, 2025

    arXiv 2025

  8. [8]

    Dream 7b,

    J. Ye, Z. Xie, L. Zheng, J. Gao, Z. Wu, X. Jiang, Z. Li, and L. Kong, “Dream 7b,” 2025

    work page 2025

Show all 38 references
  1. [9]

    Entp: Encoder-only next token prediction,

    E. Ewer, D. Chae, T. Zeng, J. Kim, and K. Lee, “Entp: Encoder-only next token prediction,” arXiv preprint arXiv:2410.01600, 2024

    2024 arXiv

  2. [10]

    Autoregressive image generation without vector quan- tization,

    T. Li, Y . Tian, H. Li, M. Deng, and K. He, “Autoregressive image generation without vector quan- tization,”Advances in Neural Information Processing Systems, vol. 37, pp. 56424–56445, 2024

    2024

  3. [11]

    Neural autoregressive distri- bution estimation,

    B. Uria, M.-A. Côté, K. Gregor, I. Murray, and H. Larochelle, “Neural autoregressive distri- bution estimation,”Journal of Machine Learning Research, vol. 17, no. 205, pp. 1–37, 2016

    2016

  4. [12]

    Xlnet: Generalized autoregressive pretraining for language understanding,

    Z. Yang, Z. Dai, Y . Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V . Le, “Xlnet: Generalized autoregressive pretraining for language understanding,”Advances in neural information processing systems, vol. 32, 2019

    2019

  5. [13]

    Autoregressive diffusion models,

    E. Hoogeboom, A. A. Gritsenko, J. Bastings, B. Poole, R. v. d. Berg, and T. Salimans, “Autoregressive diffusion models,”arXiv preprint arXiv:2110.02037, 2021

    2021 arXiv

  6. [14]

    σ-gpts: A new approach to autoregressive models,

    A. Pannatier, E. Courdier, and F. Fleuret, “σ-gpts: A new approach to autoregressive models,” inJoint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 143–159, Springer, 2024

    2024

  7. [15]

    Argmax flows and multinomial diffusion: Learning categorical distributions,

    E. Hoogeboom, D. Nielsen, P. Jaini, P. Forré, and M. Welling, “Argmax flows and multinomial diffusion: Learning categorical distributions,”Advances in neural information processing systems, vol. 34, pp. 12454–12465, 2021

    2021

  8. [16]

    Simplified and generalized masked diffusion for discrete data,

    J. Shi, K. Han, Z. Wang, A. Doucet, and M. Titsias, “Simplified and generalized masked diffusion for discrete data,”Advances in neural information processing systems, vol. 37, pp. 103131–103167, 2024

    2024

  9. [17]

    Simple and effective masked diffusion language models,

    S. Sahoo, M. Arriola, Y . Schiff, A. Gokaslan, E. Marroquin, J. Chiu, A. Rush, and V . Kuleshov, “Simple and effective masked diffusion language models,”Advances in Neural Information Processing Systems, vol. 37, pp. 130136–130184, 2024

    2024

  10. [18]

    Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling,

    K. Zheng, Y . Chen, H. Mao, M.-Y . Liu, J. Zhu, and Q. Zhang, “Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling,”arXiv preprint arXiv:2409.02908, 2024

    2024 arXiv

  11. [19]

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data,

    J. Ou, S. Nie, K. Xue, F. Zhu, J. Sun, Z. Li, and C. Li, “Your absorbing discrete diffusion secretly models the conditional distributions of clean data,”arXiv preprint arXiv:2406.03736, 2024

    2024 arXiv

  12. [20]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologie...

    2019

  13. [21]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei, “Scaling laws for neural language models,”arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  14. [22]

    Discrete copula diffusion,

    A. Liu, O. Broadrick, M. Niepert, and G. V . d. Broeck, “Discrete copula diffusion,”arXiv preprint arXiv:2410.01949, 2024

    2024 arXiv

  15. [23]

    Energy- based diffusion language models for text generation,

    M. Xu, T. Geffner, K. Kreis, W. Nie, Y . Xu, J. Leskovec, S. Ermon, and A. Vahdat, “Energy- based diffusion language models for text generation,”arXiv preprint arXiv:2410.21357, 2024

    2024 arXiv

  16. [24]

    Openwebtext corpus

    A. Gokaslan and V . Cohen, “Openwebtext corpus.” http://Skylion007.github.io/ OpenWebTextCorpus, 2019

    2019

  17. [25]

    The lambada dataset: Word prediction requiring a broad discourse context,

    D. Paperno, G. Kruszewski, A. Lazaridou, Q. N. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, and R. Fernández, “The lambada dataset: Word prediction requiring a broad discourse context,”arXiv preprint arXiv:1606.06031, 2016

    2016 arXiv

  18. [26]

    Pointer sentinel mixture models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” 2016

    2016

  19. [27]

    Building a large annotated corpus of english: The penn treebank,

    M. Marcus, B. Santorini, and M. A. Marcinkiewicz, “Building a large annotated corpus of english: The penn treebank,”Computational linguistics, vol. 19, no. 2, pp. 313–330, 1993

    1993

  20. [28]

    One billion word benchmark for measuring progress in statistical language modeling,

    C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson, “One billion word benchmark for measuring progress in statistical language modeling,”arXiv preprint arXiv:1312.3005, 2013

    2013 arXiv

  21. [29]

    Block diffusion: Interpolating between autoregressive and diffusion language models,

    M. Arriola, A. Gokaslan, J. T. Chiu, Z. Yang, Z. Qi, J. Han, S. S. Sahoo, and V . Kuleshov, “Block diffusion: Interpolating between autoregressive and diffusion language models,”arXiv preprint arXiv:2503.09573, 2025

    2025 arXiv

  22. [30]

    Film: Visual reasoning with a general conditioning layer,

    E. Perez, F. Strub, H. De Vries, V . Dumoulin, and A. Courville, “Film: Visual reasoning with a general conditioning layer,” inProceedings of the AAAI conference on artificial intelligence, vol. 32, 2018

    2018

  23. [31]

    Scalable diffusion models with transformers,

    W. Peebles and S. Xie, “Scalable diffusion models with transformers,” inProceedings of the IEEE/CVF international conference on computer vision, pp. 4195–4205, 2023

    2023

  24. [32]

    Made: Masked autoencoder for distribu- tion estimation,

    M. Germain, K. Gregor, I. Murray, and H. Larochelle, “Made: Masked autoencoder for distribu- tion estimation,” inInternational conference on machine learning, pp. 881–889, PMLR, 2015

    2015

  25. [33]

    Arbitrary conditional distributions with energy,

    R. Strauss and J. B. Oliva, “Arbitrary conditional distributions with energy,”Advances in Neural Information Processing Systems, vol. 34, pp. 752–763, 2021

    2021

  26. [34]

    Training and inference on any-order autoregressive models the right way,

    A. Shih, D. Sadigh, and S. Ermon, “Training and inference on any-order autoregressive models the right way,”Advances in Neural Information Processing Systems, vol. 35, pp. 2762–2775, 2022

    2022

  27. [35]

    Insertion-based decoding with automatically inferred generation order,

    J. Gu, Q. Liu, and K. Cho, “Insertion-based decoding with automatically inferred generation order,”Transactions of the Association for Computational Linguistics, vol. 7, pp. 661–676, 2019

    2019

  28. [36]

    Train for the worst, plan for the best: Understanding token ordering in masked diffusions,

    J. Kim, K. Shah, V . Kontonis, S. Kakade, and S. Chen, “Train for the worst, plan for the best: Understanding token ordering in masked diffusions,”arXiv preprint arXiv:2502.06768, 2025

    2025 arXiv

  29. [37]

    Randomized autoregressive visual generation,

    Q. Yu, J. He, X. Deng, X. Shen, and L.-C. Chen, “Randomized autoregressive visual generation,” arXiv preprint arXiv:2411.00776, 2024

    2024 arXiv

  30. [38]

    Randar: Decoder-only autoregressive visual generation in random orders,

    Z. Pang, T. Zhang, F. Luan, Y . Man, H. Tan, K. Zhang, W. T. Freeman, and Y .-X. Wang, “Randar: Decoder-only autoregressive visual generation in random orders,”arXiv preprint arXiv:2412.01827, 2024. 16 Table 4: AO-GPT Model Specifications Parameter Small Medium nlayers 12 24 d...

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.