{"id":"8090a88b-086f-4362-9b6a-d630bde44f20","arxiv_id":"2506.19935","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A decoder-only masked diffusion model (AO-GPT) enables a fairer comparison of autoregressive and diffusion language models, showing that uniform random token order is suboptimal and that decoder-only diffusion offers a large generation speedup.","lead":"This paper builds a decoder-only masked diffusion model, AO-GPT, to compare autoregressive and diffusion-style language modeling without changing the architecture at the same time. It finds that adding a small amount of left-to-right training data helps any-order models, and that decoder-only diffusion can generate text about 25 times faster than an encoder-only model with comparable quality after temperature tuning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (5) equates the MDM ELBO with the AO-AR loss only for an order-invariant model; AO-GPT's decoder conditionals are order-dependent, so the trained network does not satisfy the claimed equivalence, and the paper's 'decoupling' of formulation from architecture is not established.","rationale":"The reader's conditional verdict centers on whether Eq. (5)'s equivalence survives finite capacity and Monte Carlo approximation. My concern is more structural: the equivalence is between two losses for a model whose conditionals are order-invariant. AO-GPT's decoder uses causal attention with target-position conditioning, making its conditionals order-dependent, as the paper's Finding 4 and Section 4.2 demonstrate. Hence the trained AO-GPT is not optimizing the MDM ELBO; it is optimizing a different objective (the AO-AR loss over causal prefixes). The empirical comparison with SEDD therefore conflates the objective mismatch with the architectural difference, which is precisely the confound the paper aims to avoid. If the proposed test shows L_MDM ≈ L_AO-AR for the released checkpoint, the concern is resolved and the decoupling holds approximately. If not, the central theoretical claim must be revised (e.g., by treating AO-GPT strictly as an AO-AR model and limiting 'MDM' claims to the encoder-only case). Other issues—the 10% left-to-right training data, temperature tuning, and lack of error bars—are important but secondary; they affect the strength of the empirical claims without invalidating the conceptual framework as directly as the equivalence failure does. I therefore agree with the conditional verdict, but for a more fundamental reason than the reader's stated weakest assumption.","tokens_in":20416,"tokens_out":13853,"duration_ms":146696,"concrete_test":"Using the released AO-GPT-Medium checkpoint, evaluate both loss terms in Eq. (5) on the same validation batch: (1) the AO-AR loss L_AO-AR, computed by sampling permutations σ and averaging Σ_i -log p_θ(x_σ_i|x_σ_<i) under the causal prefix; (2) the MDM ELBO L_MDM, computed by sampling masking times t and masks, and evaluating the model's prediction for each masked position with the Figure 8 parallel attention mask (unmasked context in its original order) for Σ_{i masked} -log p_θ(x_i|masked). If L_MDM exceeds L_AO-AR by more than a small tolerance (e.g., 1%), then Eq. (5) does not hold for the trained decoder-only model, and the identification of AO-GPT as an MDM is unsupported. The same calculation for an encoder-only SEDD checkpoint should show approximate equality, confirming the test's sensitivity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central premise of the paper is that masked diffusion models (MDMs) and any-order autoregressive models (AO-AR) are equivalent formulations (Eq. 5), so comparing AO-GPT (decoder-only, trained with the AO-AR loss) with SEDD (encoder-only, trained with an MDM loss) isolates the architectural variable. Eq. (5) is a chain of equalities between L_MDM and L_AO-AR for a fixed conditional model pθ(x_i|x_S) that is order-invariant: the MDM term pθ(x_i^0|x_t) depends only on the set of unmasked tokens, and the NADE/RADD derivations cited assume the same network is used for both masks and ordered prefixes. The paper itself shows this assumption fails for decoder-only models. Section 4.1 (Finding 4) establishes that decoder-only AO-AR models parameterize approximately e·n! order-dependent conditionals, versus n·2^(n-1) order-invariant conditionals for encoder-only models. Section 4.2 (Finding 5) shows that averaging over M random permutations of the context (Eq. 7) is required to bring AO-GPT's perplexity in line with encoder-only models—direct empirical evidence that the trained decoder is not order-invariant. Consequently, for the actual AO-GPT network, the right-hand side of Eq. (5) (the AO-AR objective over permutation prefixes) is not equal to the left-hand side (the MDM ELBO over masks), because the conditional used during causal training differs from the conditional used in parallel masked generation (Figure 8). The trained model is therefore not optimizing the MDM ELBO, and the observed performance gap between AO-GPT and SEDD cannot be attributed solely to architecture; it also reflects an objective mismatch. This undermines the paper's headline claim of decoupling formulation from architecture.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AO-GPT, a decoder-only transformer trained with an any-order autoregressive (AO-AR) objective, and argues that because masked diffusion models (MDMs) and AO-AR models are equivalent formulations (Eq. 5), this setup permits a fair comparison of AR versus MDM paradigms within a decoder-only architecture and an architectural comparison of encoder-only versus decoder-only MDMs. Experiments on OpenWebText at 125M and 350M scales report that uniform-order training converges more slowly than left-to-right training, that mixing 10% left-to-right data improves both left-to-right and any-order perplexity, and that AO-GPT-Medium achieves roughly a 25x generation speedup over SEDD-Medium with comparable generation perplexity after temperature annealing. The paper concludes that MDM's uniform order-agnosticism is suboptimal for natural language and that decoder-only MDMs offer a favorable speed/quality trade-off.","tokens_in":20771,"tokens_out":6713,"duration_ms":78307,"significance":"If the central equivalence held, this would be a valuable clean separation of generative formulation from architectural choice, plus a practical decoder-only masked diffusion model with a large inference speedup. The paper has genuine strengths: the combinatorial count in Finding 4 is correct, the proof of Lemma 1 is sound, the ablation study of target-position injection and EMA is informative, and the code is released. However, the central theoretical identification is not established for the actually trained decoder-only model. Because AO-GPT's conditionals are order-dependent (Finding 4 and Eq. 6) and ensembling over context order is needed to approach encoder-only performance (Finding 5), the right-hand side of Eq. (5) is not the MDM ELBO for this network. The empirical findings remain potentially useful, but the paper's main interpretive claim and its comparison of AO-GPT with SEDD need substantial reframing and additional uncertainty quantification.","major_comments":[{"comment":"The equivalence L_MDM = L_AO-AR is derived for conditionals p_theta(x_i|x_S) that depend on the context only as an unordered set. The paper itself shows that decoder-only AO-GPT conditionals are order-dependent, p_theta(x_j|x_E, sigma_E), and counts approximately e*n! such conditionals versus n*2^(n-1) for the encoder. Finding 5 then shows that averaging over M context permutations is required to bring the decoder's perplexity in line with encoder-only models. Therefore, for the actual AO-GPT network, the right-hand side of Eq. (5) is not equal to the left-hand side: the trained model is not optimizing the MDM ELBO. This undermines the claim that comparing AO-GPT with SEDD isolates the architectural variable, and also undermines the title's identification of any-order GPT with a masked diffusion model. The authors should either prove the equivalence for order-dependent conditionals or explicitly reframe the paper's contribution as a comparison of decoder-only AO-AR with encoder-only MDM, adjusting the abstract and title accordingly.","section":"Section 3, Eq. (5), and Section 4.1, Finding 4"},{"comment":"The headline quantitative claims of a ~25x speedup and comparable perplexity under annealing rest on single training runs and point estimates without error bars or multiple seeds. Given the confound described in the previous comment, it is especially important to know whether the observed differences are stable. Please report means and standard deviations over at least three seeds for the generation perplexity measurements, and state the hardware and measurement conditions for the timing results. If multi-seed experiments are not feasible at this scale, the claims should be softened to single-run observations rather than presented as general findings.","section":"Section 4.3, Table 2 and Figure 4"},{"comment":"Remark 1 correctly notes that the optimal loss is the same for every permutation only for a perfect model. With finite capacity and stochastic optimization, the observed convergence gap between left-to-right and any-order training may reflect optimization difficulty rather than the intrinsic informativeness of the order distribution. The claim that many permutations are 'less informative' would be strengthened by a controlled comparison that matches the number of gradient steps per token and by ablating the 10% L2R fraction to show that the benefit is not simply a data-level trade-off. As written, the causal interpretation in Findings 1-3 is plausible but not fully established.","section":"Section 3, Remark 1 and Findings 1-3"}],"minor_comments":[{"comment":"The findings numbering is inconsistent: the text defines Finding 6 and Finding 7.1/7.2, but later refers to 'Findings 8.1 and 8.2'. Please renumber to avoid confusion.","section":"Section 4.3"},{"comment":"Several figure captions use 'OA-GPT' instead of 'AO-GPT'. Please correct these typos.","section":"Figures 9-11"},{"comment":"The ensemble definition always includes the identity permutation, so for M=1 the 'ensemble' is not an average. The text should clarify this point, as it affects the interpretation of the leftmost point in Figure 3.","section":"Section 4.2, Eq. (7)"},{"comment":"The statement that data processing 'exactly follow[s] SEDD' makes reproducibility difficult. Please include a brief description of tokenization, packing, and the exact train/evaluation split in the appendix.","section":"Section 2.3"},{"comment":"Please specify whether the generated perplexity is computed on the same output samples for both models and whether any decoding randomness (e.g., Gumbel noise) is controlled across runs.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The empirical study is careful and the released code is a strength, but the central conceptual claim connecting any-order GPT to masked diffusion is not valid for the decoder-only architecture as trained. I recommend major revision rather than rejection because the framing can be corrected: the paper can be repositioned as a study of decoder-only any-order autoregressive models and their relationship to encoder-only masked diffusion models, with the equivalence explicitly stated as holding only in the order-invariant limit. If the authors choose not to correct the central claim, I would recommend rejection. The lack of error bars on the main quantitative claims should also be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my honest read. The paper is a worthwhile empirical study of any-order decoder-only language models, but its central theoretical framing is not sound as stated. The equivalence in Eq. (5) between the MDM ELBO and the AO-AR loss is standard for order-invariant conditionals, as in NADE or RADD. AO-GPT is a causal decoder, so its conditionals are order-dependent — the paper itself proves this in Finding 4 and then uses an ensemble over context orders (Finding 5) to partially recover order-invariance. That means the trained model is not optimizing the MDM objective; it is optimizing the AO-AR loss over permutations, and that is not the same thing for a causal architecture. So the headline claim of cleanly decoupling formulation and architecture does not hold. The comparison between AO-GPT and SEDD conflates the architectural difference with an objective mismatch.\n\nThat said, the empirical results stand on their own. Finding 3 — mixing 10% left-to-right data into any-order training improves both L2R and any-order perplexity — is a concrete and original result that I expect to be reproducible. The ~25x speedup from KV-cache combined with the Lemma 1 sampling trick is real, measured at 1024 length, and practically relevant. The context-order ensemble is a neat idea that works. Finding 4's counting is correct and clearly presented.\n\nSoft spots, in proportion: the headline perplexity and speed claims are based on single runs with no error bars or multiple seeds. The AO-GPT model uses 10% L2R data while the SEDD and RADD baselines do not, so the architecture comparison is not fully controlled. 'Comparable perplexity' with SEDD appears only at temperature 0.7; without annealed sampling AO-GPT is clearly worse. These are addressable, but they should be fixed before the paper's claims are taken as established.\n\nMy bottom line: this is a solid empirical contribution that could become a good paper after a major revision. The authors should either reframe it as a study of any-order decoder-only models and explicitly note that the MDM equivalence requires order-invariance, or show empirically that the ensembled decoder matches the encoder-only objective. I would send it to peer review rather than desk-reject, because the findings are useful and the community would benefit from a carefully corrected version. I'd cite the 10% L2R training trick in my own work.","headline":"Useful empirical findings (10% L2R mix, ~25x speedup, order-ensemble) paired with an over-sold equivalence claim that breaks for the order-dependent decoder model the paper actually trains.","tokens_in":21385,"tokens_out":5718,"would_cite":true,"duration_ms":54323,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Masked diffusion models and any-order autoregressive models are the same formulation, and a decoder-only implementation of that formulation generates about 25 times faster than an encoder-only one while reaching comparable perplexity…","keywords":["masked diffusion models","any-order autoregressive","decoder-only architecture","token-order distribution","generation speedup","perplexity","discrete diffusion language models","temperature annealing"],"falsifier":"Enumerate all $n!$ orders for short sequences and compare the exact AO-AR loss with a high-precision numerical evaluation of the MDM ELBO for the same trained decoder-only network; any mismatch would show Eq. (5) fails for finite models. Alternatively, train AO-GPT and an encoder-only MDM at the same scale with the same token budget and compare zero-shot perplexity after context-order ensembling: if the decoder-only gap persists, the cause is not the larger order-dependent conditional space.","tokens_in":20165,"feed_emoji":"⚡","tokens_out":8188,"duration_ms":82429,"temperature":0.7,"pith_summary":"The paper establishes that masked diffusion models and any-order autoregressive models optimize the same objective, so the paradigm difference between them collapses to a choice of token-order distribution. Using a decoder-only architecture that can predict any token given any ordered prefix, the paper shows that the uniform any-order objective converges more slowly than left-to-right and that adding a small fraction of left-to-right data improves both left-to-right and any-order performance. Within the masked-diffusion formulation, encoder-only models represent far fewer conditional probabilities than decoder-only models, but decoder-only models reduce generation complexity from quadratic to linear and achieve a measured speedup of roughly 25x at comparable generation perplexity after temperature annealing. The reason to care is that these results separate what objective is being optimized from what network shape is used, which is the only way to compare autoregressive and diffusion language models fairly.","feed_headline":"Masked diffusion equals any-order AR, and decoder-only is 25x faster","feed_subtitle":"Uniform any-order training is the slowdown; decoder-only generation scales linearly, not quadratically.","key_machinery":"The load-bearing identity is Eq. (5), which chains two existing derivations: the RADD derivation showing that the masked-diffusion ELBO equals a nested expectation over mask levels, and the NADE derivation showing that nested expectation equals the uniform average over all $n!$ generation orders. The supporting mechanism is AO-GPT's decoder-only any-order architecture: causal attention plus adaptive LayerNorm conditioned on the target token's original position, so the network can predict any position conditioned on any ordered prefix. The speed mechanism is the pairing of a KV-cache with Lemma 1's two-stage sampling, which replaces a per-token model evaluation by a Bernoulli draw that often leaves the token masked, and a specialized parallel attention mask that predicts all simultaneously unmasked tokens in one forward pass.","core_discovery":"On the paper's own terms, the central discovery is captured by Eq. (5): $\\mathcal{L}_{\\mathrm{MDM}} = \\mathcal{L}_{\\mathrm{AO-AR}}$, so a masked diffusion language model trained with causal attention is exactly an autoregressive model trained on all $n!$ token permutations, and any empirical difference from standard left-to-right AR in the same architecture comes from the distribution over orders. With architecture held fixed, left-to-right order converges faster than uniform any-order, and mixing 10% left-to-right data into any-order training improves both objectives. With formulation held fixed, an encoder-only MDM parameterizes $n \\cdot 2^{n-1}$ order-invariant univariate conditionals while a decoder-only MDM parameterizes roughly $e \\cdot n!$ order-dependent conditionals; ensembling over context orders largely closes the perplexity gap, showing the decoder's harder task is the main source of the gap. The practical consequence is that a decoder-only MDM with KV-caching and the Lemma 1 sampling shortcut generates a sequence of length $n$ in $O(n)$ time instead of $O(n^2)$, yielding the observed ~25x wall-clock speedup and comparable generation perplexity ($4.611$ vs. $5.051$ at 1024 steps with temperature 0.7).","pith_inferences":["The paper does not test this, but the identity in Eq. (5) suggests that any token-order curriculum for AR models (block-wise order, learned orders, insertion orders) can be read as a masked-diffusion schedule, making order-distribution design a shared lever for both families.","The context-order ensembling result is evaluated at inference, but it points to a training-side extension: augmenting decoder-only any-order training with random context permutations could close the remaining perplexity gap without switching to full attention.","The equal-loss-lower-bound argument implies the left-to-right advantage is an optimization and inductive-bias effect, so at larger scale or with much longer training the gap may shrink; this is an inference, since the paper only reports models up to about 350M parameters.","For discrete modalities without a natural left-to-right order, the uniform any-order objective may fit better than it does for language, but the paper does not test this."],"forward_implications":["Any architectural or training improvement to any-order autoregressive models transfers directly to masked diffusion models, because their objectives are the same expectation.","Comparisons between AR and MDM that change architecture at the same time as formulation are uninformative; the paper's paired experiments provide the clean separation.","Uniform order sampling is a poor default for language; non-uniform or left-to-right-biased order distributions are a promising axis for improving MDM training.","Decoder-only MDMs offer a practical efficiency-accuracy trade-off: linear-time generation with comparable perplexity after annealing, at the cost of a larger conditional probability space.","Adding a small fraction of left-to-right data can serve as a cheap regularizer that improves even any-order performance."],"supporting_citations":[{"why":"Supplies the derivation that the any-order autoregressive loss equals a uniform average over permutations.","marker":"[11]"},{"why":"Shows the masked diffusion ELBO reduces to the any-order autoregressive objective, the key identity of Eq. 5.","marker":"[19]"},{"why":"Provides the sigma-GPT decoder-only any-order architecture that AO-GPT builds on for target-position injection.","marker":"[14]"},{"why":"Defines SEDD, the encoder-only masked diffusion baseline, and supplies the training and evaluation protocol.","marker":"[6]"},{"why":"Provides the OpenWebText corpus on which all models are trained.","marker":"[24]"},{"why":"Cited for the float32 Gumbel-noise temperature artifact, motivating the float64 comparison and annealing settings.","marker":"[18]"}],"fun_headline_variants":["Masked diffusion is any-order AR; decoder-only is 25x faster","Decoder-only MDM matches encoder-only with 25x speedup","Any-order AR equals MDM, but uniform order slows training","Left-to-right beats uniform any-order for MDM training","MDM as any-order AR: decoder-only scales linearly, not quadratically"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Eq. (5) holds for the actual trained network, so the slower convergence and higher perplexity of decoder-only AO-GPT are caused by the uniform token-order distribution rather than by optimization failure, Monte Carlo noise, or approximation error in the objective.","fun_headline_variants_meta":{"raw":{"variants":["Masked diffusion is any-order AR; decoder-only is 25x faster","Decoder-only MDM matches encoder-only with 25x speedup","Any-order AR equals MDM, but uniform order slows training","Left-to-right beats uniform any-order for MDM training","MDM as any-order AR: decoder-only scales linearly, not quadratically"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1353,"prompt_tokens":1076,"completion_tokens":277,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":692,"completion_tokens_details":{"reasoning_tokens":184}},"tokens_in":692,"tokens_out":277,"duration_ms":3113,"temperature":1.0,"reasoning_tokens":184,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:22:51.255135+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Enumerate all $n!$ orders for short sequences and compare the exact AO-AR loss with a high-precision numerical evaluation of the MDM ELBO for the same trained decoder-only network; any mismatch would show Eq. (5) fails for finite models. Alternatively, train AO-GPT and an encoder-only MDM at the same scale with the same token budget and compare zero-shot perplexity after context-order ensembling: if the decoder-only gap persists, the cause is not the larger order-dependent conditional space.","supporting_citations":[{"cited_title":"Neural autoregressive distri- bution estimation,","cited_arxiv_id":null,"evidence_quote":"Supplies the derivation that the any-order autoregressive loss equals a uniform average over permutations."},{"cited_title":"σ-gpts: A new approach to autoregressive models,","cited_arxiv_id":null,"evidence_quote":"Provides the sigma-GPT decoder-only any-order architecture that AO-GPT builds on for target-position injection."},{"cited_title":"Openwebtext corpus","cited_arxiv_id":null,"evidence_quote":"Provides the OpenWebText corpus on which all models are trained."}],"review_version":2}