Pith. sign in

REVIEW 3 major objections 5 minor 17 cited by

Masked diffusion now handles variable length in any order

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

FlexMDM is a discrete diffusion model that provably supports any-order generation over variable-length sequences by learning an insertion expectation alongside the unmasking posterior, validated by length-fidelity, maze-planning, and 8B-retrofit experiments.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Variable-length masked diffusion done right, with a real any-order theorem; the scary combinatorial step checks out, but the empirical package is thinner than the theory. the 3 major comments →

arxiv 2509.01025 v2 pith:I5LCFXQV submitted 2025-08-31 cs.LG

Any-Order Flexible Length Masked Diffusion

classification cs.LG MSC 60J2768T07
keywords masked diffusionvariable-length generationany-order inferencecontinuous-time Markov chainstochastic interpolanttoken insertiondiscrete diffusionsequence generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Flexible Masked Diffusion Models (FlexMDMs) claim to remove the fixed-length limitation of masked diffusion models without giving up their any-order, parallel decoding. The key idea is to start from the empty string and treat generation as two coupled operations: inserting new mask tokens between existing tokens, and unmasking tokens from a learned posterior. The paper proves that, under perfect training, the continuous-time Markov chain driven by the learned unmasking posterior and a new insertion expectation has the target variable-length distribution as its terminal law, and that the same guarantee holds when unmasking positions are chosen adaptively in any order. If the claim holds, masked diffusion becomes a viable parallel generator for variable-length text and structured outputs, with length statistics learned rather than padded.

Core claim

The paper introduces a joint interpolant (x_t, s_t), where a clean sequence x_1 is converted into x_t by deleting, masking, or revealing each token according to two schedules, while s_t records which source positions are still present. Minimizing the proposed loss makes a network f_theta recover the ground-truth unmasking posterior and a network g_theta recover the expected number of tokens to insert at each gap. The rate matrix built from these two quantities solves the Kolmogorov forward equation for the interpolant's marginals, so exact learning implies exact sampling from the target distribution p_1. The any-order theorem then states that any sampler that unmasks an arbitrary subset of m

What carries the argument

The joint interpolant: a stochastic interpolant augmented by an index-tracking set s_t. It enlarges the class of constructible rate matrices, making it possible to start from an easy-to-sample base distribution (the empty string) and to express insertion rates as a per-gap scalar, the insertion expectation, rather than a full token distribution. The two schedules alpha_t (insertion) and beta_t (unmasking), the training loss, the rate matrix, and the any-order theorem all hang on this object.

Load-bearing premise

Everything rests on the trained networks being exactly right about both what each mask hides and how many tokens belong in each gap, and on a counting identity about gaps holding up even for strings with repeated tokens.

What would settle it

Train or simulate FlexMDM with oracle (ground-truth) unmasking posterior and insertion expectation on a distribution over strings with many repeated tokens, then check whether the exact continuous-time Markov chain at t=1 matches the training length distribution; a mismatch, especially one that grows with repetition, would falsify the combinatorial step underlying the any-order theorem.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Variable-length text can be generated in parallel with masked diffusion, and length distributions are learned instead of imposed by padding to a fixed canvas.
  • Any-order inference remains theoretically grounded for FlexMDMs, so confidence-based and semi-autoregressive unmasking heuristics can be combined with insertions without breaking the sampling guarantee.
  • Pretrained masked diffusion models can be converted into FlexMDMs by adding an insertion-expectation head and fine-tuning, inheriting the base model's capabilities and adding variable-length generation.
  • On planning tasks, the model can insert tokens between subgoals instead of preallocating their positions, yielding large success-rate gains as the number of subgoals grows.
  • For math and code infilling, more sampling steps continue to improve FlexMDM performance, suggesting reasoning-style gains from additional inference compute.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: a testable extension would stress-test FlexMDMs on corpora with heavy token repetition, since the proof's key counting identity relates gap insertions to supersequence matchings and is most delicate when target sequences contain repeated tokens.
  • Inference: the same joint-interpolant construction could be adapted to other edit operations—deletion, replacement, reordering—to build diffusion models that perform structured edits of sequences rather than only insertions.
  • Inference: the path-independence mechanism that enables any-order unmasking suggests a single trained model could be used at inference with schedules quite different from the training schedules, potentially including learned or task-dependent unmasking orders.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Flexible Masked Diffusion Models (FlexMDMs), a discrete diffusion framework for variable-length sequences. The method defines a joint stochastic interpolant that deletes, masks, and unmasks tokens according to two schedules α_t and β_t, and derives a training loss (Eq. 7) whose minimizers are the unmasking posterior and an insertion expectation. Proposition 2 gives a rate matrix for the corresponding CTMC, and Section 4 claims an any-order inference guarantee: even if masked tokens are revealed in an arbitrary order, as long as unmasking uses the true posterior and insertions follow the ground-truth rate matrix, the terminal distribution is p_1. The paper reports length-distribution fidelity on OpenWebText, success rates on a maze/subgoal planning task, and a 3-day retrofit of LLaDA-8B into a FlexMDM with improved GSM8K and HumanEval-infill scores.

Significance. If the guarantees hold, FlexMDM is a meaningful advance over fixed-length masked diffusion: it models length statistics while preserving any-order decoding, and the LLaDA retrofit suggests a practical upgrade path for existing 8B MDMs. The theoretical construction is largely derived rather than assumed: the loss is a valid variational upper bound, the rate matrix is derived from the interpolant, and the any-order theorem is a substantive oracle-level statement. The main caveats are that the empirical claims are not accompanied by quantitative length-distribution metrics or error bars, and the any-order theorem requires oracle posterior and insertion-expectation access; the learned-model generalization to off-path states is not analyzed. These caveats are standard for the area, and the core contribution is sufficiently novel to merit publication if the proof issues below are fixed.

major comments (3)
  1. [Appendix E.6, Eq. (14)] This is the load-bearing step in Lemma E.2, and as printed it does not prove the required identity. In the product R_ins_t(x,x◁_i m) q_t(x*|x◁_i m), the posterior normalizer for x◁_i m carries exponent len(x1)-len(x)-1 in (1-α_t), whereas Eq. (14) writes the denominator with exponent len(x1)-len(x). The subset bijection established in the text makes the numerator and denominator equal only for the same exponent; with the correct exponent the ratio is (1-α_t). Tracking that factor against the prefactor αdot/(1-α_t) in R_ins restores the equality with the first term of Eq. (13), so the lemma is repairable. Please rewrite Eq. (14) with the correct exponents and clearly state that the bijection is over subsets of source positions (which is why repeated tokens are harmless). The current one-sentence proof also contains a typo in the final display (x* vs x1 and x vs x_t).
  2. [Appendix E.2.1 / E.5] The posterior q_t is extended to t=1 by a separate definition that is not the t→1 limit of the t<1 formula. Lemma E.1 is proved only for the t<1 formula (Eqs. (10)-(11)), but Theorem E.1's base case applies Lemma E.1 at t=1. Please give the t=1 version of Lemma E.1 (it holds by a simple normalization argument: q_1(x*|x) ∝ p(x*) 1_{x⊆x*}, so the any-order unmasking step preserves q_1) or restructure the induction to avoid this case. Without this, the proof of the any-order guarantee has a gap at the final unmasking phase.
  3. [Section 4 / Algorithm 1] Proposition 3 (informal) says 'unmasks an arbitrary subset', while the formal Definition E.1 covers only one index per unmasking step; the adaptive subroutine in Algorithm 1 assigns K positions simultaneously. The paper should clarify that the theorem covers sequential unmasking of subsets and that the simultaneous K-step is an approximation that coincides with the formal class only in the τ→0 limit (where K∼Poisson has at most one event w.h.p.). This is not fatal but is needed for the main-text claim that Algorithm 1 is subsumed by Proposition 3.
minor comments (5)
  1. [§5.1.1 vs Appendix F.1] The main text says pretraining runs for 500K iterations, while Appendix F.1 states a max training iteration of 1M. Please align these numbers.
  2. [Figure 4b/c] The claims of 'much higher fidelity' in length modeling rest on a qualitative plot. Please add a quantitative metric (e.g., total variation or KL to the empirical length distribution) and, where feasible, error bars over seeds.
  3. [Introduction] Typo: 'non-casual tasks' should be 'non-causal tasks'.
  4. [§5.2] The LLaDA comparison is between an IFT-ed baseline and a FlexMDM that additionally trains LoRA adapters, a time-embedding pathway, and a new scalar head. Please state whether the comparison controls for trainable parameters/compute, since the two setups do not have equal capacity.
  5. [Algorithm 1 / Subroutine 2] The pseudocode says 'Select K (the size of |S|)' without specifying the distribution; the distribution (Poisson with rate depending on τ and the unmasking schedule) appears only in Appendix E.1. Please make the pseudocode self-contained.

Circularity Check

0 steps flagged

No significant circularity; FlexMDM's guarantees are derived from stated interpolant definitions and proven in appendices. The only self-citations are contextual and non-load-bearing.

full rationale

FlexMDM's derivation chain is self-contained. The joint interpolant (Eq. 6 / Def. D.2) is a stipulated construction; the rate matrix (Eq. 8 / Prop. D.3) is derived by applying the generic joint-interpolant target-rate proposition (Prop. D.1) to the interpolating rate (Prop. D.2). The loss (Eq. 7 / Prop. D.4) is the path-measure KL divergence; Prop. 1 identifies its unique minimizer as the posterior and insertion expectation used in the rate matrix. This is a characterization, not an assumption of the conclusion. For any-order inference, Theorem E.1 is proven by induction; Lemmas E.1 and E.2 have explicit proofs. Lemma E.2's Eq. (14) is a combinatorial identity established by a bijection ('The key combinatorial step...'), not an imported result; even if the exposition is compressed, this is a correctness/completeness concern, not circularity. The posterior independence from beta_t (Appendix E.2.1) is computed from the interpolant, not assumed. Self-citations (Kim et al. 2025 for MDM any-order folklore; Albergo et al. for stochastic interpolants) are contextual; the paper re-derives the any-order argument for FlexMDM rather than relying on the citation. Empirical length-fidelity is an evaluation of the trained model on the training-length distribution, not a fitted parameter renamed as an independent prediction. Consequently, no load-bearing step reduces to its own input; the low score reflects only the presence of minor, non-load-bearing self-citations.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

Everything the central claims rest on beyond the standard machinery: the CTMC/interpolant toolkit (standard math, largely cited from the authors' own prior framework papers), the per-coordinate independent insertion/unmasking model, oracle-quality learning, and the sketched combinatorial identity at the heart of the any-order insertion proof. The paper is otherwise self-contained: the loss, rates, and guarantees are derived, with the caveat that a reader must take the dense appendix algebra mostly on faith.

free parameters (5)
  • insertion/unmasking schedules alpha_t, beta_t = alpha=beta=t per footnote 2; main text writes alpha=beta=t^2
    Hand-chosen schedules weight the two loss terms and define the CTMC rates; not fitted to data, but a modeling knob and the text is internally inconsistent (t vs t^2).
  • sliding-window heuristics gamma_1, gamma_2 = gamma_1=5.0, gamma_2=64
    Hand-tuned for adaptive inference in the GSM8K and HumanEval-infill evaluations; affects the LLaDA-based results.
  • sampling temperature = 0.0
    Stated as important for strong Pass@1; greedy decoding is a chosen hyperparameter applied to both models in the 8B experiments.
  • generative-perplexity length filter = sequences of <=10 tokens excluded
    Post-hoc selection for the perplexity comparison; disproportionately removes the padded MDM baseline's short and poorly scored samples.
  • LoRA configuration for the 8B retrofit = r=128, alpha=128, dropout=0.1
    Choice for the LLaDA-8B experiment; influences transfer quality and the reported math/code gains.
axioms (5)
  • standard math Kolmogorov forward/backward equations for CTMCs; path-measure KL (Radon-Nikodym) for rate matrices; data processing inequality for the terminal KL bound
    Used throughout Sections 2-3 and Appendices C-D (Eq. 3, Propositions C.1, D.1, D.4); standard results cited to Campbell et al. 2024.
  • domain assumption p_1 is supported on finite-length strings over a finite vocabulary; per-coordinate independent insertion and unmasking times in the joint interpolant (Eq. 6)
    Defines the path, the posterior form (E.2.1), and the rate matrix; if real data requires correlated insertion/unmasking, the simple scalar insertion expectation may be insufficient.
  • ad hoc to paper Oracle access to ground-truth posterior marginals q_t and insertion expectations for the any-order guarantees
    Theorem E.1 / Proposition 3 assume ground-truth quantities (Section 4, Definition E.1); the empirical results assume learned f_theta and g_theta are accurate on adaptive-inference states.
  • ad hoc to paper Combinatorial identity Eq. (14) in the proof of Lemma E.2: the summed gap-insertion counts equal the posterior's time derivative terms in the Kolmogorov backward equation
    Load-bearing step verifying that the insertion CTMC propagates the posterior correctly; stated as 'the key combinatorial step' in one sentence and delicate with repeated tokens.
  • ad hoc to paper Posterior extended to t=1 via q_1(x*|x) proportional to p(x*) times the indicator that x is a sub-mask of x*, with oracle access to any-order marginals of p
    E.2.1: needed because adaptive unmasking can outlast the insertion schedule; in practice approximated by learned marginals at t=1-delta.
invented entities (2)
  • Insertion expectation g_theta(x,t)[i] independent evidence
    purpose: Scalar per position predicting the expected number of tokens still to be inserted in a gap; defines insertion rates in (8) and is the mechanism for modeling the length distribution.
    Falsifiable handles: OpenWebText length-distribution fit (Fig. 4b), maze success rates (Table 1), and the LLaDA-8B GSM8K/infill gains directly test this quantity.
  • Joint interpolant (x_t, s_t) with index-tracking variable s_t no independent evidence
    purpose: Augments the stochastic interpolant to track which source indices remain alive, enabling an easy-to-sample base (empty string) and closed-form rate matrices.
    A mathematical construction; it has no falsifiable handle outside the trained model's behavior, so its correctness is evidenced only indirectly through the derived rates and experiments.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Any-Order Flexible Length Masked Diffusion." pith.science (2026). https://pith.science/paper/I5LCFXQV

@misc{pith2026250901025,
  author       = {Pith},
  title        = {Pith review of: Any-Order Flexible Length Masked Diffusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I5LCFXQV}},
  note         = {Machine review of arXiv:2509.01025}
}
Share X Bluesky LinkedIn Reddit HN
abstract

Masked diffusion models (MDMs) have recently emerged as a promising alternative to autoregressive models over discrete domains. MDMs generate sequences in an any-order, parallel fashion, enabling fast inference and strong performance on non-causal tasks. However, a crucial limitation is that they do not support token insertions and are thus limited to fixed-length generations. To this end, we introduce Flexible Masked Diffusion Models (FlexMDMs), a discrete diffusion paradigm that simultaneously can model sequences of flexible length while provably retaining MDMs' flexibility of any-order inference. Grounded in an extension of the stochastic interpolant framework, FlexMDMs generate sequences by inserting mask tokens and unmasking them. Empirically, we show that FlexMDMs match MDMs in perplexity while modeling length statistics with much higher fidelity. On a synthetic maze planning task, they achieve $\approx 60 \%$ higher success rate than MDM baselines. Finally, we show pretrained MDMs can easily be retrofitted into FlexMDMs: on 16 H100s, it takes only three days to fine-tune LLaDA-8B into a FlexMDM, achieving superior performance on math (GSM8K, $58\% \to 67\%$) and code infilling performance ($52\% \to 65\%$).

Figures

Figures reproduced from arXiv: 2509.01025 by Carles Domingo-Enrich, Jaeyeon Kim, Lee Cheuk-Kit, Michael Albergo, Sham Kakade, Sitan Chen, Timothy Ngotiaoco, Yilun Du.

Figure 1
Figure 1. Figure 1: Flexible Masked Diffusion Model (FlexMDM) addresses MDMs’ inability to handle variable-length sequences and token insertion while preserving any-order generation power. At each step, FlexMDM performs insertion and unmasking by predicting the expected number of mask tokens to insert (gθ) and the posterior over clean tokens (fθ), respectively. • On planning tasks, FlexMDM achieves markedly better results, be… view at source ↗
Figure 2
Figure 2. Figure 2: To draw a sample xt, one can equivalently sample the clean sequence x1 ∼ p1, draw unmasking times, and then accord￾ingly unmask or mask each coordinate’s to￾ken. MDM training. We now derive the MDM rate matrix that induces a CTMC whose marginals coincide with {pt}t∈[0,1] and how it is learned in practice. The central object is the unmasking pos￾terior: the posterior on the clean token x i 1 for masked inde… view at source ↗
Figure 3
Figure 3. Figure 3: Left (FlexMDM interpolant). To draw a sample xt, one can equivalently draw a sample x1 ∼ p1, and for each token unmask, mask, or remove it according to the unmasking and insertion times (T i 1 , Ti 2 ). An auxiliary interpolant st gives closed-form expressions for the FlexMDM rate matrices. Right (FlexMDM Inference). Learned unmasking posterior and insertion expectation are later used at inference. for t ∈… view at source ↗
Figure 5
Figure 5. Figure 5: FlexMDM performance exhibits superior scaling when more sampling steps are allocated. 6 CONCLUSION In this work we proposed Flexible Masked Diffusion Models (FlexMDM), a discrete diffusion framework over variable-length sequences. Theoretically, via a joint interpolant viewpoint, we pro￾vide rigorous guarantees for both training and inference of FlexMDM. Empirically, FlexMDM learns 3 For a fair comparison,… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding

    cs.LG 2026-07 unverdicted novelty 7.0

    Set diffusion factorizes likelihood over arbitrary token sets and uses a set-causal diffusion architecture to support KV caching and any-order decoding, yielding improved speed-quality tradeoffs versus prior diffusion LMs.

  2. Discrete Tilt Matching

    cs.LG 2026-04 unverdicted novelty 7.0

    DTM recasts dLLM fine-tuning as weighted cross-entropy matching of tilted local posteriors, with demonstrated gains on Sudoku and math tasks.

  3. Discrete Tilt Matching

    cs.LG 2026-04 unverdicted novelty 7.0

    Discrete Tilt Matching recasts dLLM fine-tuning as state-level matching of tilted local unmasking posteriors, producing a stable weighted cross-entropy loss that improves Sudoku and Countdown performance when applied ...

  4. Generative Modeling from Black-box Corruptions via Self-Consistent Stochastic Interpolants

    cs.LG 2025-12 unverdicted novelty 7.0

    SCSI iteratively refines a self-consistent transport map to invert black-box corruptions and enable generative modeling of clean data.

  5. CreditDecoding: Accelerating Parallel Decoding in Diffusion Large Language Models with Trace Credit

    cs.CL 2025-10 unverdicted novelty 7.0

    CreditDecoding accelerates parallel decoding in diffusion LLMs by fusing accumulated Trace Credit with current logits to accept early-correct tokens sooner, yielding up to 5.48x speedup and accuracy gains.

  6. From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models

    cs.LG 2026-07 conditional novelty 6.0

    Masked diffusion language models fail to exploit their any-order interface because of positional uncertainty; insertion-based (FlexMDM) and latent-segment (LatentMDM) variants recover distinct any-order inference beha...

  7. Hierarchical Domain Generalization

    cs.LG 2026-07 conditional novelty 6.0

    Over infinite domains, hierarchy-uniform domain generalization is impossible for every nontrivial hypothesis class; a length-generalization bound is a property of the length hierarchy, not a hierarchy-free guarantee.

  8. Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation

    cs.AI 2026-07 conditional novelty 6.0

    A diffusion multimodal LLM's early-layer MLP sparsity at denoising step 0 pinpoints the semantic boundary, enabling one-shot suffix truncation that raises throughput up to ~31x.

  9. VoidPadding: Let [VOID] Handle Padding in Masked Diffusion Language Models so that [EOS] Can Focus on Semantic Termination

    cs.CL 2026-06 unverdicted novelty 6.0

    VoidPadding decouples padding from termination in MDLMs via a new [VOID] token, delivering +17.84 average benchmark points and 55.7% fewer decoding steps on Dream-7B-Instruct.

  10. Generative Molecular Morphing for Flexible-Size Design via Unbalanced Optimal Transport

    cs.LG 2026-06 unverdicted novelty 6.0

    Morph is a flexible-size 3D molecular generative model using unbalanced optimal transport on geometric graphs that matches fixed-size SOTA performance while enabling out-of-distribution generation.

  11. Fast-dLLM++: Fr\'{e}chet Profile Decoding for Faster Diffusion LLM Inference

    cs.CL 2026-06 conditional novelty 6.0

    Fast-dLLM++ generalizes Fast-dLLM decoding to heterogeneous confidence profiles via Fréchet profile selection, delivering up to 37% throughput gains on GSM8K, MATH, HumanEval, and MBPP with LLaDA-8B.

  12. Edit-Based Refinement for Parallel Masked Diffusion Language Models

    cs.CL 2026-05 unverdicted novelty 6.0

    ME-DLM augments parallel masked diffusion models with edit-distance-supervised refinements to raise quality on coding and math benchmarks while using far fewer diffusion steps.

  13. Gumbel Distillation for Parallel Text Generation

    cs.CL 2026-03 conditional novelty 6.0

    Conditioning parallel decoders on Gumbel noise sampled from an autoregressive teacher's Gumbel-Max process improves generation quality on LM1B and OpenWebText.

  14. STRIDE: Post-Training LLMs to Reason and Refine Bio-Sequences via Edit Trajectories

    cs.CE 2026-03 conditional novelty 6.0

    Training LLMs to emit executable edit trajectories (INSERT/DELETE/REPLACE) from Levenshtein alignments plus policy optimization improves oracle-scored bio-sequence optimization success and novelty.

  15. Reconsidering Positional Supervision in Masked Diffusion Language Model Training

    cs.CL 2026-01 conditional novelty 6.0

    Adding a CTC-style <SLACK> token during MDLM supervised fine-tuning relaxes strict positional supervision and improves open-ended generation benchmarks over a matched cross-entropy baseline.

  16. A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models

    cs.CV 2025-11 conditional novelty 6.0

    Visual token pruning causes severe loss in discrete diffusion MLLMs; only from-scratch models on long-answer tasks recover via late denoising, so redundancy is recoverability, not dispensability.

  17. CreditDecoding: Accelerating Parallel Decoding in Diffusion Large Language Models with Trace Credit

    cs.CL 2025-10 conditional novelty 5.0

    A token-history 'credit' score fused into current logits reduces redundant re-masking in diffusion LLM decoding, yielding up to 5.48× tokens-per-forward speedup with small accuracy gains.

Reference graph

Works this paper leans on

71 extracted references · 14 canonical work pages · cited by 15 Pith papers · 1 internal anchor

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 0 6840--6851, 2020

  3. [3]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020

  4. [4]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256--2265. pmlr, 2015

  5. [5]

    Simplified and generalized masked diffusion for discrete data

    Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37: 0 103131--103167, 2024

  6. [6]

    Simple and effective masked diffusion language models

    Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37: 0 130136--130184, 2024

  7. [7]

    Discrete flow matching

    Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. Advances in Neural Information Processing Systems, 37: 0 133345--133385, 2024

  8. [8]

    Beyond autoregression: Discrete diffusion for complex reasoning and planning

    Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning. arXiv preprint arXiv:2410.14157, 2024

  9. [9]

    Large language diffusion models

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025

  10. [10]

    Dream 7b, 2025

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025. URL https://hkunlp.github.io/blog/2025/dream

  11. [11]

    Scaling up masked diffusion models on text

    Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.18514, 2024

  12. [12]

    Albergo and Eric Vanden-Eijnden

    Michael S. Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants, 2022

  13. [13]

    Stochastic interpolants: A unifying framework for flows and diffusions

    Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797, 2023 a

  14. [14]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2022

  15. [15]

    Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling

    Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409.02908, 2024

  16. [16]

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data

    Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.03736, 2024

  17. [17]

    Dreamon: Diffusion language models for code infilling beyond fixed-size canvas, 2025 a

    Zirui Wu, Lin Zheng, Zhihui Xie, Jiacheng Ye, Jiahui Gao, Yansong Feng, Zhenguo Li, Victoria W., Guorui Zhou, and Lingpeng Kong. Dreamon: Diffusion language models for code infilling beyond fixed-size canvas, 2025 a . URL https://hkunlp.github.io/blog/2025/dreamon

  18. [18]

    Edit flows: Flow matching with edit operations

    Marton Havasi, Brian Karrer, Itai Gat, and Ricky TQ Chen. Edit flows: Flow matching with edit operations. arXiv preprint arXiv:2506.09018, 2025

  19. [19]

    A continuous time framework for discrete denoising models

    Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35: 0 28266--28279, 2022

  20. [20]

    Train for the worst, plan for the best: Understanding token ordering in masked diffusions

    Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arXiv preprint arXiv:2502.06768, 2025

  21. [21]

    Albergo, Nicholas M

    Michael S. Albergo, Nicholas M. Boffi, Michael Lindsey, and Eric Vanden-Eijnden. Multimarginal generative modeling with stochastic interpolants, 2023 b . URL https://arxiv.org/abs/2310.03695

  22. [22]

    Albergo, and Eric Vanden-Eijnden

    Hugo Negrel, Florentin Coeurdoux, Michael S. Albergo, and Eric Vanden-Eijnden. Multitask learning with stochastic interpolants, 2025. URL https://arxiv.org/abs/2508.04605

  23. [23]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195--4205, 2023

  24. [24]

    Scaling diffusion language models via adaptation from autoregressive models

    Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410.17891, 2024

  25. [25]

    Openwebtext corpus

    Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019

  26. [26]

    Planning with diffusion for flexible behavior synthesis

    Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, 2022

  27. [27]

    Stochastic interpolants with data-dependent couplings

    Michael Samuel Albergo, Mark Goldstein, Nicholas Matthew Boffi, Rajesh Ranganath, and Eric Vanden-Eijnden. Stochastic interpolants with data-dependent couplings. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=FFILRGD0jG

  28. [28]

    Jiang, Jia Deng, Stella Biderman, and Sean Welleck

    Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2023

  29. [29]

    d1: Scaling reasoning in diffusion large language models via reinforcement learning

    Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. d1: Scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint arXiv:2504.12216, 2025

  30. [30]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

  31. [31]

    Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, J. H. Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, and Wei Chu. Opencoder: The open cookbook for top-tier code large language models. 2024. URL https://arxiv.org/pdf/2411.04905

  32. [32]

    Efficient training of language models to fill in the middle

    Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255, 2022

  33. [33]

    dkv-cache: The cache for diffusion language models

    Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models. arXiv preprint arXiv:2505.15781, 2025 a

  34. [34]

    Esoteric language models

    Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, and Arash Vahdat. Esoteric language models. arXiv preprint arXiv:2506.01928, 2025

  35. [35]

    Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding

    Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.22618, 2025 b

  36. [36]

    Argmax flows and multinomial diffusion: Learning categorical distributions

    Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forr \'e , and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in neural information processing systems, 34: 0 12454--12465, 2021

  37. [37]

    Structured denoising diffusion models in discrete state-spaces

    Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34: 0 17981--17993, 2021

  38. [38]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834, 2023

  39. [39]

    Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design

    Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint arXiv:2402.04997, 2024

  40. [40]

    Flow matching with general discrete paths: A kinetic-optimal perspective

    Neta Shaul, Itai Gat, Marton Havasi, Daniel Severo, Anuroop Sriram, Peter Holderrieth, Brian Karrer, Yaron Lipman, and Ricky TQ Chen. Flow matching with general discrete paths: A kinetic-optimal perspective. arXiv preprint arXiv:2412.03487, 2024

  41. [41]

    Path planning for masked diffusion model sampling

    Fred Zhangzhi Peng, Zachary Bezemek, Sawan Patel, Jarrid Rector-Brooks, Sherwood Yao, Avishek Joey Bose, Alexander Tong, and Pranam Chatterjee. Path planning for masked diffusion model sampling. arXiv preprint arXiv:2502.03540, 2025

  42. [42]

    Anchored diffusion language model

    Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. Anchored diffusion language model. arXiv preprint arXiv:2505.18456, 2025

  43. [43]

    Diffucoder: Understanding and improving masked diffusion models for code generation

    Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. Diffucoder: Understanding and improving masked diffusion models for code generation. arXiv preprint arXiv:2506.20639, 2025

  44. [44]

    Seed diffusion: A large-scale diffusion language model with high-speed inference

    Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, et al. Seed diffusion: A large-scale diffusion language model with high-speed inference. arXiv preprint arXiv:2508.02193, 2025

  45. [45]

    Mercury: Ultra-fast language models based on diffusion

    Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, et al. Mercury: Ultra-fast language models based on diffusion. arXiv preprint arXiv:2506.17298, 2025

  46. [46]

    Accelerated sampling from masked diffusion models via entropy bounded unmasking

    Heli Ben-Hamu, Itai Gat, Daniel Severo, Niklas Nolte, and Brian Karrer. Accelerated sampling from masked diffusion models via entropy bounded unmasking. arXiv preprint arXiv:2505.24857, 2025

  47. [47]

    Unified multimodal discrete diffusion

    Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, and Katerina Fragkiadaki. Unified multimodal discrete diffusion. arXiv preprint arXiv:2503.20853, 2025

  48. [48]

    Gemini diffusion, 2025

    Google DeepMind. Gemini diffusion, 2025. URL https://blog.google/technology/google-deepmind/gemini-diffusion/

  49. [49]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11315--11325, 2022

  50. [50]

    Reinforced Context Order Recovery for Adaptive Reasoning and Planning

    Long Ma, Fangwei Zhong, and Yizhou Wang. Reinforced context order recovery for adaptive reasoning and planning. arXiv preprint arXiv:2508.13070, 2025 b

  51. [51]

    Learning-order autoregressive models with application to molecular graph generation

    Zhe Wang, Jiaxin Shi, Nicolas Heess, Arthur Gretton, and Michalis K Titsias. Learning-order autoregressive models with application to molecular graph generation. arXiv preprint arXiv:2503.05979, 2025

  52. [52]

    Rewon child, david luan, dario amodei, and ilya sutskever

    Alec Radford and Jeffrey Wu. Rewon child, david luan, dario amodei, and ilya sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

  53. [53]

    Flex attention: A programming model for generating optimized attention kernels

    Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A programming model for generating optimized attention kernels. arXiv preprint arXiv:2412.05496, 2024

  54. [54]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35: 0 16344--16359, 2022

  55. [55]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Harts...

  56. [56]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

  57. [57]

    Deepseek llm: Scaling open-source language models with longtermism

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024

  58. [58]

    Qwen2 technical report

    Qwen. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

  59. [59]

    Lo RA : Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

  60. [60]

    Flow straight and fast: Learning to generate and transfer data with rectified flow, 2022

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow, 2022. URL https://arxiv.org/abs/2209.03003

  61. [61]

    Peter Holderrieth, Marton Havasi, Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky T. Q. Chen, and Yaron Lipman. Generator matching: Generative modeling with arbitrary markov processes. In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openreview.net/forum?id=RuP17cJtZo

  62. [62]

    From denoising diffusions to denoising markov models, 2024

    Joe Benton, Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. From denoising diffusions to denoising markov models, 2024. URL https://arxiv.org/abs/2211.03595

  63. [63]

    Trans-dimensional generative modeling via jump diffusion models, 2023

    Andrew Campbell, William Harvey, Christian Weilbach, Valentin De Bortoli, Tom Rainforth, and Arnaud Doucet. Trans-dimensional generative modeling via jump diffusion models, 2023. URL https://arxiv.org/abs/2305.16261

  64. [64]

    Transport meets variational inference: Controlled monte carlo diffusions, 2025

    Francisco Vargas, Shreyas Padhy, Denis Blessing, and Nikolas Nüsken. Transport meets variational inference: Controlled monte carlo diffusions, 2025. URL https://arxiv.org/abs/2307.01050

  65. [65]

    An optimal control perspective on diffusion-based generative modeling

    Julius Berner, Lorenz Richter, and Karen Ullrich. An optimal control perspective on diffusion-based generative modeling. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=oYIjw37pTP

  66. [66]

    Non-denoising forward-time diffusions, 2022

    Stefano Peluchetti. Non-denoising forward-time diffusions, 2022. URL https://openreview.net/forum?id=oVfIKuhqfC

  67. [67]

    LEAPS : A discrete neural sampler via locally equivariant networks

    Peter Holderrieth, Michael Samuel Albergo, and Tommi Jaakkola. LEAPS : A discrete neural sampler via locally equivariant networks. In Forty-second International Conference on Machine Learning, 2025 b . URL https://openreview.net/forum?id=Hq2RniQAET

  68. [68]

    Neta Shaul, Itai Gat, Marton Havasi, Daniel Severo, Anuroop Sriram, Peter Holderrieth, Brian Karrer, Yaron Lipman, and Ricky T. Q. Chen. Flow matching with general discrete paths: A kinetic-optimal perspective. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=tcvMzR2NrP

  69. [69]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  70. [70]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  71. [71]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.