Unifying Masked Diffusion Models with Various Generation Orders and Beyond

Chunsan Hong; Jong Chul Ye; Sanghyun Lee

arxiv: 2602.02112 · v2 · pith:A4PZTFTGnew · submitted 2026-02-02 · 💻 cs.LG · cs.AI· cs.CL

Unifying Masked Diffusion Models with Various Generation Orders and Beyond

Chunsan Hong , Sanghyun Lee , Jong Chul Ye This is my paper

Pith reviewed 2026-05-22 11:49 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CL

keywords masked diffusion modelsgeneration orderinglanguage modelingdiscrete diffusionjoint optimizationcontext-dependent order

0 comments

The pith

LoMDM jointly learns generation ordering and the diffusion backbone from scratch in masked diffusion models.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces an order-expressive masked diffusion model that unifies masked diffusion, autoregressive, and block diffusion processes under one framework covering arbitrary generation orders. Building on this, it presents learnable-order masked diffusion model which optimizes both the ordering policy and the diffusion parameters together through a single objective instead of separate stages. This joint training lets the model discover orders that depend on the surrounding context during text generation. The approach yields better results than prior discrete diffusion methods on several language modeling benchmarks.

Core claim

Order-expressive masked diffusion model (OeMDM) interprets a broad class of diffusion generative processes with various generation orders in a single framework. Learnable-order masked diffusion model (LoMDM) extends this by jointly learning the generation ordering and diffusion backbone through one objective from scratch, enabling context-dependent ordering without two-stage optimization.

What carries the argument

Learnable-order masked diffusion model (LoMDM) that discovers context-dependent generation orders jointly with the diffusion backbone via a single optimization objective.

If this is right

Unifies interpretation of masked diffusion models, autoregressive models, and block diffusion under one framework.
Avoids suboptimal results that come from learning an ordering policy after the diffusion backbone is already fixed.
Produces generation orders that adapt based on the specific text context rather than using a fixed schedule.
Delivers stronger empirical performance than existing discrete diffusion models on multiple language modeling benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same joint-ordering idea could be tested in diffusion models for images or audio where sequence order affects quality.
Eliminating the two-stage pipeline may shorten overall training time for order-sensitive generative tasks.
It remains open whether the learned context-dependent orders produce more coherent long-form text than rigid left-to-right schedules.

Load-bearing premise

A single joint optimization objective can simultaneously discover effective context-dependent generation orders and train a high-quality diffusion backbone without the suboptimal solutions that arise in two-stage training pipelines.

What would settle it

An experiment in which a two-stage approach with a separately learned ordering policy achieves equal or higher performance than LoMDM on language modeling benchmarks, or in which LoMDM's learned orders show no measurable dependence on context.

Figures

Figures reproduced from arXiv: 2602.02112 by Chunsan Hong, Jong Chul Ye, Sanghyun Lee.

**Figure 1.** Figure 1: Conceptual illustration of learnable-order masked diffusion model (LoMDM) and other language models. Black text denotes already generated tokens, while the colored tokens indicate the generation candidates, with a lower color represents low generation order priority. In training time, LoMDM jointly learns what to generate and where to generate next, and in inferencetime, LoMDM selects where to unmask ne… view at source ↗

**Figure 2.** Figure 2: Illustration of αarm,ϵ(t) that makes OeMDM to generate in L2R order. The explicit function formulation is in Appendix D.1 continuous time is given as follows: − log pθ,αˆF (x) ≤ LOeMDM(x, θ, αF , αˆF ) = Z 1 0 EqαF "X L i=1 ⟨z (i) t , m⟩ ( −A (i) log⟨x (i) θ (zt, t), x (i) ⟩ | {z } Lmain + A (i) (log A (i) − log Aˆ(i) ) − (A (i) − Aˆ(i) ) | {z } Lvelocity )#dt, (2) where the structure of Lmain is equal to … view at source ↗

**Figure 3.** Figure 3: Model structure of LoMDM. We view backbone of diffusion model θ as a feature extractor of zt or x, and train θ,αϕ, and αˆψ jointly. Depending on the input type, final layers are switched off or on. For example in the above figure, the input is zt so the final diffusion MLP layer and αˆψ is activated. Meanwhile, if input was x, only αϕ would be activated. We detach the gradient of αϕ and αˆψ from flowing to… view at source ↗

**Figure 4.** Figure 4: Pearson correlation per training step for LoMDM trained on OWT. We report correlations among A (i) ϕ (x, t), Aˆ (i) ψ (zt, t), and ⟨x (i) θ (zt, t), x (i) ⟩. When measuring correlation with ⟨x (i) θ (zt, t), x (i) ⟩, we compute it only over masked positions in zt, since x (i) θ (zt, t) is zero at unmasked positions. of Penn Tree Bank (PTB; Marcus et al. (1993)), WikiText (Merity et al., 2017), LM1B, Lambad… view at source ↗

**Figure 5.** Figure 5: Test PPL per wall-clock-time during training on OWT. We truncate the curves at the point where LoMDM matches the 1M-step MDLM performance (PPL = 23.0). At this cutoff, MDLM had reached PPL = 24.9 with ∼0.30M steps, while our method had reached PPL = 23.0 with ∼0.18M steps. across the shared wall-clock budget in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

read the original abstract

Masked diffusion models (MDMs) are a potential alternative to autoregressive models (ARMs) for language generation, but generation quality depends critically on the generation order. Prior work either hard-codes an ordering (e.g., blockwise left-to-right) or learns an ordering policy for a pretrained MDM, which incurs extra cost and can yield suboptimal solutions due to the two-stage optimization. Motivated by this, we propose order-expressive masked diffusion model (OeMDM) for a broad class of diffusion generative processes with various generation orders, enabling the interpretation of MDM, ARM, and block diffusion in a single framework. Furthermore, building on OeMDM, we introduce learnable-order masked diffusion model (LoMDM), which jointly learns the generation ordering and diffusion backbone through a single objective from scratch, enabling the diffusion model to generate text in context-dependent ordering. Empirically, we confirm that LoMDM outperforms various discrete diffusion models across multiple language modeling benchmarks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper gives a clean unification of masked diffusion under different orders and tries joint single-objective training for the order policy, but the joint optimization looks vulnerable to early collapse and the empirical support is thin on details.

read the letter

The core contribution is OeMDM, which frames MDM, ARM, and block diffusion as instances of the same order-expressive process. That framing is useful because it makes the role of generation order explicit instead of treating it as a fixed hyperparameter. LoMDM then adds a learnable ordering component trained end-to-end with the diffusion backbone under one loss, avoiding the two-stage pipeline that prior work used. This is the part that could matter for people trying to move beyond left-to-right or blockwise schedules in discrete diffusion for text. The abstract reports better benchmark numbers than existing discrete diffusion models, which is the main empirical claim. The unification itself does not appear to introduce new parameters or change the core diffusion math, so the lift comes from the joint training and the broader order class. The obvious risk is exactly the one in the stress-test note: once the backbone starts to improve, gradients on the ordering variables can vanish or the policy can lock into a near-constant order, turning LoMDM into a standard MDM plus overhead. Nothing in the abstract shows an explicit regularizer or curriculum that would prevent that mode of failure, and the lack of reported ablations on order diversity during training leaves the context-dependent claim unverified. The paper is written for researchers already working on non-autoregressive or diffusion-based language models who want a more flexible order mechanism. It is not a foundational rethinking of diffusion itself, but the joint-training angle is a reasonable incremental idea. I would send it to review because the unification is technically coherent and the empirical direction is worth checking, even if the current evidence is preliminary and the collapse concern needs direct experiments.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes OeMDM, a unifying framework for masked diffusion models that accommodates arbitrary generation orders and recovers MDM, ARM, and block diffusion as special cases. Building on OeMDM, it introduces LoMDM, which jointly optimizes a learnable ordering policy and the diffusion backbone parameters from scratch under a single objective, with the goal of enabling context-dependent generation orders. The authors report that LoMDM outperforms prior discrete diffusion baselines on several language modeling benchmarks.

Significance. If the joint optimization reliably discovers non-collapsing, context-dependent orders while training a competitive diffusion backbone, the result would meaningfully advance non-autoregressive text generation by removing the need for two-stage pipelines and their associated suboptimality. The unification via OeMDM also provides a clean theoretical lens for comparing order choices.

major comments (2)

[§4.2] §4.2, Eq. (8)–(10): the joint objective is presented as simultaneously optimizing ordering logits and diffusion parameters, yet no analysis is given of the gradient flow on the ordering variables once the backbone begins to improve. A concrete demonstration that the ordering entropy remains high (or that orders vary meaningfully with context) is required to rule out premature collapse to near-fixed orders.
[Table 2] Table 2, LoMDM rows: the reported perplexity gains are modest (roughly 1–3 points) and the paper does not include an ablation that isolates the contribution of the learned ordering versus simply using a stronger backbone or longer training. Without this, it is difficult to attribute the improvement to the joint-order mechanism rather than other implementation details.

minor comments (2)

[Figure 3] Figure 3 caption: the legend for the ordering heatmaps is too small to read; enlarge or add a separate colorbar.
[§3.1] §3.1: the notation for the masking schedule mixes t and k without an explicit mapping; a short clarifying sentence would help readers.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We are grateful to the referee for the constructive and insightful comments. We address each major comment below and describe the revisions we plan to make.

read point-by-point responses

Referee: [§4.2] §4.2, Eq. (8)–(10): the joint objective is presented as simultaneously optimizing ordering logits and diffusion parameters, yet no analysis is given of the gradient flow on the ordering variables once the backbone begins to improve. A concrete demonstration that the ordering entropy remains high (or that orders vary meaningfully with context) is required to rule out premature collapse to near-fixed orders.

Authors: We agree that an analysis of the gradient flow through the ordering logits and empirical verification against collapse are important for substantiating the joint-optimization claim. The current manuscript does not contain such an analysis. In the revision we will add both a brief theoretical discussion of the gradient dynamics on the ordering variables and new empirical results, including plots of ordering entropy throughout training together with qualitative examples of context-dependent orders. revision: yes
Referee: Table 2, LoMDM rows: the reported perplexity gains are modest (roughly 1–3 points) and the paper does not include an ablation that isolates the contribution of the learned ordering versus simply using a stronger backbone or longer training. Without this, it is difficult to attribute the improvement to the joint-order mechanism rather than other implementation details.

Authors: We acknowledge that the reported gains are modest and that stronger evidence is needed to attribute them specifically to the learned ordering rather than other factors. While the main experiments compare LoMDM against prior discrete diffusion models under comparable settings, the manuscript does not contain the requested ablation. In the revised version we will add a controlled ablation that trains a fixed-order MDM using the identical backbone architecture, training schedule, and compute budget for direct comparison with LoMDM. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation chain is self-contained

full rationale

The paper proposes OeMDM as a unifying framework for masked diffusion models with variable generation orders and LoMDM as an extension that jointly optimizes ordering and the diffusion backbone via a single objective. No equations, derivations, or load-bearing steps in the abstract or described claims reduce the performance gains or context-dependent ordering to a fitted parameter by construction, a self-citation chain, or an ansatz smuggled from prior author work. The central premise is presented as building on but distinct from two-stage pipelines, with empirical results on language benchmarks serving as external validation rather than tautological re-expression of inputs. This is the expected outcome for a methodological proposal whose novelty lies in the joint training architecture itself.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 2 invented entities

Review performed on abstract only; full manuscript may contain additional fitted parameters or modeling assumptions not visible here. The ledger therefore records only the high-level constructs explicitly named in the abstract.

axioms (1)

domain assumption Masked diffusion processes can be expressed as a broad class that includes various generation orders
This assumption underpins the claim that OeMDM interprets MDM, ARM, and block diffusion in one framework.

invented entities (2)

OeMDM no independent evidence
purpose: Provide a unified generative process for masked diffusion with arbitrary generation orders
New model class introduced to enable the unification.
LoMDM no independent evidence
purpose: Jointly optimize generation order and diffusion parameters from scratch under one objective
New model built on OeMDM to remove the two-stage training limitation.

pith-pipeline@v0.9.0 · 5701 in / 1388 out tokens · 64983 ms · 2026-05-22T11:49:27.895441+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We introduce order-expressive masked diffusion model (OeMDM) ... generalized NELBO ... velocity A(u,t) := −∂t αF[I](u,t) ⊘ (1−αF[I](u,t)) ... Lvelocity quantifies the gap between the unmasking order and the forward noise process.
IndisputableMonolith/Foundation/AlphaCoordinateFixation.lean alpha_pin_under_high_calibration unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

LoMDM jointly learns the generation ordering and diffusion backbone through a single objective from scratch

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Understanding and Accelerating the Training of Masked Diffusion Language Models
cs.LG 2026-05 conditional novelty 6.0

Bell-shaped time sampling accelerates masked diffusion language model training by roughly 4x on LM1B by countering locality bias in language data.

Reference graph

Works this paper leans on

19 extracted references · 19 canonical work pages · cited by 1 Pith paper · 2 internal anchors

[1]

S., Bui, T., Kim, S., Chang, W., and Goharian, N

Cohan, A., Dernoncourt, F., Kim, D. S., Bui, T., Kim, S., Chang, W., and Goharian, N. A discourse-aware attention model for abstractive summarization of long documents. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 615–621,

work page 2018
[2]

In: Zong, C., Xia, F., Li, W., Navigli, R

doi: 10.18653/v1/ N18-2097. URL https://aclanthology.org/ N18-2097/. Gokaslan, A., Cohen, V ., Pavlick, E., and Tellex, S. Open- webtext corpus. http://Skylion007.github. io/OpenWebTextCorpus,

work page doi:10.18653/v1/ 2097
[3]

The Llama 3 Herd of Models

URL https://arxiv.org/abs/2407.21783. He, Z., Sun, T., Tang, Q., Wang, K., Huang, X.-J., and Qiu, X. Diffusionbert: Improving generative masked language models with diffusion models. InProceedings of the 61st annual meeting of the association for computational linguistics (ACL), volume 1, pp. 4521–4534,

work page internal anchor Pith review Pith/arXiv arXiv
[4]

Hoogeboom, E., Nielsen, D., Jaini, P., Forr´e, P., and Welling, M

URL https://arxiv.org/ abs/2510.05725. Hoogeboom, E., Nielsen, D., Jaini, P., Forr´e, P., and Welling, M. Argmax flows and multinomial diffusion: Learning categorical distributions. InAdvances in Neural Infor- mation Processing Systems (NeurIPS), volume 34, pp. 12454–12465,

work page arXiv
[5]

Z., Bezemek, Z., Patel, S., Rector-Brooks, J., Yao, S., Bose, A

Peng, F. Z., Bezemek, Z., Patel, S., Rector-Brooks, J., Yao, S., Bose, A. J., Tong, A., and Chatterjee, P. Path plan- ning for masked diffusion model sampling, 2025a. URL https://arxiv.org/abs/2502.03540. Peng, F. Z., Bezemek, Z., Rector-Brooks, J., Zhang, S., Zhang, A. R., Bronstein, M., Bose, A. J., and Tong, A. Planner aware path learning in diffusion ...

work page arXiv
[6]

Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling

URLhttps://arxiv.org/abs/2505.17384. Ye, J., Xie, Z., Zheng, L., Gao, J., Wu, Z., Jiang, X., Li, Z., and Kong, L. Dream 7b: Diffusion large language mod- els,

work page internal anchor Pith review Pith/arXiv arXiv
[7]

11 Unifying Masked Diffusion Models with Various Generation Orders and Beyond A. Related works Discrete diffusion models.Diffusion probabilistic models have become a dominant approach for continuous domains such as images, audio, and video (Ho et al., 2020; Song et al., 2021). This success has motivated extensions to discrete domains, including text, lead...

work page 2020
[8]

Similar phenomena appear in large-scale time-agnostic MDMs: when using random ordering, large-scale MDMs (Ye et al., 2025; Nie et al.,

injects a block-wise left-to-right bias and substantially improves over MDLM. Similar phenomena appear in large-scale time-agnostic MDMs: when using random ordering, large-scale MDMs (Ye et al., 2025; Nie et al.,

work page 2025
[9]

often lag behind large-scale ARMs such as LLaMA-3 (Grattafiori et al., 2024). However, adopting structured decoding—e.g., block-wise L2R generation, within each block, revealing the highest-confidence positions first—can close this gap and even surpass ARMs in some settings. Overall, these results highlight that the choice of unmasking order matters in MD...

work page 2024
[10]

improve the order only via post-training (Hong et al., 2025; Peng et al., 2025a),

work page 2025
[11]

To address these limitations, we return to the canonical continuous-time MDLM formulation and focus on the scheduler as the central mechanism that governs the unmasking process

fail to fully capture context-dependent ordering over the entire sequence (Shi et al., 2024). To address these limitations, we return to the canonical continuous-time MDLM formulation and focus on the scheduler as the central mechanism that governs the unmasking process. This perspective enables a principled integration of MDM training with learnable, con...

work page 2024
[12]

Following conventional MDMs (Sahoo et al., 2024a; Shi et al., 2024), we take the terminal forward distribution to be fully masked: qαF (z(i) t(T) | x) = Cat(m) for all i∈[L]

such that s(τ) =t(τ−1) . Following conventional MDMs (Sahoo et al., 2024a; Shi et al., 2024), we take the terminal forward distribution to be fully masked: qαF (z(i) t(T) | x) = Cat(m) for all i∈[L] . Likewise, the model’s initial noise distribution is fully masked:pθ,ˆαF (z(i) t(T) ) = Cat(m) for alli∈[L]. Finally, the reconstruction distribution att(0)i...

work page 2024
[13]

αmdlm(t) 1−α mdlm(t) logp θ,bd3lm(xb |z b t ,x <b) # (80) = BX b=1 Et∼Unif[0,1] Ezb t ∼qαmdlm(·|xb)

Summing overi= 1, . . . , Lproves LOeMDM(x, θ, αarm,ε, αarm,ε) = LX i=1 −log⟨x (i) θ (yi),x (i)⟩+O(ε) =−log LY i=1 ⟨x(i) θ (yi),x (i)⟩+O(ε), which completes the proof. 28 Unifying Masked Diffusion Models with Various Generation Orders and Beyond Furthermore, we can extend the above theoretical results for the auto-regressive modeling of any fixed order: C...

work page 2025
[14]

dog”] > αGen(t)[“cat

defines vocabulary-wise scheduler αGen :T →[0,1] V+1 where each dimension represents the different amount of noise for corresponding word. In this regard, forward process adds different amount of noise to each position depending on whichwordit owes,e.g., if αGen(t)[“dog”] > αGen(t)[“cat”]3, scheduler noise “cat” more than “dog”. To investigate further, we...

work page 2024
[15]

Training Stabilization Techniques E.1

E. Training Stabilization Techniques E.1. REINFORCE Leave-One-Out In Section 4.3, we have stated that Eqαϕ [∇ϕ logq αϕ ·(L main +L velocity)] in ∇ϕLLoMDM =E qαϕ [∇ϕ logq αϕ ·(L main + Lvelocity)] +E qαϕ [∇ϕ(Lmain +L velocity)] is a high-variance estimator. Therefore, we use a low-variance estimator of such optimization problems proposed by Kool et al. (20...

work page 2019
[16]

However, the image and text domains differ substantially, making it nontrivial to directly transfer such techniques

Unlike discrete diffusion, which has only recently begun to be explored, the continuous domain (e.g., image diffusion) has a longer history of studyinglearnable noise schedulers. However, the image and text domains differ substantially, making it nontrivial to directly transfer such techniques. We therefore first briefly review how learnable schedulers ar...

work page 2021
[17]

While continuous diffusion often allows highly flexible parameterizations of α, in text space we instead require a suitably regularized and simple functional form

Parametrizion of LoMDM.From the continuous-diffusion case above, we draw one key lesson: the velocityA should admit asimple formthat enables direct optimization. While continuous diffusion often allows highly flexible parameterizations of α, in text space we instead require a suitably regularized and simple functional form. Motivated by the simplest sched...

work page 2025
[18]

Experimental Details G.1

G. Experimental Details G.1. Experimental Settings We detokenize the One Billion Words dataset following Lou et al. (2024); Sahoo et al. (2024a), whose official code can be found. We tokenize LM1B using theBERT-BASE-UNCASEDtokenizer, consistent with He et al. (2023). We then concatenate and pack the sequences to a fixed length of 128 (Raffel et al., 2020)...

work page 2024
[19]

(2024); Sahoo et al

from Lou et al. (2024); Sahoo et al. (2024a). We use 12 layers, a hidden dimension of 768, 12 attention heads, and for LoMDM, one more same size of transformer layer for each ϕ, ψ. For diffusion backbone, we use the AdamW optimizer with a batch size of 512, constant learning rate warmup from 0 to a learning rate of 3e-4 for 2,500 steps following prior wor...

work page 2024

[1] [1]

S., Bui, T., Kim, S., Chang, W., and Goharian, N

Cohan, A., Dernoncourt, F., Kim, D. S., Bui, T., Kim, S., Chang, W., and Goharian, N. A discourse-aware attention model for abstractive summarization of long documents. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 615–621,

work page 2018

[2] [2]

In: Zong, C., Xia, F., Li, W., Navigli, R

doi: 10.18653/v1/ N18-2097. URL https://aclanthology.org/ N18-2097/. Gokaslan, A., Cohen, V ., Pavlick, E., and Tellex, S. Open- webtext corpus. http://Skylion007.github. io/OpenWebTextCorpus,

work page doi:10.18653/v1/ 2097

[3] [3]

The Llama 3 Herd of Models

URL https://arxiv.org/abs/2407.21783. He, Z., Sun, T., Tang, Q., Wang, K., Huang, X.-J., and Qiu, X. Diffusionbert: Improving generative masked language models with diffusion models. InProceedings of the 61st annual meeting of the association for computational linguistics (ACL), volume 1, pp. 4521–4534,

work page internal anchor Pith review Pith/arXiv arXiv

[4] [4]

Hoogeboom, E., Nielsen, D., Jaini, P., Forr´e, P., and Welling, M

URL https://arxiv.org/ abs/2510.05725. Hoogeboom, E., Nielsen, D., Jaini, P., Forr´e, P., and Welling, M. Argmax flows and multinomial diffusion: Learning categorical distributions. InAdvances in Neural Infor- mation Processing Systems (NeurIPS), volume 34, pp. 12454–12465,

work page arXiv

[5] [5]

Z., Bezemek, Z., Patel, S., Rector-Brooks, J., Yao, S., Bose, A

Peng, F. Z., Bezemek, Z., Patel, S., Rector-Brooks, J., Yao, S., Bose, A. J., Tong, A., and Chatterjee, P. Path plan- ning for masked diffusion model sampling, 2025a. URL https://arxiv.org/abs/2502.03540. Peng, F. Z., Bezemek, Z., Rector-Brooks, J., Zhang, S., Zhang, A. R., Bronstein, M., Bose, A. J., and Tong, A. Planner aware path learning in diffusion ...

work page arXiv

[6] [6]

Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling

URLhttps://arxiv.org/abs/2505.17384. Ye, J., Xie, Z., Zheng, L., Gao, J., Wu, Z., Jiang, X., Li, Z., and Kong, L. Dream 7b: Diffusion large language mod- els,

work page internal anchor Pith review Pith/arXiv arXiv

[7] [7]

11 Unifying Masked Diffusion Models with Various Generation Orders and Beyond A. Related works Discrete diffusion models.Diffusion probabilistic models have become a dominant approach for continuous domains such as images, audio, and video (Ho et al., 2020; Song et al., 2021). This success has motivated extensions to discrete domains, including text, lead...

work page 2020

[8] [8]

Similar phenomena appear in large-scale time-agnostic MDMs: when using random ordering, large-scale MDMs (Ye et al., 2025; Nie et al.,

injects a block-wise left-to-right bias and substantially improves over MDLM. Similar phenomena appear in large-scale time-agnostic MDMs: when using random ordering, large-scale MDMs (Ye et al., 2025; Nie et al.,

work page 2025

[9] [9]

often lag behind large-scale ARMs such as LLaMA-3 (Grattafiori et al., 2024). However, adopting structured decoding—e.g., block-wise L2R generation, within each block, revealing the highest-confidence positions first—can close this gap and even surpass ARMs in some settings. Overall, these results highlight that the choice of unmasking order matters in MD...

work page 2024

[10] [10]

improve the order only via post-training (Hong et al., 2025; Peng et al., 2025a),

work page 2025

[11] [11]

To address these limitations, we return to the canonical continuous-time MDLM formulation and focus on the scheduler as the central mechanism that governs the unmasking process

fail to fully capture context-dependent ordering over the entire sequence (Shi et al., 2024). To address these limitations, we return to the canonical continuous-time MDLM formulation and focus on the scheduler as the central mechanism that governs the unmasking process. This perspective enables a principled integration of MDM training with learnable, con...

work page 2024

[12] [12]

Following conventional MDMs (Sahoo et al., 2024a; Shi et al., 2024), we take the terminal forward distribution to be fully masked: qαF (z(i) t(T) | x) = Cat(m) for all i∈[L]

such that s(τ) =t(τ−1) . Following conventional MDMs (Sahoo et al., 2024a; Shi et al., 2024), we take the terminal forward distribution to be fully masked: qαF (z(i) t(T) | x) = Cat(m) for all i∈[L] . Likewise, the model’s initial noise distribution is fully masked:pθ,ˆαF (z(i) t(T) ) = Cat(m) for alli∈[L]. Finally, the reconstruction distribution att(0)i...

work page 2024

[13] [13]

αmdlm(t) 1−α mdlm(t) logp θ,bd3lm(xb |z b t ,x <b) # (80) = BX b=1 Et∼Unif[0,1] Ezb t ∼qαmdlm(·|xb)

Summing overi= 1, . . . , Lproves LOeMDM(x, θ, αarm,ε, αarm,ε) = LX i=1 −log⟨x (i) θ (yi),x (i)⟩+O(ε) =−log LY i=1 ⟨x(i) θ (yi),x (i)⟩+O(ε), which completes the proof. 28 Unifying Masked Diffusion Models with Various Generation Orders and Beyond Furthermore, we can extend the above theoretical results for the auto-regressive modeling of any fixed order: C...

work page 2025

[14] [14]

dog”] > αGen(t)[“cat

defines vocabulary-wise scheduler αGen :T →[0,1] V+1 where each dimension represents the different amount of noise for corresponding word. In this regard, forward process adds different amount of noise to each position depending on whichwordit owes,e.g., if αGen(t)[“dog”] > αGen(t)[“cat”]3, scheduler noise “cat” more than “dog”. To investigate further, we...

work page 2024

[15] [15]

Training Stabilization Techniques E.1

E. Training Stabilization Techniques E.1. REINFORCE Leave-One-Out In Section 4.3, we have stated that Eqαϕ [∇ϕ logq αϕ ·(L main +L velocity)] in ∇ϕLLoMDM =E qαϕ [∇ϕ logq αϕ ·(L main + Lvelocity)] +E qαϕ [∇ϕ(Lmain +L velocity)] is a high-variance estimator. Therefore, we use a low-variance estimator of such optimization problems proposed by Kool et al. (20...

work page 2019

[16] [16]

However, the image and text domains differ substantially, making it nontrivial to directly transfer such techniques

Unlike discrete diffusion, which has only recently begun to be explored, the continuous domain (e.g., image diffusion) has a longer history of studyinglearnable noise schedulers. However, the image and text domains differ substantially, making it nontrivial to directly transfer such techniques. We therefore first briefly review how learnable schedulers ar...

work page 2021

[17] [17]

While continuous diffusion often allows highly flexible parameterizations of α, in text space we instead require a suitably regularized and simple functional form

Parametrizion of LoMDM.From the continuous-diffusion case above, we draw one key lesson: the velocityA should admit asimple formthat enables direct optimization. While continuous diffusion often allows highly flexible parameterizations of α, in text space we instead require a suitably regularized and simple functional form. Motivated by the simplest sched...

work page 2025

[18] [18]

Experimental Details G.1

G. Experimental Details G.1. Experimental Settings We detokenize the One Billion Words dataset following Lou et al. (2024); Sahoo et al. (2024a), whose official code can be found. We tokenize LM1B using theBERT-BASE-UNCASEDtokenizer, consistent with He et al. (2023). We then concatenate and pack the sequences to a fixed length of 128 (Raffel et al., 2020)...

work page 2024

[19] [19]

(2024); Sahoo et al

from Lou et al. (2024); Sahoo et al. (2024a). We use 12 layers, a hidden dimension of 768, 12 attention heads, and for LoMDM, one more same size of transformer layer for each ϕ, ψ. For diffusion backbone, we use the AdamW optimizer with a batch size of 512, constant learning rate warmup from 0 to a learning rate of 3e-4 for 2,500 steps following prior wor...

work page 2024