REVIEW 3 major objections 5 minor 17 cited by
Masked diffusion now handles variable length in any order
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
FlexMDM is a discrete diffusion model that provably supports any-order generation over variable-length sequences by learning an insertion expectation alongside the unmasking posterior, validated by length-fidelity, maze-planning, and 8B-retrofit experiments.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection Variable-length masked diffusion done right, with a real any-order theorem; the scary combinatorial step checks out, but the empirical package is thinner than the theory. the 3 major comments →
Any-Order Flexible Length Masked Diffusion
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The paper introduces a joint interpolant (x_t, s_t), where a clean sequence x_1 is converted into x_t by deleting, masking, or revealing each token according to two schedules, while s_t records which source positions are still present. Minimizing the proposed loss makes a network f_theta recover the ground-truth unmasking posterior and a network g_theta recover the expected number of tokens to insert at each gap. The rate matrix built from these two quantities solves the Kolmogorov forward equation for the interpolant's marginals, so exact learning implies exact sampling from the target distribution p_1. The any-order theorem then states that any sampler that unmasks an arbitrary subset of m
What carries the argument
The joint interpolant: a stochastic interpolant augmented by an index-tracking set s_t. It enlarges the class of constructible rate matrices, making it possible to start from an easy-to-sample base distribution (the empty string) and to express insertion rates as a per-gap scalar, the insertion expectation, rather than a full token distribution. The two schedules alpha_t (insertion) and beta_t (unmasking), the training loss, the rate matrix, and the any-order theorem all hang on this object.
Load-bearing premise
Everything rests on the trained networks being exactly right about both what each mask hides and how many tokens belong in each gap, and on a counting identity about gaps holding up even for strings with repeated tokens.
What would settle it
Train or simulate FlexMDM with oracle (ground-truth) unmasking posterior and insertion expectation on a distribution over strings with many repeated tokens, then check whether the exact continuous-time Markov chain at t=1 matches the training length distribution; a mismatch, especially one that grows with repetition, would falsify the combinatorial step underlying the any-order theorem.
If this is right
- Variable-length text can be generated in parallel with masked diffusion, and length distributions are learned instead of imposed by padding to a fixed canvas.
- Any-order inference remains theoretically grounded for FlexMDMs, so confidence-based and semi-autoregressive unmasking heuristics can be combined with insertions without breaking the sampling guarantee.
- Pretrained masked diffusion models can be converted into FlexMDMs by adding an insertion-expectation head and fine-tuning, inheriting the base model's capabilities and adding variable-length generation.
- On planning tasks, the model can insert tokens between subgoals instead of preallocating their positions, yielding large success-rate gains as the number of subgoals grows.
- For math and code infilling, more sampling steps continue to improve FlexMDM performance, suggesting reasoning-style gains from additional inference compute.
Where Pith is reading between the lines
- Inference: a testable extension would stress-test FlexMDMs on corpora with heavy token repetition, since the proof's key counting identity relates gap insertions to supersequence matchings and is most delicate when target sequences contain repeated tokens.
- Inference: the same joint-interpolant construction could be adapted to other edit operations—deletion, replacement, reordering—to build diffusion models that perform structured edits of sequences rather than only insertions.
- Inference: the path-independence mechanism that enables any-order unmasking suggests a single trained model could be used at inference with schedules quite different from the training schedules, potentially including learned or task-dependent unmasking orders.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Flexible Masked Diffusion Models (FlexMDMs), a discrete diffusion framework for variable-length sequences. The method defines a joint stochastic interpolant that deletes, masks, and unmasks tokens according to two schedules α_t and β_t, and derives a training loss (Eq. 7) whose minimizers are the unmasking posterior and an insertion expectation. Proposition 2 gives a rate matrix for the corresponding CTMC, and Section 4 claims an any-order inference guarantee: even if masked tokens are revealed in an arbitrary order, as long as unmasking uses the true posterior and insertions follow the ground-truth rate matrix, the terminal distribution is p_1. The paper reports length-distribution fidelity on OpenWebText, success rates on a maze/subgoal planning task, and a 3-day retrofit of LLaDA-8B into a FlexMDM with improved GSM8K and HumanEval-infill scores.
Significance. If the guarantees hold, FlexMDM is a meaningful advance over fixed-length masked diffusion: it models length statistics while preserving any-order decoding, and the LLaDA retrofit suggests a practical upgrade path for existing 8B MDMs. The theoretical construction is largely derived rather than assumed: the loss is a valid variational upper bound, the rate matrix is derived from the interpolant, and the any-order theorem is a substantive oracle-level statement. The main caveats are that the empirical claims are not accompanied by quantitative length-distribution metrics or error bars, and the any-order theorem requires oracle posterior and insertion-expectation access; the learned-model generalization to off-path states is not analyzed. These caveats are standard for the area, and the core contribution is sufficiently novel to merit publication if the proof issues below are fixed.
major comments (3)
- [Appendix E.6, Eq. (14)] This is the load-bearing step in Lemma E.2, and as printed it does not prove the required identity. In the product R_ins_t(x,x◁_i m) q_t(x*|x◁_i m), the posterior normalizer for x◁_i m carries exponent len(x1)-len(x)-1 in (1-α_t), whereas Eq. (14) writes the denominator with exponent len(x1)-len(x). The subset bijection established in the text makes the numerator and denominator equal only for the same exponent; with the correct exponent the ratio is (1-α_t). Tracking that factor against the prefactor αdot/(1-α_t) in R_ins restores the equality with the first term of Eq. (13), so the lemma is repairable. Please rewrite Eq. (14) with the correct exponents and clearly state that the bijection is over subsets of source positions (which is why repeated tokens are harmless). The current one-sentence proof also contains a typo in the final display (x* vs x1 and x vs x_t).
- [Appendix E.2.1 / E.5] The posterior q_t is extended to t=1 by a separate definition that is not the t→1 limit of the t<1 formula. Lemma E.1 is proved only for the t<1 formula (Eqs. (10)-(11)), but Theorem E.1's base case applies Lemma E.1 at t=1. Please give the t=1 version of Lemma E.1 (it holds by a simple normalization argument: q_1(x*|x) ∝ p(x*) 1_{x⊆x*}, so the any-order unmasking step preserves q_1) or restructure the induction to avoid this case. Without this, the proof of the any-order guarantee has a gap at the final unmasking phase.
- [Section 4 / Algorithm 1] Proposition 3 (informal) says 'unmasks an arbitrary subset', while the formal Definition E.1 covers only one index per unmasking step; the adaptive subroutine in Algorithm 1 assigns K positions simultaneously. The paper should clarify that the theorem covers sequential unmasking of subsets and that the simultaneous K-step is an approximation that coincides with the formal class only in the τ→0 limit (where K∼Poisson has at most one event w.h.p.). This is not fatal but is needed for the main-text claim that Algorithm 1 is subsumed by Proposition 3.
minor comments (5)
- [§5.1.1 vs Appendix F.1] The main text says pretraining runs for 500K iterations, while Appendix F.1 states a max training iteration of 1M. Please align these numbers.
- [Figure 4b/c] The claims of 'much higher fidelity' in length modeling rest on a qualitative plot. Please add a quantitative metric (e.g., total variation or KL to the empirical length distribution) and, where feasible, error bars over seeds.
- [Introduction] Typo: 'non-casual tasks' should be 'non-causal tasks'.
- [§5.2] The LLaDA comparison is between an IFT-ed baseline and a FlexMDM that additionally trains LoRA adapters, a time-embedding pathway, and a new scalar head. Please state whether the comparison controls for trainable parameters/compute, since the two setups do not have equal capacity.
- [Algorithm 1 / Subroutine 2] The pseudocode says 'Select K (the size of |S|)' without specifying the distribution; the distribution (Poisson with rate depending on τ and the unmasking schedule) appears only in Appendix E.1. Please make the pseudocode self-contained.
Circularity Check
No significant circularity; FlexMDM's guarantees are derived from stated interpolant definitions and proven in appendices. The only self-citations are contextual and non-load-bearing.
full rationale
FlexMDM's derivation chain is self-contained. The joint interpolant (Eq. 6 / Def. D.2) is a stipulated construction; the rate matrix (Eq. 8 / Prop. D.3) is derived by applying the generic joint-interpolant target-rate proposition (Prop. D.1) to the interpolating rate (Prop. D.2). The loss (Eq. 7 / Prop. D.4) is the path-measure KL divergence; Prop. 1 identifies its unique minimizer as the posterior and insertion expectation used in the rate matrix. This is a characterization, not an assumption of the conclusion. For any-order inference, Theorem E.1 is proven by induction; Lemmas E.1 and E.2 have explicit proofs. Lemma E.2's Eq. (14) is a combinatorial identity established by a bijection ('The key combinatorial step...'), not an imported result; even if the exposition is compressed, this is a correctness/completeness concern, not circularity. The posterior independence from beta_t (Appendix E.2.1) is computed from the interpolant, not assumed. Self-citations (Kim et al. 2025 for MDM any-order folklore; Albergo et al. for stochastic interpolants) are contextual; the paper re-derives the any-order argument for FlexMDM rather than relying on the citation. Empirical length-fidelity is an evaluation of the trained model on the training-length distribution, not a fitted parameter renamed as an independent prediction. Consequently, no load-bearing step reduces to its own input; the low score reflects only the presence of minor, non-load-bearing self-citations.
Axiom & Free-Parameter Ledger
free parameters (5)
- insertion/unmasking schedules alpha_t, beta_t =
alpha=beta=t per footnote 2; main text writes alpha=beta=t^2
- sliding-window heuristics gamma_1, gamma_2 =
gamma_1=5.0, gamma_2=64
- sampling temperature =
0.0
- generative-perplexity length filter =
sequences of <=10 tokens excluded
- LoRA configuration for the 8B retrofit =
r=128, alpha=128, dropout=0.1
axioms (5)
- standard math Kolmogorov forward/backward equations for CTMCs; path-measure KL (Radon-Nikodym) for rate matrices; data processing inequality for the terminal KL bound
- domain assumption p_1 is supported on finite-length strings over a finite vocabulary; per-coordinate independent insertion and unmasking times in the joint interpolant (Eq. 6)
- ad hoc to paper Oracle access to ground-truth posterior marginals q_t and insertion expectations for the any-order guarantees
- ad hoc to paper Combinatorial identity Eq. (14) in the proof of Lemma E.2: the summed gap-insertion counts equal the posterior's time derivative terms in the Kolmogorov backward equation
- ad hoc to paper Posterior extended to t=1 via q_1(x*|x) proportional to p(x*) times the indicator that x is a sub-mask of x*, with oracle access to any-order marginals of p
invented entities (2)
-
Insertion expectation g_theta(x,t)[i]
independent evidence
-
Joint interpolant (x_t, s_t) with index-tracking variable s_t
no independent evidence
Cite this review
Pith. "Pith review of Any-Order Flexible Length Masked Diffusion." pith.science (2026). https://pith.science/paper/I5LCFXQV
@misc{pith2026250901025,
author = {Pith},
title = {Pith review of: Any-Order Flexible Length Masked Diffusion},
year = {2026},
howpublished = {\url{https://pith.science/paper/I5LCFXQV}},
note = {Machine review of arXiv:2509.01025}
}
abstract
Masked diffusion models (MDMs) have recently emerged as a promising alternative to autoregressive models over discrete domains. MDMs generate sequences in an any-order, parallel fashion, enabling fast inference and strong performance on non-causal tasks. However, a crucial limitation is that they do not support token insertions and are thus limited to fixed-length generations. To this end, we introduce Flexible Masked Diffusion Models (FlexMDMs), a discrete diffusion paradigm that simultaneously can model sequences of flexible length while provably retaining MDMs' flexibility of any-order inference. Grounded in an extension of the stochastic interpolant framework, FlexMDMs generate sequences by inserting mask tokens and unmasking them. Empirically, we show that FlexMDMs match MDMs in perplexity while modeling length statistics with much higher fidelity. On a synthetic maze planning task, they achieve $\approx 60 \%$ higher success rate than MDM baselines. Finally, we show pretrained MDMs can easily be retrofitted into FlexMDMs: on 16 H100s, it takes only three days to fine-tune LLaDA-8B into a FlexMDM, achieving superior performance on math (GSM8K, $58\% \to 67\%$) and code infilling performance ($52\% \to 65\%$).
Figures
Forward citations
Cited by 17 Pith papers
-
Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding
Set diffusion factorizes likelihood over arbitrary token sets and uses a set-causal diffusion architecture to support KV caching and any-order decoding, yielding improved speed-quality tradeoffs versus prior diffusion LMs.
-
Discrete Tilt Matching
DTM recasts dLLM fine-tuning as weighted cross-entropy matching of tilted local posteriors, with demonstrated gains on Sudoku and math tasks.
-
Discrete Tilt Matching
Discrete Tilt Matching recasts dLLM fine-tuning as state-level matching of tilted local unmasking posteriors, producing a stable weighted cross-entropy loss that improves Sudoku and Countdown performance when applied ...
-
Generative Modeling from Black-box Corruptions via Self-Consistent Stochastic Interpolants
SCSI iteratively refines a self-consistent transport map to invert black-box corruptions and enable generative modeling of clean data.
-
CreditDecoding: Accelerating Parallel Decoding in Diffusion Large Language Models with Trace Credit
CreditDecoding accelerates parallel decoding in diffusion LLMs by fusing accumulated Trace Credit with current logits to accept early-correct tokens sooner, yielding up to 5.48x speedup and accuracy gains.
-
From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
Masked diffusion language models fail to exploit their any-order interface because of positional uncertainty; insertion-based (FlexMDM) and latent-segment (LatentMDM) variants recover distinct any-order inference beha...
-
Hierarchical Domain Generalization
Over infinite domains, hierarchy-uniform domain generalization is impossible for every nontrivial hypothesis class; a length-generalization bound is a property of the length hierarchy, not a hierarchy-free guarantee.
-
Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation
A diffusion multimodal LLM's early-layer MLP sparsity at denoising step 0 pinpoints the semantic boundary, enabling one-shot suffix truncation that raises throughput up to ~31x.
-
VoidPadding: Let [VOID] Handle Padding in Masked Diffusion Language Models so that [EOS] Can Focus on Semantic Termination
VoidPadding decouples padding from termination in MDLMs via a new [VOID] token, delivering +17.84 average benchmark points and 55.7% fewer decoding steps on Dream-7B-Instruct.
-
Generative Molecular Morphing for Flexible-Size Design via Unbalanced Optimal Transport
Morph is a flexible-size 3D molecular generative model using unbalanced optimal transport on geometric graphs that matches fixed-size SOTA performance while enabling out-of-distribution generation.
-
Fast-dLLM++: Fr\'{e}chet Profile Decoding for Faster Diffusion LLM Inference
Fast-dLLM++ generalizes Fast-dLLM decoding to heterogeneous confidence profiles via Fréchet profile selection, delivering up to 37% throughput gains on GSM8K, MATH, HumanEval, and MBPP with LLaDA-8B.
-
Edit-Based Refinement for Parallel Masked Diffusion Language Models
ME-DLM augments parallel masked diffusion models with edit-distance-supervised refinements to raise quality on coding and math benchmarks while using far fewer diffusion steps.
-
Gumbel Distillation for Parallel Text Generation
Conditioning parallel decoders on Gumbel noise sampled from an autoregressive teacher's Gumbel-Max process improves generation quality on LM1B and OpenWebText.
-
STRIDE: Post-Training LLMs to Reason and Refine Bio-Sequences via Edit Trajectories
Training LLMs to emit executable edit trajectories (INSERT/DELETE/REPLACE) from Levenshtein alignments plus policy optimization improves oracle-scored bio-sequence optimization success and novelty.
-
Reconsidering Positional Supervision in Masked Diffusion Language Model Training
Adding a CTC-style <SLACK> token during MDLM supervised fine-tuning relaxes strict positional supervision and improves open-ended generation benchmarks over a matched cross-entropy baseline.
-
A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models
Visual token pruning causes severe loss in discrete diffusion MLLMs; only from-scratch models on long-answer tasks recover via late denoising, so redundancy is recoverability, not dispensability.
-
CreditDecoding: Accelerating Parallel Decoding in Diffusion Large Language Models with Trace Credit
A token-history 'credit' score fused into current logits reduces redundant re-masking in diffusion LLM decoding, yielding up to 5.48× tokens-per-forward speedup with small accuracy gains.
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 0 6840--6851, 2020
2020
-
[3]
Score-based generative modeling through stochastic differential equations
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020
Pith/arXiv arXiv 2011
-
[4]
Deep unsupervised learning using nonequilibrium thermodynamics
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256--2265. pmlr, 2015
2015
-
[5]
Simplified and generalized masked diffusion for discrete data
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37: 0 103131--103167, 2024
2024
-
[6]
Simple and effective masked diffusion language models
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37: 0 130136--130184, 2024
2024
-
[7]
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. Advances in Neural Information Processing Systems, 37: 0 133345--133385, 2024
work page 2024
-
[8]
Beyond autoregression: Discrete diffusion for complex reasoning and planning
Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning. arXiv preprint arXiv:2410.14157, 2024
Pith/arXiv arXiv 2024
-
[9]
Large language diffusion models
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025
Pith/arXiv arXiv 2025
-
[10]
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025. URL https://hkunlp.github.io/blog/2025/dream
work page 2025
-
[11]
Scaling up masked diffusion models on text
Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.18514, 2024
Pith/arXiv arXiv 2024
-
[12]
Albergo and Eric Vanden-Eijnden
Michael S. Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants, 2022
work page 2022
-
[13]
Stochastic interpolants: A unifying framework for flows and diffusions
Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797, 2023 a
Pith/arXiv arXiv 2023
-
[14]
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2022
2022
-
[15]
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409.02908, 2024
Pith/arXiv arXiv 2024
-
[16]
Your absorbing discrete diffusion secretly models the conditional distributions of clean data
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.03736, 2024
Pith/arXiv arXiv 2024
-
[17]
Dreamon: Diffusion language models for code infilling beyond fixed-size canvas, 2025 a
Zirui Wu, Lin Zheng, Zhihui Xie, Jiacheng Ye, Jiahui Gao, Yansong Feng, Zhenguo Li, Victoria W., Guorui Zhou, and Lingpeng Kong. Dreamon: Diffusion language models for code infilling beyond fixed-size canvas, 2025 a . URL https://hkunlp.github.io/blog/2025/dreamon
work page 2025
-
[18]
Edit flows: Flow matching with edit operations
Marton Havasi, Brian Karrer, Itai Gat, and Ricky TQ Chen. Edit flows: Flow matching with edit operations. arXiv preprint arXiv:2506.09018, 2025
arXiv 2025
-
[19]
A continuous time framework for discrete denoising models
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35: 0 28266--28279, 2022
2022
-
[20]
Train for the worst, plan for the best: Understanding token ordering in masked diffusions
Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arXiv preprint arXiv:2502.06768, 2025
Pith/arXiv arXiv 2025
-
[21]
Michael S. Albergo, Nicholas M. Boffi, Michael Lindsey, and Eric Vanden-Eijnden. Multimarginal generative modeling with stochastic interpolants, 2023 b . URL https://arxiv.org/abs/2310.03695
Pith/arXiv arXiv 2023
-
[22]
Albergo, and Eric Vanden-Eijnden
Hugo Negrel, Florentin Coeurdoux, Michael S. Albergo, and Eric Vanden-Eijnden. Multitask learning with stochastic interpolants, 2025. URL https://arxiv.org/abs/2508.04605
-
[23]
Scalable diffusion models with transformers
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195--4205, 2023
2023
-
[24]
Scaling diffusion language models via adaptation from autoregressive models
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410.17891, 2024
Pith/arXiv arXiv 2024
-
[25]
Openwebtext corpus
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019
2019
-
[26]
Planning with diffusion for flexible behavior synthesis
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, 2022
work page 2022
-
[27]
Stochastic interpolants with data-dependent couplings
Michael Samuel Albergo, Mark Goldstein, Nicholas Matthew Boffi, Rajesh Ranganath, and Eric Vanden-Eijnden. Stochastic interpolants with data-dependent couplings. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=FFILRGD0jG
work page 2024
-
[28]
Jiang, Jia Deng, Stella Biderman, and Sean Welleck
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2023
2023
-
[29]
d1: Scaling reasoning in diffusion large language models via reinforcement learning
Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. d1: Scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint arXiv:2504.12216, 2025
Pith/arXiv arXiv 2025
-
[30]
Training verifiers to solve math word problems
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021
Pith/arXiv arXiv 2021
-
[31]
Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, J. H. Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, and Wei Chu. Opencoder: The open cookbook for top-tier code large language models. 2024. URL https://arxiv.org/pdf/2411.04905
Pith/arXiv arXiv 2024
-
[32]
Efficient training of language models to fill in the middle
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255, 2022
Pith/arXiv arXiv 2022
-
[33]
dkv-cache: The cache for diffusion language models
Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models. arXiv preprint arXiv:2505.15781, 2025 a
Pith/arXiv arXiv 2025
-
[34]
Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, and Arash Vahdat. Esoteric language models. arXiv preprint arXiv:2506.01928, 2025
Pith/arXiv arXiv 2025
-
[35]
Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.22618, 2025 b
Pith/arXiv arXiv 2025
-
[36]
Argmax flows and multinomial diffusion: Learning categorical distributions
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forr \'e , and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in neural information processing systems, 34: 0 12454--12465, 2021
2021
-
[37]
Structured denoising diffusion models in discrete state-spaces
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34: 0 17981--17993, 2021
2021
-
[38]
Discrete diffusion modeling by estimating the ratios of the data distribution
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834, 2023
Pith/arXiv arXiv 2023
-
[39]
Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint arXiv:2402.04997, 2024
Pith/arXiv arXiv 2024
-
[40]
Flow matching with general discrete paths: A kinetic-optimal perspective
Neta Shaul, Itai Gat, Marton Havasi, Daniel Severo, Anuroop Sriram, Peter Holderrieth, Brian Karrer, Yaron Lipman, and Ricky TQ Chen. Flow matching with general discrete paths: A kinetic-optimal perspective. arXiv preprint arXiv:2412.03487, 2024
Pith/arXiv arXiv 2024
-
[41]
Path planning for masked diffusion model sampling
Fred Zhangzhi Peng, Zachary Bezemek, Sawan Patel, Jarrid Rector-Brooks, Sherwood Yao, Avishek Joey Bose, Alexander Tong, and Pranam Chatterjee. Path planning for masked diffusion model sampling. arXiv preprint arXiv:2502.03540, 2025
arXiv 2025
-
[42]
Anchored diffusion language model
Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. Anchored diffusion language model. arXiv preprint arXiv:2505.18456, 2025
Pith/arXiv arXiv 2025
-
[43]
Diffucoder: Understanding and improving masked diffusion models for code generation
Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. Diffucoder: Understanding and improving masked diffusion models for code generation. arXiv preprint arXiv:2506.20639, 2025
Pith/arXiv arXiv 2025
-
[44]
Seed diffusion: A large-scale diffusion language model with high-speed inference
Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, et al. Seed diffusion: A large-scale diffusion language model with high-speed inference. arXiv preprint arXiv:2508.02193, 2025
Pith/arXiv arXiv 2025
-
[45]
Mercury: Ultra-fast language models based on diffusion
Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, et al. Mercury: Ultra-fast language models based on diffusion. arXiv preprint arXiv:2506.17298, 2025
Pith/arXiv arXiv 2025
-
[46]
Accelerated sampling from masked diffusion models via entropy bounded unmasking
Heli Ben-Hamu, Itai Gat, Daniel Severo, Niklas Nolte, and Brian Karrer. Accelerated sampling from masked diffusion models via entropy bounded unmasking. arXiv preprint arXiv:2505.24857, 2025
Pith/arXiv arXiv 2025
-
[47]
Unified multimodal discrete diffusion
Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, and Katerina Fragkiadaki. Unified multimodal discrete diffusion. arXiv preprint arXiv:2503.20853, 2025
Pith/arXiv arXiv 2025
-
[48]
Google DeepMind. Gemini diffusion, 2025. URL https://blog.google/technology/google-deepmind/gemini-diffusion/
work page 2025
-
[49]
Maskgit: Masked generative image transformer
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11315--11325, 2022
work page 2022
-
[50]
Reinforced Context Order Recovery for Adaptive Reasoning and Planning
Long Ma, Fangwei Zhong, and Yizhou Wang. Reinforced context order recovery for adaptive reasoning and planning. arXiv preprint arXiv:2508.13070, 2025 b
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[51]
Learning-order autoregressive models with application to molecular graph generation
Zhe Wang, Jiaxin Shi, Nicolas Heess, Arthur Gretton, and Michalis K Titsias. Learning-order autoregressive models with application to molecular graph generation. arXiv preprint arXiv:2503.05979, 2025
Pith/arXiv arXiv 2025
-
[52]
Rewon child, david luan, dario amodei, and ilya sutskever
Alec Radford and Jeffrey Wu. Rewon child, david luan, dario amodei, and ilya sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019
work page 2019
-
[53]
Flex attention: A programming model for generating optimized attention kernels
Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A programming model for generating optimized attention kernels. arXiv preprint arXiv:2412.05496, 2024
Pith/arXiv arXiv 2024
-
[54]
Flashattention: Fast and memory-efficient exact attention with io-awareness
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35: 0 16344--16359, 2022
2022
-
[55]
Llama 2: Open foundation and fine-tuned chat models
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Harts...
Pith/arXiv arXiv 2023
-
[56]
Decoupled weight decay regularization
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017
Pith/arXiv arXiv 2017
-
[57]
Deepseek llm: Scaling open-source language models with longtermism
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024
Pith/arXiv arXiv 2024
-
[58]
Qwen. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024
Pith/arXiv arXiv 2024
-
[59]
Lo RA : Low-rank adaptation of large language models
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9
2022
-
[60]
Flow straight and fast: Learning to generate and transfer data with rectified flow, 2022
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow, 2022. URL https://arxiv.org/abs/2209.03003
Pith/arXiv arXiv 2022
-
[61]
Peter Holderrieth, Marton Havasi, Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky T. Q. Chen, and Yaron Lipman. Generator matching: Generative modeling with arbitrary markov processes. In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openreview.net/forum?id=RuP17cJtZo
work page 2025
-
[62]
From denoising diffusions to denoising markov models, 2024
Joe Benton, Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. From denoising diffusions to denoising markov models, 2024. URL https://arxiv.org/abs/2211.03595
Pith/arXiv arXiv 2024
-
[63]
Trans-dimensional generative modeling via jump diffusion models, 2023
Andrew Campbell, William Harvey, Christian Weilbach, Valentin De Bortoli, Tom Rainforth, and Arnaud Doucet. Trans-dimensional generative modeling via jump diffusion models, 2023. URL https://arxiv.org/abs/2305.16261
Pith/arXiv arXiv 2023
-
[64]
Transport meets variational inference: Controlled monte carlo diffusions, 2025
Francisco Vargas, Shreyas Padhy, Denis Blessing, and Nikolas Nüsken. Transport meets variational inference: Controlled monte carlo diffusions, 2025. URL https://arxiv.org/abs/2307.01050
Pith/arXiv arXiv 2025
-
[65]
An optimal control perspective on diffusion-based generative modeling
Julius Berner, Lorenz Richter, and Karen Ullrich. An optimal control perspective on diffusion-based generative modeling. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=oYIjw37pTP
2024
-
[66]
Non-denoising forward-time diffusions, 2022
Stefano Peluchetti. Non-denoising forward-time diffusions, 2022. URL https://openreview.net/forum?id=oVfIKuhqfC
2022
-
[67]
LEAPS : A discrete neural sampler via locally equivariant networks
Peter Holderrieth, Michael Samuel Albergo, and Tommi Jaakkola. LEAPS : A discrete neural sampler via locally equivariant networks. In Forty-second International Conference on Machine Learning, 2025 b . URL https://openreview.net/forum?id=Hq2RniQAET
work page 2025
-
[68]
Neta Shaul, Itai Gat, Marton Havasi, Daniel Severo, Anuroop Sriram, Peter Holderrieth, Brian Karrer, Yaron Lipman, and Ricky T. Q. Chen. Flow matching with general discrete paths: A kinetic-optimal perspective. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=tcvMzR2NrP
work page 2025
-
[69]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...
-
[70]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...
-
[71]
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.