Pith. sign in

REVIEW 2 major objections 4 minor 14 cited by

Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Entropy-bounded unmasking speeds masked diffusion sampling 2-3x

desk verdict A simple, well-motivated adaptive multi-token unmasking rule for masked diffusion models, with honest experiments and a real but bridgeable gap between the theory and the implementation. read the letter →

arxiv 2505.24857 v1 pith:CZ6EL7JC submitted 2025-05-30 cs.LG

Heli Ben-Hamu , Itai Gat , Daniel Severo , Niklas Nolte , Brian Karrer This is my paper · ORCID
classification cs.LG
keywords maskeddiffusionmodelsefficientsamplingentropy-boundedunmaskingadaptivemulti-tokenjointdependenceerrornon-autoregressivegenerationcodeandmathreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EB-Sampler is a drop-in replacement for the standard sampling loop of masked diffusion models. It observes that one forward pass of the model often contains reliable predictions for several masked tokens at once, and it unmasks a variable number of tokens per step, stopping when the sum of the chosen tokens' entropies minus the largest entropy exceeds a threshold $\gamma$. The paper claims this entropy bound approximately controls the joint dependence error of sampling tokens independently, and that the resulting sampler sits on a better accuracy-versus-compute Pareto frontier than fixed Top-k samplers. On the LLaDa 8B and Dream 7B models it reports 2-3x fewer function evaluations on code and math reasoning benchmarks without loss in pass@1 accuracy, and similar gains on maze and Sudoku tasks.

What carries the argument

The load-bearing mechanism is the entropy-bound criterion of Equation (2): unmask the largest subset $U$ of masked tokens sorted by error proxy such that $\sum_{l\in U} H(p_\theta(x_l \mid x_{z_{<i}})) - \max_{l\in U} H(p_\theta(x_l \mid x_{z_{<i}})) \leq \gamma$. This expression approximates a rigorous upper bound on the joint dependence error $D_{\mathrm{KL}}(q(x_U \mid x_{z_{<i}}) \,\|\, \prod_{l\in U} q(x_l \mid x_{z_{<i}}))$, replacing true data entropies with model entropies. The bound decides both which tokens are unmasked (low error-proxy first) and how many tokens are unmasked at once, making the step count adaptive to how predictable the remaining sequence is.

What would settle it

On a held-out corpus, estimate the true joint dependence term $D_{\mathrm{KL}}(q(x_U \mid x_{z_{<i}}) \,\|\, \prod_{l\in U} q(x_l \mid x_{z_{<i}}))$ for the subsets $U$ that EB-Sampler actually selects, and compare it with the entropy expression computed from model entropies; if the model-entropy expression frequently underestimates the true term, or if the selected subsets show large true dependence at matched accuracy, the central acceleration claim fails.

Watch

Extended reading notes

Core claim

The central claim is that a single function evaluation of a masked diffusion model carries enough information to determine several tokens at once, provided those tokens are highly predictable, and that a sampler can exploit this by unmasking multiple tokens per step subject to an entropy bound. The paper formalizes sampling as an ordered partition generated by an adaptive policy $\varphi$, and decomposes the KL divergence between the sampler's distribution and the true data distribution into a model-error term and a joint dependence error term. It then shows the joint dependence error is upper-bounded by the sum of the data entropies of the selected tokens minus their maximum, which in practice is approximated by model entropies. EB-Sampler applies this by sorting masked tokens by a model-error proxy (confidence, entropy, or margin) and unmasking the longest prefix whose cumulative entropy minus the running maximum stays below $\gamma$. The paper reports that this accelerates LLaDa 8B and Dream 7B by roughly 2-3x on standard code and math reasoning benchmarks at matched accuracy.

Load-bearing premise

The load-bearing premise is that the model's predicted entropy and error-proxy scores are reliable stand-ins for true data uncertainty and true model error; if they are miscalibrated, the entropy bound no longer bounds the real joint dependence error and the speedups rest on an empirical heuristic.

Editorial extensions

If this is right

  • Any pre-trained masked diffusion model can adopt EB-Sampler without retraining, since the change is confined to the sampling loop.
  • At matched pass@1 accuracy, EB-Sampler needs roughly 2-3x fewer function evaluations than one-token-per-step sampling on HumanEval, MBPP, GSM8K, and MATH.
  • At matched function-evaluation budgets, EB-Sampler achieves higher accuracy than fixed Top-k samplers, improving the accuracy-NFE Pareto frontier.
  • On maze navigation and Sudoku, EB-Sampler retains near-full accuracy at around 10-15 function evaluations where fixed Top-k samplers degrade.
  • The KL decomposition implies that any adaptive unmasking policy's error is the sum of model error and joint dependence error, giving a principled target for future sampler design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If model entropies are reasonably calibrated, EB-Sampler's adaptive step count can double as a difficulty signal: easy continuations decode in few evaluations and hard ones in many, which could guide early stopping or test-time compute allocation.
  • The same entropy-bounded selection criterion should transfer to other structured discrete generation tasks, such as protein or molecule design, wherever a model outputs factorized conditionals; the paper's maze and Sudoku results are initial evidence.
  • EB-Sampler is complementary to speculative decoding schemes: it could be used to speed up the draft model, since the paper's sampling loop does not change the underlying distribution of the target model.
  • A stronger theoretical variant would replace model entropies with calibrated uncertainty estimates or a learned proxy for true joint dependence, tightening the bound without altering the algorithm.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces EB-Sampler, a drop-in replacement for the sampling procedures used with masked diffusion models (MDMs). At each function evaluation, EB-Sampler sorts the currently masked tokens by an error proxy (model confidence, entropy, or margin), then unmasks a prefix of that sorted list whose cumulative model-prediction entropies satisfy sum(U) - max(U) <= gamma, so that the number of tokens unmasked per step varies adaptively. The authors provide a KL error decomposition (Section 5, Appendix A.1) that splits the discrepancy between the sampler's distribution and the data distribution into a model error term and a joint dependence error term, and they upper-bound the latter by sum-minus-max of data entropies. They then motivate the entropy stopping rule by replacing data entropies with model entropies under an explicit approximation assumption. Empirically, on LLaDa 8B and Dream 7B, EB-Sampler improves the accuracy-versus-NFE Pareto frontier on HumanEval, MBPP, GSM8K, and MATH, reporting 2-3x speed-ups at matched accuracy, and shows similar advantages on small maze and Sudoku models. The paper also discusses measurement subtleties in MDM efficiency, introducing a generate_until logic and semi-autoregressive ablations to avoid overstating gains.

Significance. If the empirical results hold, this is a practically valuable contribution: it offers a simple, training-free way to speed up sampling from existing large masked diffusion models, with code-level clarity and honest efficiency measurements. The theoretical error decomposition is clean and self-contained, and the connection between the bound and the algorithm is clearly laid out, even where it relies on assumptions. The paper is commendably careful about efficiency evaluation, including the generate_until post-process and semi-autoregressive ablations, and it makes the implementation change explicit in a short PyTorch snippet. The main caveat is that the rigorous bound (8) uses data entropies while the implemented rule (2) uses model entropies, with no calibration evidence; this weakens the advertised 'predefined approximate error tolerance' but does not invalidate the empirical Pareto-front findings.

major comments (2)
  1. [Section 4, Eq. (2) and Section 5, Eqs. (8)-(9)] The rigorous joint dependence bound in Eq. (8) is stated in terms of data entropies H(q(x_l | x_{z<i})), but the EB-Sampler stopping rule in Eq. (2) uses model entropies H(p_theta(x_l | x_{z<i})). The only justification is the sentence in Section 5, 'Assume we can identify low model error tokens and we design phi to only select z_i from such tokens, where for all l in z_i, p_theta(...) ≈ q(...).' No calibration or consistency check is provided. If the model is overconfident, model entropy can be much smaller than data entropy, so the selected set U may have true joint dependence error well above gamma. Consequently, the abstract's claim of a 'predefined approximate error tolerance' and the phrase in Section 4 that Eq. (2) 'approximately bounds a rigorous joint dependence error' are not substantiated by the theory as written. Please either add an empirical calibration study (e.g., comparing model and data entropies on a held-out set, and/or measuring the actual joint dependence error of EB-Sampler-selected subsets) or soften the claims to describe gamma as a tuned hyperparameter with empirical justification.
  2. [Section 5, 'Expressiveness of phi' and Eq. (7)] The theoretical motivation rests on the assumption that the error proxies (entropy, confidence, margin) identify low-model-error tokens, but the paper never directly measures model error (e.g., D_KL(q(x_l | x_{z<i}), p_theta(x_l | x_{z<i}))) or verifies that the proxy ordering correlates with it. Figures 2 and 3 show downstream accuracy improvements from these proxies, which is suggestive but not the same as validating the assumption. As a result, the reader cannot tell from the paper whether the observed gains come from the entropy bound per se or from the adaptive-k heuristic that unmask more tokens when the model is confident. A direct test of the bound's components, or at least a discussion of this gap, would strengthen the claim that the theory 'motivates' the algorithmic choices.
minor comments (4)
  1. [Section 5, after Eq. (7)] The sentence 'This second KL divergence is precisely joint mutual information' is technically imprecise for subsets of more than two tokens: the divergence between a joint distribution and the product of its marginals is total correlation (multi-information), not mutual information. The mathematical content is unaffected, but the terminology should be corrected.
  2. [Section 4, paragraph after Eq. (2)] The text states that 'gamma = 0 will unmask one token each step,' but the condition sum(U) - max(U) <= 0 also permits multiple tokens with zero model entropy (e.g., deterministic predictions under zero-temperature sampling) to be unmasked in one step. The statement is therefore not strictly accurate and could be clarified.
  3. [Appendix A.1, first paragraph] There is an incomplete sentence: 'the sum and expectation can be interchanged since the sum does not' breaks off mid-phrase. Please complete or rephrase.
  4. [Related Work, Section 7] The name 'Messionic' appears to be a typo for 'Meissonic' (the reference list entry uses 'Meissonic'). Please correct.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the error decomposition is derived from definitions, and the model-entropy substitution is an explicit approximation rather than a tautology.

full rationale

The paper's derivation chain is self-contained. Equation (7) follows from the definitions of p_phi and q_phi in Equations (5)-(6) with the phi terms cancelling; Appendix A.1 supplies the algebra. Equation (8) is the standard information identity D_KL(q(x_zi | x_{<i}) || prod_l q(x_l | x_{<i})) = sum_l H(q(x_l | x_{<i})) - H(q(x_zi | x_{<i})) <= sum_l H(q(x_l | x_{<i})) - max_l H(q(x_l | x_{<i})), since the joint entropy is at least the maximum marginal entropy. This is a mathematical inequality, not an assumption of the result. The EB-Sampler stopping rule in Equation (2) replaces the true entropies with model entropies H(p_theta) at Equation (9), but the paper explicitly conditions this replacement on 'Assume we can identify low model error tokens... p_theta(x_l | x_{z<i}) approx q(x_l | x_{z<i})'. That is an unproven calibration or approximation assumption and a genuine correctness risk, but it is not circular: the error being bounded is still the KL divergence against q, not the model-computed expression itself. The empirical claims are validated against external models (LLaDa 8B, Dream 7B) and standard benchmarks, with gamma swept as a hyperparameter and Pareto fronts reported rather than fitted to the test set. Self-citations (Gat et al. 2024, Kitouni et al. 2024, Nolte et al. 2024) appear only for data generation, training code, and related-work context, and none is load-bearing for the entropy-bound derivation. Appendix A.1 contains a truncated sentence ('since the sum does not'), but the omitted interchange of sum and expectation is standard and does not affect the derivation.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim is supported by a standard KL decomposition with one load-bearing substitution: model entropy is used in place of true data entropy in the dependence bound. This is not an invented entity, but it is an unvalidated assumption. The only fitted free parameter is gamma, which is swept per task. The empirical claims rely on the existing models LLaDa and Dream, and the small maze/Sudoku models are trained by the authors but not released.

free parameters (1)
  • gamma (entropy threshold) = 0.001, 0.01, 0.1 depending on task
    The threshold gamma is chosen by hand after sweeping over a range of values per task. It controls the accuracy-efficiency tradeoff and directly determines how many tokens are unmasked per step, so the central claim depends on the existence of a gamma that yields the reported Pareto improvements.
assumptions (4)
  • domain assumption The masked diffusion model p_theta learns factorized conditionals q(x_l | x_{bar M}) for all masked sets, as proven in prior work (Ou et al. 2025; Zheng et al. 2024).
    The entire EB-Sampler construction assumes the model's per-token predictions are meaningful conditional distributions. The paper cites prior proofs for this, but the practical validity depends on the model having learned such conditionals.
  • domain assumption The error proxies (entropy, confidence, margin) from Eq. (1) rank tokens by local model error.
    Section 3.1 empirically motivates this for LLaDa, but EB-Sampler's theoretical guarantee only applies if the selected tokens are actually low model error. The paper states 'Assume we can identify low model error tokens' in Section 5.
  • ad hoc to paper Model entropy can be substituted for data entropy in the joint dependence bound (Eq. 9).
    The bound in Eq. (8) uses H(q(x_l | x_{z<i})), but the algorithm in Eq. (2) uses H(p_theta(...)). The paper does not prove that the model's entropy approximates the data entropy, nor does it provide calibration analysis. This is a gap between the stated theory and the implemented algorithm.
  • standard math The KL divergence between the model and data joint distributions over (x, z) is a useful proxy for the sampling error D_KL(q(x), p_phi(x)).
    The paper uses the ELBO-style inequality D_KL(q(x), p_phi(x)) <= D_KL(q_phi(x, z), p_phi(x, z)) and notes that equality holds for deterministic phi. This is standard and correctly stated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking." pith.science (2026). https://pith.science/paper/CZ6EL7JC

@misc{pith2026250524857,
  author       = {Pith},
  title        = {Pith review of: Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CZ6EL7JC}},
  note         = {Machine review of arXiv:2505.24857}
}
read the original abstract

Recent masked diffusion models (MDMs) have shown competitive performance compared to autoregressive models (ARMs) for language modeling. While most literature has focused on performance enhancing sampling procedures, efficient sampling from MDMs has been scarcely explored. We make the observation that often a given sequence of partially masked tokens determines the values of multiple unknown tokens deterministically, meaning that a single prediction of a masked model holds additional information unused by standard sampling procedures. Based on this observation, we introduce EB-Sampler, a simple drop-in replacement for existing samplers, utilizing an Entropy Bounded unmasking procedure that dynamically unmasks multiple tokens in one function evaluation with predefined approximate error tolerance. We formulate the EB-Sampler as part of a broad family of adaptive samplers for which we provide an error analysis that motivates our algorithmic choices. EB-Sampler accelerates sampling from current state of the art MDMs by roughly 2-3x on standard coding and math reasoning benchmarks without loss in performance. We also validate the same procedure works well on smaller reasoning tasks including maze navigation and Sudoku, tasks ARMs often struggle with.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Ripple-Pivot Search: Active Parallel Decoding for Diffusion Large Language Models

    cs.CL 2026-08 conditional novelty 7.0 of 10

    RPS accelerates diffusion LLM decoding by committing a mid-entropy pivot token selected through lookahead, yielding 4-10x wall-clock speedups with preserved accuracy across three dLLMs and four benchmarks.

  2. Context-weighted Discrete Flow Matching

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Reweighting discrete-flow-matching updates by local context—via a context-weighted sampler or a scaled cross-entropy loss—improves text and molecular generation, cutting generative perplexity on OpenWebText by up to 63%.

  3. The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models

    cs.CL 2026-01 conditional novelty 7.0 of 10

    Arbitrary-order generation lowers the reasoning Pass@k of diffusion LLMs; constraining RL to autoregressive order (JustGRPO) improves accuracy and preserves parallel decoding.

  4. From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Masked diffusion language models fail to exploit their any-order interface because of positional uncertainty; insertion-based (FlexMDM) and latent-segment (LatentMDM) variants recover distinct any-order inference beha...

  5. Rethinking the Generation Order of Block Diffusion Language Models

    cs.CL 2026-07 accept novelty 6.0 of 10

    BDLMs are training-biased toward left-to-right contexts, and PARD exploits that bias with leftmost-prefix parallel unmasking to improve quality–speed trade-offs over existing parallel DLM samplers.

  6. DSB: Dynamic Sliding Block Scheduling for Diffusion LLMs

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Dynamic Sliding Block scheduling adapts the decoding block size and position in diffusion LLMs based on confidence, improving quality and speed over fixed blocks.

  7. Orchestrating Dual-Boundaries: An Arithmetic Intensity Inspired Acceleration Framework for Diffusion Language Models

    cs.CL 2025-11 conditional novelty 6.0 of 10

    ODB-dLLM accelerates diffusion LLM inference by adaptively shortening responses and using a jump-share speculative decoding strategy, achieving 2.6-6.3x over Fast-dLLM.

  8. ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs

    cs.LG 2025-10 conditional novelty 6.0 of 10

    Parallel decoding in diffusion LLMs degrades quality on token-dependent tasks, and existing strategies cannot adapt parallelism to preserve accuracy, as shown by the new 17-task ParallelBench benchmark.

  9. Set Block Decoding is a Language Model Inference Accelerator

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Fine-tuning an autoregressive LLM with an extra masked-token loss lets it decode several non-consecutive future tokens per forward pass, giving 3-5x fewer passes at similar accuracy.

  10. Any-Order Flexible Length Masked Diffusion

    cs.LG 2025-08 conditional novelty 6.0 of 10

    FlexMDM is a discrete diffusion model that provably supports any-order generation over variable-length sequences by learning an insertion expectation alongside the unmasking posterior, validated by length-fidelity, ma...

  11. Plan for Speed: Dilated Scheduling for Masked Diffusion Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Dilated Unmasking Scheduler partitions sequence positions into non-adjacent dilated groups for parallel unmasking in MDLMs, recovering performance lost in naive parallel decoding and yielding deterministic speedups up...

  12. Can I Have Your Order? Monte-Carlo Tree Search for Slot Filling Ordering in Diffusion Language Models

    cs.AI 2026-02 reject novelty 5.0 of 10

    Monte Carlo tree search can pick better slot-filling orders for masked diffusion language models, but the claimed advantage over autoregressive models depends on truncating the autoregressive baselines.

  13. Adaptive Visual Autoregressive Acceleration via Dual-Linkage Entropy Analysis

    cs.CV 2026-02 conditional novelty 5.0 of 10

    A training-free entropy-guided token-pruning framework accelerates VAR image generation up to 2.9× with negligible benchmark loss by activating pruning at an adaptive entropy-growth inflection point and adjusting rati...

  14. Discrete Diffusion Models: A Unified Framework from Tokenization to Generation

    cs.LG 2026-07 unverdicted novelty 4.0 of 10

    Discrete diffusion models are re-framed as instances of a tokenization-centric, four-component design space (corruption, denoiser, objective, sampler) in a broad survey with no new experimental or theoretical results.

Reference graph

Works this paper leans on

43 extracted references · 12 canonical work pages · cited by 14 Pith papers

  1. [1]

    Ali Alp. Sudoku. https://github.com/alicommit-malp/sudoku/tree/main, 2024

    work page 2024

  2. [2]

    Block diffusion: Interpolating between autoregressive and diffusion language models

    Marianne Arriola, Subham Sekhar Sahoo, Aaron Gokaslan, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Justin T Chiu, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. In The Thirteenth International Conference on Learning Representations, 2025. https://openreview.net/forum?id=tyEyYT267x

    work page 2025

  3. [3]

    Structured denoising diffusion models in discrete state-spaces

    Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34: 0 17981--17993, 2021 a

    2021

  4. [4]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021 b

    arXiv 2021

  5. [5]

    Meissonic: Revitalizing masked generative transformers for efficient high-resolution text-to-image synthesis

    Jinbin Bai, Tian Ye, Wei Chow, Enxin Song, Qing-Guo Chen, Xiangtai Li, Zhen Dong, Lei Zhu, and Shuicheng Yan. Meissonic: Revitalizing masked generative transformers for efficient high-resolution text-to-image synthesis. In The Thirteenth International Conference on Learning Representations, 2024

    work page 2024

  6. [6]

    Halton scheduler for masked generative image transformer, 2025

    Victor Besnier, Mickael Chen, David Hurych, Eduardo Valle, and Matthieu Cord. Halton scheduler for masked generative image transformer, 2025. https://arxiv.org/abs/2503.17076

    arXiv 2025

  7. [7]

    Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design

    Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint arXiv:2402.04997, 2024

    arXiv 2024

  8. [8]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11315--11325, 2022

    2022

Show all 43 references
  1. [9]

    Accelerating large language model decoding with speculative sampling

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023

    2023 arXiv

  2. [10]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  3. [11]

    Speculative diffusion decoding: Accelerating language generation through diffusion

    Jacob K Christopher, Brian R Bartoldson, Tal Ben-Nun, Michael Cardei, Bhavya Kailkhura, and Ferdinando Fioretto. Speculative diffusion decoding: Accelerating language generation through diffusion. arXiv preprint arXiv:2408.05636, 2024

    2024 arXiv

  4. [12]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  5. [13]

    Accelerated diffusion models via speculative sampling

    Valentin De Bortoli, Alexandre Galashov, Arthur Gretton, and Arnaud Doucet. Accelerated diffusion models via speculative sampling. arXiv preprint arXiv:2501.05370, 2025

    2025 arXiv

  6. [14]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langu...

    2019

  7. [15]

    Scaling rectified flow transformers for high-resolution image synthesis, 2024

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transf...

    2024 arXiv

  8. [16]

    Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. arXiv preprint arXiv:2407.15595, 2024

    2024 arXiv

  9. [17]

    Reviving any-subset autoregressive models with principled parallel sampling and speculative decoding

    Gabe Guo and Stefano Ermon. Reviving any-subset autoregressive models with principled parallel sampling and speculative decoding. arXiv preprint arXiv:2504.20456, 2025

    2025 arXiv

  10. [18]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  11. [19]

    Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans

    Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In International Conference on Learning Representations, 2022. https://openreview.net/forum?id=Lm8T39vLDTE

    2022

  12. [20]

    Train for the worst, plan for the best: Understanding token ordering in masked diffusions

    Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arXiv preprint arXiv:2502.06768, 2025

    2025 arXiv

  13. [21]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. An introduction to variational autoencoders. CoRR, abs/1906.02691, 2019. http://arxiv.org/abs/1906.02691

    1906 arXiv

  14. [22]

    Disk: A diffusion model for structured knowledge, 2024

    Ouail Kitouni, Niklas Nolte, James Hensman, and Bhaskar Mitra. Disk: A diffusion model for structured knowledge, 2024. https://arxiv.org/abs/2312.05253

    2024 arXiv

  15. [23]

    Discrete predictor-corrector diffusion models for image synthesis

    Jose Lezama, Tim Salimans, Lu Jiang, Huiwen Chang, Jonathan Ho, and Irfan Essa. Discrete predictor-corrector diffusion models for image synthesis. In The Eleventh International Conference on Learning Representations, 2022

    2022

  16. [24]

    Think while you generate: Discrete diffusion with planned denoising

    Sulin Liu, Juno Nam, Andrew Campbell, Hannes St \"a rk, Yilun Xu, Tommi Jaakkola, and Rafael G \'o mez-Bombarelli. Think while you generate: Discrete diffusion with planned denoising. arXiv preprint arXiv:2410.06264, 2024

    2024 arXiv

  17. [25]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834, 2023

    2023 arXiv

  18. [26]

    Scaling up masked diffusion models on text

    Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. In The Thirteenth International Conference on Learning Representations, 2025 a . https://openreview.net/forum?id=WNvvwK0tut

    2025

  19. [27]

    Large language diffusion models, 2025 b

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models, 2025 b . https://arxiv.org/abs/2502.09992

    2025 arXiv

  20. [28]

    Transformers can navigate mazes with multi-step prediction, 2024

    Niklas Nolte, Ouail Kitouni, Adina Williams, Mike Rabbat, and Mark Ibrahim. Transformers can navigate mazes with multi-step prediction, 2024. https://arxiv.org/abs/2412.05117

    2024 arXiv

  21. [29]

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data, 2025

    Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data, 2025. https://arxiv.org/abs/2406.03736

    2025 arXiv

  22. [30]

    -gpts: A new approach to autoregressive models

    Arnaud Pannatier, Evann Courdier, and Fran c ois Fleuret. -gpts: A new approach to autoregressive models. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 143--159. Springer, 2024

    2024

  23. [31]

    : Optimizing sampling schedule of discrete diffusion models

    Yong-Hyun Park, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, and Yuki Mitsufuji. : Optimizing sampling schedule of discrete diffusion models. CoRR, 2024

    2024

  24. [32]

    Path planning for masked diffusion model sampling

    Fred Zhangzhi Peng, Zachary Bezemek, Sawan Patel, Jarrid Rector-Brooks, Sherwood Yao, Alexander Tong, and Pranam Chatterjee. Path planning for masked diffusion model sampling. arXiv preprint arXiv:2502.03540, 2025

    2025

  25. [33]

    Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Jagadeesh, Kunpeng Li, Lu...

    2024 arXiv

  26. [34]

    Fast solvers for discrete diffusion models: Theory and applications of high-order algorithms

    Yinuo Ren, Haoxuan Chen, Yuchen Zhu, Wei Guo, Yongxin Chen, Grant M Rotskoff, Molei Tao, and Lexing Ying. Fast solvers for discrete diffusion models: Theory and applications of high-order algorithms. arXiv preprint arXiv:2502.00234, 2025

    2025

  27. [35]

    Simple and effective masked diffusion language models

    Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Marroquin, Alexander M Rush, Yair Schiff, Justin T Chiu, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In The Thirty-eighth Annual Conference on Neural Information Processing ...

    2024

  28. [36]

    A deep and tractable density estimator

    Benigno Uria, Iain Murray, and Hugo Larochelle. A deep and tractable density estimator. In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 467--475, Bejing...

    2014

  29. [37]

    Remasking discrete diffusion models with inference-time scaling

    Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. arXiv preprint arXiv:2503.00307, 2025

    2025

  30. [38]

    Beyond autoregression: Discrete diffusion for complex reasoning and planning

    Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning. arXiv preprint arXiv:2410.14157, 2024

    2024 arXiv

  31. [39]

    Dream 7b, 2025

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025. https://hkunlp.github.io/blog/2025/dream

    2025

  32. [40]

    Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling

    Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409.02908, 2024

    2024 arXiv

  33. [41]

    A reparameterized discrete diffusion model for text generation

    Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.05737, 2023

    2023 arXiv

  34. [42]

    Di [M] o: Distilling masked diffusion models into one-step generator

    Yuanzhi Zhu, Xi Wang, St \'e phane Lathuili \`e re, and Vicky Kalogeiton. Di [M] o: Distilling masked diffusion models into one-step generator. arXiv preprint arXiv:2503.15457, 2025

    2025 arXiv

  35. [43]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.