Pith. sign in

REVIEW 4 major objections 5 minor 19 references

This paper claims that few-step discrete diffusion samplers are not distributionally correct even when the learned denoiser is replaced by an exact oracle posterior, so the sampling dynamics themselves introduce error that standard metrics

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:34 UTC pith:K57GA7OE

load-bearing objection Useful HMM-oracle framework for isolating sampler error, but the abstract overclaims and the LLaDA result is a coupling artifact rather than clean evidence. the 4 major comments →

arxiv 2602.19619 v2 pith:K57GA7OE submitted 2026-02-23 cs.LG

Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models

Luhan Tang , Longxuan Yu , Shaorong Zhang , Greg Ver Steeg This is my paper
classification cs.LG
keywords discrete diffusionsampler-induced errororacle evaluationhidden Markov modeltransition-level correctnessgenerative perplexityMAUVEmasked diffusion
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Discrete diffusion language models generate text by iterative denoising rather than left-to-right prediction, but the paper argues that their evaluation conflates two distinct errors: the denoiser's approximation error and the error introduced by the sampling procedure itself. To separate these, the authors build a ground-truth Markov chain over tokens and compute the exact posterior under partial observations using hidden Markov model smoothing. Replacing the learned denoiser in SEDD, MDLM, LLaDA, and ReMDM with this exact oracle, they claim that few-step samplers still deviate substantially from the true transition distribution, with the deviation fading only as the number of steps approaches the sequence length. They also claim that common metrics can mislead: negative log-likelihood and generative perplexity can improve under low-temperature sharpening while the sampler becomes less correct, and MAUVE can stay flat while transition-level error changes dramatically. The practical stake is that reported few-step quality may be partly a property of the sampling dynamics, not of the model.

Core claim

The paper's central claim is that few-step discrete diffusion samplers are not distributionally correct even under an oracle denoiser. Across four sampler families, the empirical token-to-token transition distribution differs from the ground-truth Markov kernel at small step counts, with transition KL, NLL, and entropy approaching the autoregressive baseline only when the number of steps approaches the sequence length; LLaDA is the exception, degenerating further as steps increase instead of converging. The paper further claims that likelihood-based metrics can improve under biased sampling and that MAUVE can remain insensitive to transition-level mismatch, so improvements in those numbers d

What carries the argument

The central object is an oracle evaluation framework built from a ground-truth discrete-time Markov chain over tokens. Partial observations from the masking process turn the chain into a hidden Markov model, and exact forward-backward smoothing yields the posterior marginal at every position; replacing each sampler's learned denoiser with these exact marginals (and, for SEDD, the corresponding concrete scores) while keeping the sampling logic unchanged isolates sampler-induced error. A sparse top-K transition kernel with a small teleport term keeps the exact message passing at O(VK) per step.

Load-bearing premise

The claim that observed error is purely sampler-induced rests on replacing each learned denoiser with the exact HMM posterior without altering the sampling logic, a substitution the paper itself notes fails for LLaDA because that sampler's re-masking relies on denoiser confidence scores carrying linguistic priors the oracle does not share.

What would settle it

Use the paper's own setup (ground-truth Markov prior, oracle posterior, sequence length 1024) and run MDLM with 8 diffusion steps; if the empirical transition KL equals the autoregressive baseline (about 0.2454 in the OpenWebText configuration) or is within statistical noise, the claim of persistent few-step sampler error would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If true, fluent or low-NLL output from few-step discrete diffusion samplers cannot be taken as evidence of correct sampling; the sampling dynamics alone can bias the distribution.
  • The number of steps is itself a correctness parameter: transition-level mismatch vanishes only as steps approach the sequence length, so step count determines how closely sampling matches the target transition law.
  • Generative perplexity can be systematically improved by local sharpening without faithful sampling, so GenPPL should be treated as an evaluator-alignment measure rather than a correctness metric.
  • MAUVE can stay high while transition-level errors change substantially, motivating transition-aware metrics such as transition KL, total variation, and entropy rate against a known kernel.
  • Sampler design can create induced collapse: under an oracle posterior, LLaDA's confidence-based remasking produces increasing template-like repetition and entropy collapse as steps grow.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One implication the authors leave implicit: if sampling dynamics alone can produce template repetition under an exact oracle, part of the degeneration seen in real diffusion language models may be attributable to the decoder rather than the trained model; a direct test is to run the same frozen model through two different samplers and compare transition statistics.
  • The oracle benchmark is synthetic by design, but it suggests a testable extension: before trusting few-step outputs from a new discrete sampler, run it under this Markov-chain oracle and require its transition KL to approach the autoregressive baseline.
  • Because the ground-truth prior is first-order Markov, extending the framework to higher-order Markov priors with exact inference would show whether the step-length dependence of sampler error worsens or softens for language-like long-range dependencies.
  • The finding that GenPPL drops under sharpening in a pure autoregressive bigram generator implies that any few-step diffusion result reported primarily via GenPPL should be accompanied by diversity and entropy diagnostics to be interpretable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces an oracle evaluation framework for discrete diffusion samplers. A ground-truth first-order Markov chain is defined, its exact HMM posterior under masking observations is computed via forward-backward smoothing, and this posterior (or the derived concrete score) is substituted for the learned denoiser in SEDD, MDLM, LLaDA, and ReMDM. The resulting samplers are compared against the oracle transition kernel on OWT and Text8 with transition-level, surface, and external-LM metrics. The central claim is that, except for LLaDA, transition-level mismatch persists at small step counts and vanishes only as the number of steps approaches the sequence length, and that GenPPL/MAUVE/NLL do not reliably reflect this mismatch.

Significance. The framework is well suited to its stated purpose: the forward-backward derivation (App. A) and the concrete-score identity (App. B) are correct and standard, and replacing a learned denoiser with an exact posterior is a clean way to separate discretization/independence error from model approximation error. The GenPPL sharpening sanity check is an independent and convincing construction. The availability of code and the controlled comparisons across two datasets are strengths. If the empirical conclusions hold, the paper offers a useful caution that common surface metrics do not certify distributional correctness and that parallel finite-step samplers have intrinsic error. The principal limitations are the toy Markov-chain ground truth, the single-seed/no-error-bar experimental protocol, and the LLaDA exception, which together make the abstract's general claim too strong.

major comments (4)
  1. [Sec. 7.5 (and Sec. 4)] The headline claim treats LLaDA as evidence of sampler-induced error, but Sec. 7.5 concedes that LLaDA's degeneration under the oracle is caused by the substitution itself: the sampler "relies on denoiser-produced confidence scores that encode rich linguistic priors," and replacing them with the oracle posterior "breaks" that coupling. This is not the same type of failure as an incorrect update rule given an exact denoiser; it is an interaction between the oracle and a confidence heuristic designed for a different denoiser. The framework's claim in Sec. 4 that the oracle is used "in a method-consistent manner" therefore does not hold for LLaDA. Please either exclude LLaDA from the sampler-error claims or reframe it explicitly as a sensitivity/mis-specification diagnostic, and revise the abstract accordingly.
  2. [Sec. 6, Tables 1-2] All headline numbers are reported for a single seed (seed 123) and N=512 sequences, with no error bars, multiple seeds, or bootstrap intervals. The central empirical conclusion is a convergence pattern: e.g., on OWT, MDLM/SEDD/ReMDM transition KL at S=1024 is 0.2446-0.2564 versus an AR baseline of 0.2454. These differences are small enough that sampling noise could affect the claim that convergence occurs only near S=T. Please add seed variation or bootstrap confidence intervals for the main transition-level and metric comparisons, or explicitly state that the reported trends are single-seed observations.
  3. [Abstract / Sec. 7.1] The abstract states that transition-level mismatch "vanishes only as the number of steps approaches the sequence length," but Sec. 7.1 and Table 1 show that LLaDA does not converge at all: its KL remains around 1.3 at S=1024 while entropy and diversity keep collapsing. The abstract omits the "except LLaDA" qualifier that appears in the main text. The claim should be restated so that it applies to SEDD/MDLM/ReMDM, with LLaDA described as a separate non-converging case, or the abstract should be changed to match the actual evidence.
  4. [Sec. 6 / Sec. 3.1] The ground-truth distribution is a first-order Markov chain (a bigram prior with top-K truncation on OWT, dense bigram on Text8). The paper's general conclusion about "few-step discrete diffusion samplers" is therefore supported only for these low-order Markov priors, which are far from the dependencies found in natural language. Even if the exact-HMM oracle correctly isolates sampler error, the scope of the conclusion should be stated as "for the studied Markov priors," unless additional experiments with higher-order or neural-learned ground-truth distributions are provided.
minor comments (5)
  1. [Abstract] The abstract's contribution 2 should include the "except LLaDA" caveat that appears in the Figure 1 caption and Sec. 7.1; otherwise the summary is internally inconsistent.
  2. [Sec. 7.2 vs 7.4] The paper says MAUVE "more faithfully detects low-temperature degradation" in Sec. 7.2, but Sec. 7.4 says MAUVE is "largely insensitive to transition-level deviations." These statements are not contradictory if the type of distortion differs, but the conditions under which MAUVE is or is not sensitive should be stated explicitly.
  3. [Sec. 6 and App. C] The paper uses different samplers with different noise schedules, but the exact schedule α_t and its dependence on S are not fully specified in the main text. Without this, the phrase "number of steps approaches T" is hard to interpret across SEDD, MDLM, and ReMDM. Please include the schedule equations in the main text or reference the appendix more precisely.
  4. [App. E.4] The teleport term ε is described in Eq. (7) as part of the ground-truth kernel P′, but App. E.4 calls it a "numerical safeguard." Please state clearly whether all metrics are evaluated against the teleport-smoothed P′ or against the pure top-K kernel before mixing.
  5. [App. F.5] The phrase "LLaDA-pisamples" appears to be a typo for "LLaDA-π samples." Please fix the spacing/formatting.

Circularity Check

0 steps flagged

No significant circularity: the oracle is fixed from a ground-truth kernel before sampling, and the mismatch is measured, not fitted.

full rationale

The paper's central finding is an empirical measurement rather than a derived prediction. A ground-truth Markov kernel P' is fixed in Eq. (7); the HMM oracle posterior γi is computed by exact forward-backward smoothing (Eqs. 12-14); each sampler's learned denoiser is replaced by this fixed oracle; and generated transition statistics are then compared against the same P'. The target quantities (transition KL, NLL, entropy) are not used to fit any parameter of the oracle or the metrics, and the oracle is not derived from any sampler output. The observed convergence as S approaches T follows from the Markov property: with single-token sequential unmasking, the oracle posterior conditioned on previous tokens becomes exactly the transition kernel P', so the asymptotic agreement is a mathematical property of the construction, not a fitted agreement. The GenPPL/MAUVE demonstrations are independent sanity checks using a separate autoregressive bigram generator, so they do not reuse the fitted oracle. No load-bearing self-citation or imported uniqueness theorem appears; the HMM machinery and teleport construction are standard external tools. The only notable caveat is LLaDA, where Section 7.5 explicitly concedes that replacing the learned denoiser with the oracle breaks the confidence-score coupling, making that particular result an intervention artifact rather than a pure sampler-error measurement. This is a validity and external-generalization limitation, not a circularity: the paper does not use LLaDA's behavior to define or fit anything, and the limitation is stated in the paper itself. Thus no step of the derivation reduces to its own inputs.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The framework is an evaluation construction, not a new physical theory. No new entities are postulated. The free parameters are choices that define the synthetic ground truth and the sampler instantiations; none are fit to the measured outputs. The main unstated burden is that a first-order Markov chain stands in for natural language and that each sampler's logic is preserved under oracle substitution.

free parameters (6)
  • Sparsity level K = 206
    Chosen as the 90th percentile of per-state thresholds needed to capture 99% transition mass on OWT (Sec. E.3). Defines the ground-truth kernel P' and therefore all downstream KL/NLL numbers.
  • Teleport probability epsilon = 1e-4
    Chosen to keep P' strictly positive and oracle computations stable (Sec. E.4). Although small, it is part of the ground-truth transition kernel.
  • Temperature beta (SEDD perturbation)
    Hand-selected range of sharpening factors applied to oracle scores (Sec. 4.1, Sec. 6). Used to probe sampler sensitivity, not fitted to data.
  • ReMDM loop/conf hyperparameters = eta_cap=0.02, t_on=0.55, t_off=0.05, alpha=0.9, p=0.9
    Taken from Wang et al. (2025) and reused verbatim (Sec. 6). They are hand-chosen constants that affect the measured sampler error.
  • Sequence length T and sample count N = T=1024, N=512
    Evaluation settings fixed across all runs (Sec. 6). The 'vanishes only near T' trend is measured at these values; results could differ at other T/N.
  • MDLM unmasking schedule
    Described only as 'random or uniform selection' (Sec. 4.2); the exact schedule is a free choice not pinned down, which affects reproducibility.
axioms (6)
  • domain assumption Ground-truth data distribution is a first-order Markov chain (Eq. 6).
    The whole oracle derives from a bigram model. Real language has longer-range dependencies; the transfer of conclusions is an assumption.
  • domain assumption Forward noising is position-wise independent masking with schedule alpha_t (Eq. 1).
    Standard for SEDD/MDLM/LLaDA/ReMDM; the oracle posterior assumes this factorization.
  • standard math Forward-backward smoothing gives the exact posterior marginals under hard evidence (Eqs. 12-14).
    Standard HMM inference; the paper supplies the derivation in Appendix A.
  • standard math Concrete score reduces to a scaled posterior marginal under the absorbing channel (Eqs. 52, 59).
    Correct ratio calculation for SEDD's concrete score; derived in Appendix B.
  • domain assumption The samplers' implementations (tau-leaping, parallel unmasking, confidence remasking, ReMDM reverse posterior Eq. 62) faithfully match the original methods.
    If the instantiations are not faithful, the measured error is not the original sampler's error. Section 7.5 shows this assumption fails for LLaDA.
  • domain assumption GPT-2 Large is a representative external evaluator for GenPPL and MAUVE.
    Selected as in the dLLM literature; the sharpening result may not hold for another evaluator.

pith-pipeline@v1.3.0-alltime-deepseek · 26470 in / 19412 out tokens · 163568 ms · 2026-08-02T21:34:37.209132+00:00 · methodology

0 comments
read the original abstract

Discrete diffusion language models (dLLMs) provide a fast and flexible alternative to autoregressive models (ARMs) via iterative denoising with parallel updates. However, their evaluation is challenging: existing metrics conflate denoiser approximation error with sampler-induced error from the sampling dynamics, a problem that does not arise for ARMs whose autoregressive sampling exactly reflects the learned probability model. We introduce a sampler-centric oracle framework that replaces learned denoisers with an exact Hidden Markov Model posterior derived from a ground-truth Markov chain, isolating sampler-induced error in a controlled setting. We show that few-step discrete diffusion samplers are not distributionally correct even under an oracle denoiser, with transition-level mismatch that vanishes only as the number of steps approaches the sequence length. Moreover, improvements in negative log-likelihood (NLL), generative perplexity (GenPPL), or MAUVE do not imply correct sampling. Code is available at https://luhantang.github.io/dllm_sampler

Figures

Figures reproduced from arXiv: 2602.19619 by Greg Ver Steeg, Longxuan Yu, Luhan Tang, Shaorong Zhang.

Figure 1
Figure 1. Figure 1: Transition-level metrics on OpenWebText (OWT) under an oracle denoiser. We report transition KL, NLL, entropy rate, and n-gram diversity as functions of the number of sampling steps for several discrete diffusion samplers, using an exact oracle posterior to isolate sampler-induced error. For all samplers, substantial transition-level error persists at small step counts, with convergence to the autoregressi… view at source ↗
Figure 2
Figure 2. Figure 2: Temperature-induced score sharpening in SEDD under an oracle denoiser (OWT). As β increases, transition KL rises and 3-gram diversity falls, while sequence NLL and GenPPL decrease. MAUVE drops sharply at low temperatures, indicating severe distributional degradation. This reveals a misalignment between likelihood-based metrics and transition-level correctness. Duplication remains negligible except under ex… view at source ↗
Figure 3
Figure 3. Figure 3: GenPPL under controlled local sharpening in an autoregressive bigram generator (OWT). We introduce a sharp￾ening factor β (β = 1 corresponds to accurate sampling) and evaluate fixed samples using a pretrained GPT-2 Large model. As β increases, GenPPL decreases monotonically, while 3-gram di￾versity collapses and sentence entropy steadily declines, indicating increasing concentration of probability mass. Ex… view at source ↗
Figure 4
Figure 4. Figure 4: Step-wise evaluation of oracle ReMDM variants on OWT. Across transition-level metrics, the ReMDM-loop sampler exhibits larger deviations from the AR baseline than ReMDM-conf, indicating higher sampler error, while ReMDM-conf remains consistently closer to the oracle transition kernel. Applying nucleus sampling reduces transition KL, NLL, and entropy, indicating improved alignment with the oracle kernel, bu… view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of effective sparsity k ∗ i in OpenWebText (OWT) under the 99% cumulative mass criterion. The distribution is strongly right-skewed: most tokens require only a small number of successors, while a small fraction exhibit heavy-tailed behavior. The chosen global sparsity level K = 206 corresponds to the 90th percentile of this distribution. F. Additional Experiment Results F.1. Metric Outputs Tab… view at source ↗
Figure 6
Figure 6. Figure 6: Transition-level overview on Text8 (top row) and OpenWebText (bottom row). Despite differences in vocabulary size and tokenization granularity, both datasets exhibit the same qualitative trends: SEDD, MDLM, and ReMDM converge toward the AR baseline with increasing diffusion steps, whereas LLaDA shows persistent deviation and entropy collapse [PITH_FULL_IMAGE:figures/full_fig_p026_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Behavior of SEDD under large temperature scaling factors on Text8. As β increases beyond the moderate regime, the sampler exhibits qualitatively different behaviors. For intermediate values (e.g., β ≈10), transition KL and NLL become large at small diffusion steps and entropy and 3-gram diversity drop sharply, indicating near-deterministic but non-repeating transition dynamics. For larger values (β ≥ 20), … view at source ↗
Figure 8
Figure 8. Figure 8: Step-wise evaluation of oracle ReMDM variants on OpenWebText (OWT). We report transition-level metrics (KL, per-token NLL, entropy, support fraction), surface statistics (3-gram diversity, duplication rate), and external language-model metrics (MAUVE and GenPPL) across diffusion steps. ReMDM-conf remains consistently closer to the oracle transition kernel, while ReMDM-loop exhibits larger transition-level … view at source ↗
Figure 9
Figure 9. Figure 9: Transition-level evaluation of oracle LLaDA on Text8 (character-level). Results are shown for unconditional generation and for conditioning on a single initial token. Across all metrics, the two settings are nearly indistinguishable, indicating that conditioning on one token does not materially affect the sampling dynamics. This is expected, as a single observed token is rapidly forgotten by the sampling p… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 9 linked inside Pith

  1. [1]

    F., and Chen, L.-C

    Albrychiewicz, E., Valiente, A. F., and Chen, L.-C. Dy- namical regimes of multimodal diffusion models.arXiv preprint arXiv:2602.04780,

    arXiv

  2. [8]

    I., Cho, N

    Kang, W., Galim, K., Oh, S., Lee, M., Zeng, Y ., Zhang, S., Hooper, C., Hu, Y ., Koo, H. I., Cho, N. I., et al. Parallel- bench: Understanding the trade-offs of parallel decoding in diffusion llms.arXiv preprint arXiv:2510.04767,

    Pith/arXiv arXiv

  3. [9]

    Discrete diffu- sion models: Novel analysis and new sampler guarantees

    Liang, Y ., Liang, Y ., Lai, L., and Shroff, N. Discrete diffu- sion models: Novel analysis and new sampler guarantees. arXiv preprint arXiv:2509.16756,

    arXiv

  4. [12]

    URL https://openreview.net/forum? id=pD6TiCpyDR. Peng, F. Z., Bezemek, Z., Patel, S., Rector-Brooks, J., Yao, S., Bose, A. J., Tong, A., and Chatterjee, P. Path planning for masked diffusion model sampling.arXiv preprint arXiv:2502.03540,

    arXiv

  5. [14]

    S., and Kuleshov, V

    Wang, G., Schiff, Y ., Sahoo, S. S., and Kuleshov, V . Re- masking discrete diffusion models with inference-time scaling.arXiv preprint arXiv:2503.00307,

    arXiv

  6. [15]

    L., Juergens, D., Bennett, N

    Watson, J. L., Juergens, D., Bennett, N. R., Trippe, B. L., Yim, J., Eisenach, H. E., Ahern, W., Borst, A. J., Ragotte, R. J., Milles, L. F., et al. Broadly applicable and ac- curate protein design by integrating structure prediction networks and diffusion generative models.BioRxiv, pp. 2022–12,

    2022

  7. [16]

    Zheng, K., Chen, Y ., Mao, H., Liu, M.-Y ., Zhu, J., and Zhang, Q

    URL https: //openreview.net/forum?id=HDeIb67lJe. Zheng, K., Chen, Y ., Mao, H., Liu, M.-Y ., Zhu, J., and Zhang, Q. Masked diffusion models are secretly time- agnostic masked models and exploit inaccurate categori- cal sampling.arXiv preprint arXiv:2409.02908,

    Pith/arXiv arXiv

  8. [18]

    Neural mask predictor.In LLaDA, the denoiser (mask predictor) is trained to predict clean tokens at masked positions. In the unconditional setting, the network models pθ(x0,i |z t),(66) while in the conditional setting with a promptp 0, it models pθ(x0,i |p 0, zt),(67) where x0,i denotes the clean token at position i. During training, the cross-entropy lo...

    2024

  9. [206]

    Each truncated row is then renormalized to sum to one

    Only the top-Koutgoing transitions are retained for each state. Each truncated row is then renormalized to sum to one. This percentile-based rule prevents a small number of extremely heavy-tailed states from dominating the global sparsity choice while ensuring that the vast majority of states retain at least99%of their outgoing mass. E.4. Teleport Mixture...

    2012

  10. [1996]

    Mask-predict: Parallel decoding of conditional masked language models.arXiv preprint arXiv:1904.09324,

    Ghazvininejad, M., Levy, O., Liu, Y ., and Zettlemoyer, L. Mask-predict: Parallel decoding of conditional masked language models.arXiv preprint arXiv:1904.09324,

    Pith/arXiv arXiv 1904

  11. [2006]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

    1901

  12. [2012]

    Large language diffusion models.arXiv preprint arXiv:2502.09992,

    Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y ., Wen, J.-R., and Li, C. Large language diffusion models.arXiv preprint arXiv:2502.09992,

    Pith/arXiv arXiv

  13. [2015]

    I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al

    Team, G., Georgiev, P., Lei, V . I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context.arXiv preprint arXiv:2403.05530,

    Pith/arXiv arXiv

  14. [2019]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  15. [2022]

    Optimal inference sched- ules for masked diffusion models.arXiv preprint arXiv:2511.04647,

    Chen, S., Cong, K., and Li, J. Optimal inference sched- ules for masked diffusion models.arXiv preprint arXiv:2511.04647,

    arXiv

  16. [2023]

    inaccurate oracle SEDD

    define a continuous-time Markov jump process over discrete sequences with an absorbing [MASK] state. Let pt denote the marginal distribution of the noisy sequence at time t. The reverse-time dynamics are parameterized byconcrete score ratiosbetween discrete states, which determine the token-level jump rates. Under the absorbing noise model, the only nontr...

    2023

  17. [2024]

    The curious case of neural text degeneration.arXiv preprint arXiv:1904.09751,

    Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y . The curious case of neural text degeneration.arXiv preprint arXiv:1904.09751,

    Pith/arXiv arXiv 1904

  18. [2025]

    Discrete diffusion model- ing by estimating the ratios of the data distribution.arXiv preprint arXiv:2310.16834,

    Lou, A., Meng, C., and Ermon, S. Discrete diffusion model- ing by estimating the ratios of the data distribution.arXiv preprint arXiv:2310.16834,

    Pith/arXiv arXiv

  19. [2026]

    T., Yang, Z., Qi, Z., Han, J., Sahoo, S

    Arriola, M., Gokaslan, A., Chiu, J. T., Yang, Z., Qi, Z., Han, J., Sahoo, S. S., and Kuleshov, V . Block diffusion: Inter- polating between autoregressive and diffusion language models.arXiv preprint arXiv:2503.09573,

    Pith/arXiv arXiv