Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Score-entropy discrete diffusion can predict score vectors that no clean-token posterior induces; mean-to-score (M2S) predicts the posterior mean and maps it through the forward kernel, enforcing Bayes realizability and improving generation

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 07:37 UTC pith:HZEXDGYO

load-bearing objection A solid, clearly-written advance in discrete diffusion with a genuinely new theoretical idea; the empirical story is consistent, but the cleanest causal experiment underdetermines the paper's central claim. the 3 major comments →

arxiv 2607.21372 v1 pith:HZEXDGYO submitted 2026-07-23 cs.LG cs.AI

Mean-to-Score Discrete Diffusion: Posterior-Mean Denoisers for Score Entropy

Jingyuan Li , Xiaoyi Jiang , Yixuan Jiang , Wei Liu , Yi Zhu , Zuoqiang Shi , Pipi Hu This is my paper
classification cs.LG cs.AI MSC 60J2768T07
keywords discrete diffusionscore entropyBayes realizabilityposterior mean denoiserbridge polytopeuniform corruptioncontinuous-time Markov chaingenerative perplexity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Score Entropy Discrete Diffusion parameterizes reverse rates with positive score ratios, but positivity alone does not ensure that the full vector of scores at a noisy state is consistent with any single clean-token posterior. This paper proves that, under a mild support condition, the complete concrete-score vector is exactly the forward bridge B_t applied to the one-site clean-token posterior (Theorem 4.1), and introduces mean-to-score (M2S), which predicts that posterior and converts it to scores through the known linear map. The paper shows the score-entropy loss has the correct population optimum yet does not enforce this Bayes-realizability constraint away from it; in a trained pure-uniform SEDD checkpoint about one quarter of score vectors violate coordinate bounds and more than half are jointly invalid, which can create negative weights in finite-step sampling. Projecting raw scores onto the bridge polytope removes all observed negative weights and lowers generative PPL from 203.6 to 175.1, and M2S improves image and language generation across controlled comparisons. The central claim is that Bayes realizability is a structural requirement for score-based discrete diffusion, not a nicety.

Core claim

The paper's central claim is that a complete discrete score vector s⋆_i(x,t;y) is Bayes-realizable exactly when it equals B_t applied to the one-site clean-token posterior π⋆, where (B_t μ)_y = Σ_z μ(z) P_t^{(i)}(y|z)/P_t^{(i)}(x_i|z). Under Assumption A.1 this identity holds for the true score (Theorem 4.1), so predicting the posterior mean and applying the fixed linear map enforces the constraint by construction. The paper further shows that SEDD's score-entropy objective has the correct population optimum but does not enforce realizability away from it; that unconstrained positive scores can fall outside the coordinate box, and even inside it, outside the strictly smaller bridge polytope.

What carries the argument

The bridge map B_t: for a site i with current token x_i, (B_t μ)_y = Σ_z μ(z) P_t^{(i)}(y|z)/P_t^{(i)}(x_i|z), the expectation of the forward-kernel ratio against the one-site clean posterior. It carries the argument: Theorem 4.1 shows the true score is B_t π⋆, so constraining the network to outputs of B_t on the simplex enforces Bayes realizability. The support condition Assumption A.1 is the exact condition under which the one-site posterior determines all scores; the uniform kernel satisfies it, and the absorbing-mask case reduces the M2S objective to the MD4 loss.

Load-bearing premise

The bridge formula (Theorem 4.1) requires Assumption A.1: whenever the forward kernel can transition from a clean token z to the current noisy token with zero probability, it must also have zero probability to every candidate token y; otherwise the one-site posterior no longer determines the full score vector.

What would settle it

Take a trained pure-uniform discrete diffusion model, project every score vector onto the bridge polytope during sampling with a fixed random stream, and check whether any negative pre-normalization weight remains or whether generative PPL fails to drop; a checkpoint that still shows negative weights after projection would falsify the claim that projection removes all observed negative weights. Alternatively, construct a coordinate-wise forward kernel with asymmetric zero transitions that violates Assumption A.1 and check whether the bridge-applied one-site posterior reproduces the true concre

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • For any coordinate-wise CTMC satisfying the support condition, the one-site clean posterior determines all concrete scores via the bridge, so score parameterizations can be replaced by posterior-mean predictions without changing the population optimum.
  • SEDD's score entropy has the correct population optimum, but off-optimum scores can lie outside the bridge polytope; constraining scores to the polytope is necessary for Bayes realizability and removes negative sampler weights.
  • Projecting a fixed pure-uniform SEDD checkpoint's scores onto the bridge polytope lowered external generative PPL from 203.6 to 175.1 with the sampler and random stream unchanged.
  • In controlled image runs, M2S lowered CIFAR-10 test BPD from 3.173 to 3.129 and FID-50k from 42.83 to 28.09; MNIST FID dropped by more than 52 points under identical settings.
  • On OpenWebText at a 262B-token budget, M2S beat pure-uniform SEDD, GIDD, and Neural CTMC at all tested sampling budgets, reaching generative PPL 143.3 at 128 steps with Bayes sampling on a linear grid.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The bridge-polytope decomposition suggests a practical diagnostic: for any score-based discrete diffusion model, auditing how often predicted complete score vectors fall in C_t,k \ P_t,k may predict sampler-weight pathologies and sample-quality loss without retraining.
  • M2S is a reparameterization rather than a new objective; the same posterior-mean-to-score construction could be applied to other score matching losses (e.g., concrete or target-concrete score matching) for any kernel satisfying the support condition, potentially with similar realizability benefits.
  • The support condition ties the construction to kernels whose zero-transition structure is symmetric in the required sense; for kernels with asymmetric zeros, a one-site posterior may no longer pin down all scores, and a multi-site posterior or a modified bridge would be needed. This is testable by constructing such a kernel and checking whether the bridge-applied posterior reproduces the true conc
  • The fixed-checkpoint projection result implies that inference-time repair can improve sample quality even without retraining; a natural extension is to test whether the benefit grows with vocabulary size or with the measured joint-violation rate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper introduces 'mean-to-score' (M2S), a reparameterization of discrete diffusion reverse models in which the network outputs a per-site clean-token posterior μ_θ and converts it to concrete scores via the known forward kernel. Under the paper's support condition (Assumption A.1), Theorem 4.1 shows the true score vector is the image under this linear map of the one-site posterior; hence only 'Bayes-realizable' score vectors are reachable. Proposition 4.3 and Theorems 4.2/4.4 establish consistency of the score-entropy loss and posterior recovery under a rank condition; Theorem 4.6 shows the absorbing-mask case reduces to MD4. The empirical sections present a same-checkpoint intervention on a pure-uniform SEDD model (projecting scores onto the bridge polytope removes negative sampler weights and lowers GenPPL from 203.6 to 175.1), a controlled CIFAR-10 comparison (BPD 3.173→3.129, FID-50k 42.83→28.09), an MNIST comparison, and an OpenWebText study where M2S is reported to outperform pure-uniform and mixture baselines at all tested sampling budgets.

Significance. The theoretical core is clean and appears correct; the bridge-posterior equivalence and posterior-recovery results give a principled way to enforce a structural constraint (joint Bayes realizability) that is not captured by coordinate-wise positivity. The MD4 recovery and the explicit rank condition are useful reductions. The fixed-checkpoint projection experiment is a well-designed intervention in principle and goes beyond reporting aggregate metrics. The main risk is empirical attribution and reproducibility: the projection experiment lacks a control that separates joint realizability from generic contraction, and several headline numbers are single-run/best-seed point estimates with no code released. If the missing controls confirm the attribution, this would be a solid contribution to discrete diffusion.

major comments (3)
  1. [§5.1 / D.3, Eq. (11)-(13)] The projection experiment is the paper's strongest evidence for the causal claim that joint Bayes realizability, rather than mere coordinate-feasibility or generic shrinkage, improves sampling. As defined in Eq. (11)-(13), P_{t,k} ⊊ C_{t,k}, and projecting onto P_{t,k} changes scores in three ways at once: it enforces the coordinate box [ρ_t, ρ_t^{-1}], it bounds the total reverse rate, and it imposes joint realizability. The paper reports that 56.23% of SEDD score vectors lie in C_{t,k} \ P_{t,k} (Table 6). To attribute the GenPPL gain from 203.6 to 175.1 to joint realizability, the authors must run a control projecting only onto C_{t,k} (or onto another superset with matched coordinate/rate restrictions) at the same checkpoint. Without this control, the observed improvement could equally be explained by a generic contraction of the score toward the feasible set. This control is feasibl
  2. [§5.2, Tables 1, 2, 4; §C.2/C.3] The main quantitative claims are single-run point estimates or best-of-seed values. The CIFAR BPD and FID comparisons (3.129 vs 3.173; 28.09 vs 42.83), the MNIST FID table, and the OpenWebText GenPPL table do not report confidence intervals or seed variability; the text explicitly states that for OWT 'we report the best PPL across seeds.' Best-of-seed selection inflates the apparent advantage of a method and complicates the claim of outperforming baselines at every budget. The authors should report mean±std over at least three to five seeds for the key comparisons, or bootstrap intervals following their own Table 7 practice, and justify the best-seed convention if it is retained.
  3. [§5.2, Table 2; Abstract; Conclusion] The claim 'outperforms the evaluated pure-uniform SEDD, GIDD, and Neural CTMC checkpoints at every tested sampling budget' is supported only among checkpoints with a 262B-token budget; at 128 steps the table's best overall is SEDD (mask) at 127.2 with 682B tokens, and the text notes that M2S ranks second overall. The abstract and conclusion emphasize the superiority without prominently stating this token-budget asymmetry, so a reader may infer a claim the data do not support. The comparison should be framed strictly against matched 262B-token checkpoints unless a token-budget ablation is provided. Relatedly, the abstract still contains unresolved LaTeX placeholders for the CIFAR FID values.
minor comments (6)
  1. [Abstract] The unresolved macros '$\CifarSEDDFID$' and '$\CifarMtwoSFID$' appear in the abstract; replace with the numerical values 42.83 and 28.09 used in the body.
  2. [Table 2 / Figure 3(b)] The text reports that Bayes sampling on the linear grid achieves a 128-step GenPPL of 143.3, but Table 2's M2S row is labeled 'Bayes, cosine' and reports 148.8 at 128 steps. Please add the best configuration to the table or clearly state in the caption that Figure 3(b) contains the configuration omitted from the table.
  3. [Appendix D.2, Table 6] The sentence 'their union corresponds to C_{t,k} \ P_{t,k} under the computed-sign convention' and the parenthetical '1.2584%' are confusing. Spell out how the overlap in the partition is counted and why the boundary row is not double-counted.
  4. [Eq. (6)] The symbol B is used before being explicitly defined as a matrix; define B_{t,i} as a (K−1)×K matrix depending on the current token x_i, and state its dimensions and the indexing convention.
  5. [Appendix A, proof of Theorem 4.2] The proof invokes 'Theorem 3.6 of Lou et al.' but does not state its content. Including the statement of that theorem would make the ELBO derivation self-contained and easier to verify.
  6. [General / Reproducibility] No code or checkpoints are linked. Given the sensitivity of the paired/projection results to sampling details and random-number streams, releasing the checkpoint and sampler code would substantially strengthen reproducibility.

Circularity Check

0 steps flagged

No significant circularity: the bridge map, loss optimum, and MD4 equivalence are derived from the stated forward kernel and external results.

full rationale

The derivation chain is self-contained with respect to its own inputs. Theorem 4.1 derives the concrete-score identity from the forward kernel P_t and the law of total expectation; the M2S score in Eq. (6) is then defined as B_t applied to a predicted posterior, not fitted to any target metric. Proposition 4.3 proves the conditional score-entropy risk is uniquely minimized at the true score using the standard h(s,r) divergence, and Theorem 4.4 establishes posterior recovery under an explicit rank condition, with the uniform-kernel case verified directly rather than imported. The ELBO bound in Theorem 4.2 invokes Theorem 3.6 of Lou et al. (SEDD), an external, publicly available result, and the MD4 reduction in Theorem 4.6 is an algebraic equivalence with an external algorithm, not a self-citation. The same-checkpoint projection experiment compares raw SEDD scores with their Euclidean projection onto the bridge polytope under a fixed sampler and paired randomness; although the causal attribution of the gain to joint realizability rather than generic score contraction is an interpretive claim, it is an empirical underdetermination concern, not a reduction of the prediction to its inputs. The paper's own limitation statements in Appendix C.2 and D.3 explicitly disclaim that the theorem implies a universal FID ordering and that post-hoc projection equals M2S training. No fitted parameter is renamed as a prediction, and no load-bearing self-citation chain is present. Reference [14] is a same-author baseline but is not used to justify the M2S construction. Assumption A.1 is stated, and its satisfaction for the kernels used is verifiable from the kernel definitions, so it is not smuggled in by citation. Score 0 is appropriate.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

No constants are fitted to data in the derivation. The forward schedule (α_t=1−t or e^{−t}) is a modeling choice inherited from SEDD, not a parameter fitted to the reported metrics. The bridge polytope is a defined subset of score space, not an invented physical entity.

axioms (3)
  • domain assumption Assumption A.1 support condition: for every site i, candidate y, and clean token z in the site-i data support, P_t^{(i)}(x_i|z)=0 implies P_t^{(i)}(y|z)=0.
    Invoked in Theorem 4.1 to extend the posterior-sum to all z and to justify Eq. (5); it is stated explicitly in Appendix A and holds for the uniform and absorbing kernels used.
  • standard math Continuous-time ELBO for discrete diffusion (Lou et al. Theorem 3.6, building on Campbell et al.)
    Used in Theorem 4.2 to obtain the upper bound Eq. (9); treated as an external result, not re-derived.
  • domain assumption Forward kernel has the form P_t = α_t I + β_t 1π^T with α_t+β_t=1, α_0=1, α_T=0.
    This defines the corruption processes under study (uniform π=1/K, absorbing π=e_m) and is the basis for the closed-form bridge Eq. (7).

pith-pipeline@v1.3.0-alltime-deepseek · 24308 in / 12959 out tokens · 126909 ms · 2026-08-01T07:37:57.811783+00:00 · methodology

0 comments
read the original abstract

Score Entropy Discrete Diffusion (SEDD) parameterizes discrete reverse processes with unconstrained positive score ratios. While positivity guarantees nonnegative reverse jump rates, it does not ensure Bayes realizability: ratios at a noisy state need not be jointly induced by any clean-token posterior under the forward kernel. The score-entropy loss has the correct population optimum but does not enforce this constraint away from it. In a trained pure-uniform SEDD checkpoint, roughly one quarter of complete score vectors violate the coordinate box, while more than half lie inside it yet remain materially incompatible with any valid posterior. Such violations can produce negative pre-normalization weights in finite-step sampling. Projecting raw scores onto the bridge polytope removes all observed negative weights and improves external generative PPL from $203.6$ to $175.1$ without changing the sampler. We introduce \emph{mean-to-score} (M2S), which predicts a clean-token posterior mean and converts it to the score through an exact kernel-dependent linear map. The construction applies to any known coordinate-wise continuous-time Markov chain (CTMC) satisfying a mild support condition. For uniform corruption, it maps the probability simplex onto the bridge polytope; for absorbing-mask corruption, the resulting objective recovers MD4 exactly. In a controlled 28.4M-parameter CIFAR-10 comparison, M2S lowers test BPD from $3.173$ to $3.129$ and FID-50k from $\CifarSEDDFID$ to $\CifarMtwoSFID$. A 170M-parameter M2S model trained on about 262B OpenWebText token slots outperforms the evaluated pure-uniform SEDD, GIDD, and Neural CTMC checkpoints at every tested sampling budget, reaching generative PPL $143.3$ at 128 steps versus $183.6$ for the strongest pure-uniform baseline.

Figures

Figures reproduced from arXiv: 2607.21372 by Jingyuan Li, Pipi Hu, Wei Liu, Xiaoyi Jiang, Yixuan Jiang, Yi Zhu, Zuoqiang Shi.

Figure 1
Figure 1. Figure 1: M2S occupies the Bayes-realizable subset of the scalar envelope, whereas SEDD can output any positive score. In a K = 3 uniform toy, panels (a)–(b) vary one score along sb(c) = (cs⋆ y1 , s⋆ y2 ) around the shared optimum c = 1. The labels c− and c+ mark the limits imposed by the ρt-based coordinate envelope, while the inner dark-blue band is the smaller Bayes-realizable subset enforced by M2S and enlarged … view at source ↗
Figure 2
Figure 2. Figure 2: M2S enforces Bayes realizability and achieves lower final loss on both text and image generation. Panels (a)–(b) audit score realizability and sampler-weight validity. Panels (c)– (d) compare the score-entropy loss of M2S and SEDD on OpenWebText and CIFAR-10, respectively. 2. We derive M2S to enforce Bayes realizability. M2S maps a one-site clean-token posterior to all concrete scores through an exact kern… view at source ↗
Figure 3
Figure 3. Figure 3: (a) Generative-PPL comparison between M2S and the evaluated baselines. (b) Generative [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: shows the first 64 uncurated samples from the 200-step runs. M2S produces clearer strokes and fewer isolated bright pixels than SEDD, consistent with the FID comparison. (a) MNIST test data (b) M2S, FID 73.4 (c) SEDD, FID 126.2 [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: M2S converges to a lower CIFAR-10 training objective. Panel (a) includes every optimizer step in the 512M-image-presentation run; panel (b) enlarges the final 2,048 equivalent epochs. Faint curves show raw per-step losses (subsampled only for rendering), and bold curves show trailing 625-step means, a 51.2-equivalent-epoch window. Both axes use equivalent epoch rather than wall-clock time or dataloader pas… view at source ↗
Figure 6
Figure 6. Figure 6: M2S suppresses the isolated chromatic artifacts visible in SEDD under a fully paired sampler. Each panel contains generated indices 0–63 from its corresponding 50,000-image FID run, in index order, without ranking, filtering, or manual selection. Corresponding positions use the same seeded random-number stream, 256-step Euler discretization, and cosine grid. Connection to Bayes realizability. Theorem 4.1 s… view at source ↗
Figure 7
Figure 7. Figure 7: M2S training loss over 161.8k optimization steps. The gray curve shows raw loss recorded every 100 steps, the blue curve shows its trailing 5,000-step average, and the dashed line marks the end of warmup. GenPPL evaluation. For each method and sampling-step budget, we draw 1024 unconditional samples and score 512 tokens per sample with the same Gemma2-9B evaluator. For checkpoints with max_len=1024, we eva… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective

    cs.LG 2026-07 conditional novelty 6.5

    SEDD, MDLM/GIDD, M2S, and Neural CTMC share one reverse-rate KL loss, enabling direct GPT-2 adaptation to uniform discrete diffusion with a strong GenPPL–entropy frontier.

Reference graph

Works this paper leans on

32 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [1]

    Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg

    Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. InAdvances in Neural Information Processing Systems, volume 34, pages 17981–17993. Curran Associates, Inc., 2021. URLhttps://proceedings. neurips.cc/paper/2021/hash/958c530554f78bcd8e97125b70e6973d-Abstract.html

    2021

  2. [2]

    From denoising diffusions to denoising Markov models.Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(2):286–301, 2024

    Joe Benton, Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. From denoising diffusions to denoising Markov models.Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(2):286–301, 2024

    2024

  3. [3]

    Breckon, and Chris G

    Sam Bond-Taylor, Peter Hessey, Hiroshi Sasaki, Toby P. Breckon, and Chris G. Willcocks. Unleashing transformers: Parallel token prediction with discrete absorbing diffusion for fast high-resolution image generation from vector-quantized codes. InComputer Vision – ECCV 2022, volume 13683 ofLecture Notes in Computer Science, pages 170–188. Springer, 2022. d...

    work page doi:10.1007/978-3-031-20050-2_11 2022

  4. [4]

    A continuous time framework for discrete denoising models

    Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. InAdvances in Neural In- formation Processing Systems, volume 35, pages 28266–28279. Curran Associates, Inc., 2022. doi: 10.52202/068431-2049. URL https://proceedings.neurips.cc/paper_files/pa...

    work page doi:10.52202/068431-2049 2022

  5. [5]

    PixelSNAIL: An improved autore- gressive generative model

    Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. PixelSNAIL: An improved autore- gressive generative model. InProceedings of the 35th International Conference on Machine Learn- ing, volume 80 ofProceedings of Machine Learning Research, pages 864–872. PMLR, 2018. URL https://proceedings.mlr.press/v80/chen18h.html

    2018

  6. [6]

    Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

    Pith/arXiv arXiv 1904

  7. [7]

    Fast projection onto the simplex and theℓ1 ball.Mathematical Programming, 158(1–2): 575–585, 2016

    Laurent Condat. Fast projection onto the simplex and theℓ1 ball.Mathematical Programming, 158(1–2): 575–585, 2016. 10

    2016

  8. [8]

    Efficient projections onto the ℓ1-ball for learning in high dimensions

    John Duchi, Shai Shalev-Shwartz, Yoram Singer, and Tushar Chandra. Efficient projections onto the ℓ1-ball for learning in high dimensions. InProceedings of the 25th International Conference on Machine Learning, pages 272–279, New York, NY, USA, 2008. Association for Computing Machinery. doi: 10.1145/1390156.1390191. URLhttps://doi.org/10.1145/1390156.1390191

    arXiv 2008

  9. [9]

    OpenWebText corpus

    Aaron Gokaslan and Vanya Cohen. OpenWebText corpus. http://Skylion007.github.io/ OpenWebTextCorpus, 2019

    2019

  10. [10]

    Xing, and Alain Durmus

    Samson Gourevitch, Yazid Janati, Dario Shariatian, Umut Simsekli, Eric Moulines, Eric P. Xing, and Alain Durmus. Uniform diffusion models revisited: Leave-one-out denoiser and absorbing state reformulation.arXiv preprint arXiv:2605.22765, 2026

    Pith/arXiv arXiv 2026

  11. [11]

    Diffusion- BERT:Improvinggenerativemaskedlanguagemodelswithdiffusionmodels

    Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. Diffusion- BERT:Improvinggenerativemaskedlanguagemodelswithdiffusionmodels. InProceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4521–4534, Toronto, Canada, 2023. Association for Computational Linguist...

    doi:10.18653/v1/2023.acl-long.248 2023

  12. [12]

    Argmax flows and multinomial diffusion: Learning categorical distributions

    Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. InAdvances in Neural Information Processing Systems, volume 34, pages 12454–12465. Curran Asso- ciates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/hash/ 67d96d458abdef21792e6...

    2021

  13. [13]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  14. [14]

    Neural continuous-time Markov chain: Discrete diffusion via decoupled jump timing and direction.arXiv preprint arXiv:2604.15694, 2026

    Jingyuan Li, Xiaoyi Jiang, Fukang Wen, Wei Liu, Renqian Luo, Yi Zhu, Zuoqiang Shi, and Pipi Hu. Neural continuous-time Markov chain: Discrete diffusion via decoupled jump timing and direction.arXiv preprint arXiv:2604.15694, 2026

    Pith/arXiv arXiv 2026

  15. [15]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 32819–32848. PMLR, 2024. URL https://proceedings.mlr.press/v235/lou24a.html

    2024

  16. [16]

    Concrete score match- ing: Generalized score matching for discrete data

    Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score match- ing: Generalized score matching for discrete data. InAdvances in Neural Information Processing Systems, volume 35, pages 34532–34545. Curran Associates, Inc., 2022. doi: 10.52202/068431-2502. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/ df04a35d907e894d59d...

    work page doi:10.52202/068431-2502 2022

  17. [17]

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data

    Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. InInternational Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=sMyXP8Tanm

    2025

  18. [18]

    Image transformer

    Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. InProceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pages 4055–4064. PMLR, 2018. URL https://proceedings.mlr.press/v80/parmar18a.html

    2018

  19. [19]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195– 4205, 2023. URL https://openaccess.thecvf.com/content/ICCV2023/html/Peebles_Scalable_ Diffusion_Models_with_Transformers_ICCV_2023_paper.html

    2023

  20. [20]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019. 11

    2019

  21. [21]

    Chiu, Alexander Rush, and Volodymyr Kuleshov

    Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems, volume 37, pages 130136–130184. Curran Associates, Inc., 2024. doi: 10.52202/079017-4135. URLhttps://procee...

    doi:10.52202/079017-4135 2024

  22. [22]

    Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. PixelCNN++: Improving the PixelCNNwithdiscretizedlogisticmixturelikelihoodandothermodifications. InInternational Conference on Learning Representations, 2017. URLhttps://openreview.net/forum?id=BJrFC6ceg

    2017

  23. [23]

    Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simpli- fied and generalized masked diffusion for discrete data. InAdvances in Neural Information Processing Systems, volume 37, pages 103131–103167. Curran Associates, Inc., 2024. doi: 10.52202/079017-3277. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/ bad233b9849...

    doi:10.52202/079017-3277 2024

  24. [24]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InProceedings of the 32nd International Conference on Machine Learning, volume 37 ofProceedings of Machine Learning Research, pages 2256–2265. PMLR,

  25. [25]

    Score-based continuous-time discrete diffusion models

    Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. InInternational Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=BYWWwSY2G5s

    2023

  26. [26]

    Pixel recurrent neural networks

    Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 ofProceedings of Machine Learning Research, pages 1747–1756. PMLR, 2016. URLhttps://proceedings.mlr.press/ v48/oord16.html

    2016

  27. [27]

    Conditional image generation with PixelCNN decoders

    Aäron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with PixelCNN decoders. InAdvances in Neural Information Processing Systems, volume 29, pages 4790–4798. Curran Associates, Inc., 2016. URLhttps: //proceedings.neurips.cc/paper/2016/hash/b1301141feffabac455e1f90a7de2054-Abs...

    2016

  28. [28]

    Generalized interpolating discrete diffusion

    Dimitri von Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf, and Thomas Hofmann. Generalized interpolating discrete diffusion. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, pages 61810–61843. PMLR,

  29. [29]

    Susskind, and Navdeep Jaitly

    Ruixiang Zhang, Shuangfei Zhai, Yizhe Zhang, James Thornton, Zijing Ou, Joshua M. Susskind, and Navdeep Jaitly. Target concrete score matching: A holistic framework for discrete diffusion. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, pages 76716–76753. PMLR, 2025. URLhttps:...

    2025

  30. [30]

    fight” Russia’s aggression against ISIS with “love for neighbour

    Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. InConference on Language Modeling, 2024. URLhttps://openreview.net/forum? id=PEQFHRUFca. 12 Appendix Contents A Proofs 13 B Algorithms 18 B.1 Training Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B.2 ...

    2024

  31. [2015]

    URLhttps://proceedings.mlr.press/v37/sohl-dickstein15.html

  32. [2025]

    URLhttps://proceedings.mlr.press/v267/von-rutte25a.html