Pith. sign in

REVIEW 4 major objections 4 minor 18 cited by

Continuous diffusion fails on discrete data because of a vocabulary-size mismatch; a hybrid masking-plus-Gaussian kernel restores its benefits.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 08:02 UTC pith:ENXRTYYS

load-bearing objection Worth reading and worth refereeing: a genuinely new hybrid diffusion kernel with credible low-NFE and guidance results, though the training loss is borrowed rather than derived, leaving the continuous-score story on shakier ground than the paper claims. the 4 major comments →

arxiv 2510.22510 v3 pith:ENXRTYYS submitted 2025-10-26 cs.LG stat.ML

CANDI: Hybrid Discrete-Continuous Diffusion Models

Patrick Pynadath , Jiaxin Shi , Ruqi Zhang This is my paper
classification cs.LG stat.ML
keywords discrete diffusioncontinuous diffusiontoken identifiabilitytemporal dissonancemasked diffusionclassifier guidanceone-hot diffusiontext generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims to explain why continuous (Gaussian) diffusion underperforms pure discrete diffusion on text: Gaussian noise corrupts categorical tokens through two mechanisms that scale differently with vocabulary size. Discrete identity corruption — whether the argmax of the noisy latent picks the wrong token — grows rapidly with vocabulary size, while continuous rank degradation — how many wrong tokens outrank the correct one — is independent of vocabulary size. This creates a temporal dissonance: at noise levels where clean anchor tokens remain for learning conditional structure, the continuous denoising task is trivial; at noise levels where continuous denoising is informative, no identifiable tokens remain. To escape this, the paper proposes CANDI, a hybrid kernel that uses discrete masking to keep chosen positions clean while adding Gaussian noise to the rest, making both corruption rates linear in time and re-aligning the two learning signals. If correct, this unlocks continuous-diffusion advantages for discrete spaces: better text generation at low numbers of function evaluations, and classifier guidance with off-the-shelf classifiers via simple gradient addition.

Core claim

The paper introduces token identifiability along two axes — discrete identity corruption (the probability that the argmax of a noisy one-hot is the wrong token, equation 5) and continuous rank degradation (the expected fraction of incorrect tokens outranking the correct one, equation 6) — and shows that the former grows exponentially with vocabulary size while the latter does not. This temporal dissonance means that under ordinary Gaussian noise on one-hot vectors, the noise levels needed to learn a meaningful continuous score are exactly the levels at which the model has no clean positions to condition on. CANDI resolves this by decoupling the two corruption schedules: a masking process kee

What carries the argument

The central object is the structured noising kernel X_t = X_0 ⊙ M_t + X̃_t ⊙ (1 − M_t), where M_t is a Bernoulli masking vector with keep-rate α(t)=1−t and X̃_t follows a variance-exploding Gaussian diffusion with noise level σ(t) chosen so that rank degradation r*(t) is linear in time. This kernel decouples discrete identity corruption (controlled by the mask) from continuous rank degradation (controlled by σ(t)), eliminating the vocabulary-size dependence that plagues pure Gaussian noise. Inference combines an ancestral discrete update for newly unmasked positions with a probability-flow ODE step for the Gaussian component, and an approximate algorithm replaces expensive matrix multiplicat

Load-bearing premise

The reweighted cross-entropy loss taken from masked diffusion is assumed to correctly train the network to represent the conditional Gaussian score E[X0|Xt]; the paper does not derive an evidence lower bound for the hybrid corruption kernel, so if this training signal does not match the score, the reverse ODE update will be miscalibrated.

What would settle it

On a small synthetic vocabulary where P(X0|Xt) is computable in closed form (e.g., a Gaussian location mixture), compare the learned score from a CANDI-trained model against the analytic conditional expectation across noise levels; a systematic bias would indicate the misaligned training objective. Alternatively, replace the reweighted cross-entropy loss with an explicit score-matching loss for the Gaussian component and check whether the low-NFE generation frontier changes.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Pure continuous diffusion on one-hot or embedding spaces degrades sharply as vocabulary grows, because the noise levels that make the score informative leave no clean anchor tokens; CANDI's decoupled kernel removes this dependency.
  • Because CANDI keeps clean positions while iterating the rest through the probability-flow ODE, it can exploit continuous gradient information at low NFE, giving better perplexity–entropy frontiers than masked diffusion for NFE up to 64.
  • Classifier guidance reduces to adding a scaled gradient from a classifier trained only on clean data, with no diffusion-specific corruption-aware classifier, matching the frontier of specialized discrete guidance methods on molecule-property optimization.
  • The token-identifiability analysis explains why self-conditioning and random masking were empirically useful in prior continuous diffusion work: they act as substitute clean anchors.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is to apply CANDI-style position masking to embedding diffusion rather than one-hot space; the paper proves embedding diffusion suffers the same dissonance but does not implement the fix there, so the benefit of decoupling in that setting is an open empirical question.
  • The paper chooses linear schedules for both mask rate and rank degradation for simplicity; the optimal relationship between the two is left unexplored, and a data-dependent schedule could further improve the frontier.
  • The training objective in equation 13 is a reweighted cross-entropy loss inherited from masked diffusion, not a derived ELBO for the hybrid kernel; if this loss does not fully train the Gaussian score component, some of the observed gains could come from the masking structure alone, a hypothesis one could test by ablating the score-based ODE update.
  • The frontier-analysis evaluation method—sweeping temperature and comparing diversity–coherence trade-offs—is portable beyond this paper and could change how discrete diffusion models are compared in future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces token identifiability, a two-axis analytical framework for how Gaussian noise corrupts one-hot token representations, and identifies a 'temporal dissonance': discrete identity corruption ρ(t) scales with vocabulary size while continuous rank degradation r(t) does not. To address this, the authors propose CANDI, a hybrid discrete-continuous diffusion process that combines masking (preserving clean positions) with Gaussian noise on corrupted positions, with linear schedules for both corruption rates. The method is evaluated on text generation (Text8, OpenWebText) and molecular property guidance (QM9). The main empirical claims are that CANDI outperforms masked diffusion at low NFE on OpenWebText and enables classifier-based guidance with off-the-shelf clean-data classifiers on QM9. The paper also advocates frontier-based (entropy–perplexity) evaluation to avoid single-temperature comparison artifacts.

Significance. If the claims hold, the paper makes a valuable contribution: it provides a principled explanation for why continuous diffusion underperforms on discrete data at large vocabulary sizes, and offers a simple hybrid method that appears to combine the strengths of discrete and continuous diffusion. The token identifiability framework is elegant and the empirical results, especially the low-NFE gains and guidance with unmodified classifiers, are practically significant. The paper ships code and introduces a frontier-analysis evaluation methodology that could be widely useful. However, the theoretical status of the training objective and the reverse-time ODE is informal, which limits the certainty about why the method works; these gaps are load-bearing for the central claims.

major comments (4)
  1. [§5.2, Eq. (13)] The training objective is the reweighted cross-entropy loss from masked diffusion, adopted for the hybrid forward kernel without derivation. Unlike masked diffusion, the corrupted positions in Eq. (11) are not mask tokens but continuous Gaussian vectors, and the loss does not include a continuous-likelihood term. While cross-entropy over tokens should drive P_θ(X0|Xt) toward the true posterior, the paper does not show that the 1/(1−α(t)) weighting is appropriate for the hybrid kernel, nor does it provide an ELBO or other variational justification. Since the model output is used via Eq. (4) to estimate the score for the ODE update in Eq. (14), the claimed benefits of continuous geometry rest on this heuristic. Please derive the objective for the hybrid process or clearly frame it as a heuristic with an ablation demonstrating that the weighting choice does not materially affect the learned
  2. [§5.3, Eq. (14) and Algorithm 1] The reverse update for the continuous component applies the probability-flow ODE to corrupted positions while clean positions are held fixed, using a score estimated by a model that conditions on the full hybrid state. This is not the exact reverse of the hybrid forward kernel: the VE-SDE probability-flow ODE describes the marginal evolution of the Gaussian component, not the conditional evolution given clean anchors. The paper should state this as an approximation, derive the correct conditional score (or argue that the model's posterior mean yields it), and discuss the error. The label 'exact inference' for Algorithm 1 is therefore misleading.
  3. [§5.1, Eq. (10) and Appendix B.4] The target degradation function r*(t) is defined in terms of r_min and r_max, but these values are never specified anywhere in the paper. Similarly, the corruption bias interpolation λ in Appendix B.4 is left unspecified. These are free parameters that determine the noise schedule and the model's ability to distinguish clean from corrupted positions, and they are necessary for reproducibility. Please report the values used in all experiments and, ideally, a sensitivity analysis.
  4. [§6.2] The pure continuous one-hot diffusion baseline is described only as 'following the SDE defined in equation 9', but the specific noise schedule σ(t) used for training and sampling is not given (the appendix details only the embedding-diffusion schedule). Without this, the vocabulary-scaling experiment cannot be reproduced, and the comparison to CANDI is not fully specified.
minor comments (4)
  1. [Figure 5] The caption and text mention NFE∈{8,16,32,64}, but the figure axes show a different range. Please clarify the exact NFE values and the temperature settings for each panel; the current figure is hard to read.
  2. [§6.3] The TPS ratio is computed using GPT-2 and MDLM as representatives; the paper should state that this ratio is hardware- and implementation-dependent and should not be interpreted as a method-specific advantage.
  3. [Appendix A.1] The subsection titled 'Discrete Identity Corruption for t>0' introduces state-dependent ρ(t, X_τ) but the notation is confusing because ρ was previously defined as a function of t only. Consider using a different symbol or clarifying the conditional nature.
  4. [Related work] The concurrent works CADD and CCDD are discussed, but a comparison table or a more structured delineation of methodological differences would make the contribution clearer.

Circularity Check

0 steps flagged

No significant circularity. The ρ/r metrics are definitions; the CANDI schedule is a stated design choice; the main claims are empirical. The only self-citation is minor and non-load-bearing.

full rationale

Token identifiability is introduced as an explicit analytical framework: ρ(t) (Eq. 5) and r(t) (Eq. 6) are defined quantities whose scaling with vocabulary size is derived, not fitted. The temporal-dissonance claim follows from those definitions for a fixed σ(t), and the paper then tests it empirically by comparing continuous diffusion on Text8 vs OWT (Section 6.2), which is external, falsifiable evidence rather than a circular reduction. The CANDI kernel decouples corruption by construction: α(t)=1−t (Eq. 7) and σ(t) is obtained by inverting a user-specified linear r*(t) (Eq. 10), with the paper explicitly stating 'this linear schedule was selected for simplicity.' Thus the linear ρ–r relationship is a design choice, not a prediction derived from data. The training loss (Eq. 13) is adopted from masked-diffusion reweighting rather than derived from a new ELBO for the hybrid kernel; this is a validation gap or correctness caveat, not circularity, because the reverse ODE update (Eq. 14) and guidance rule (Eq. 16) do not presuppose the loss's optimality beyond the standard score parameterization (Eq. 4). The only self-citation is Pynadath et al. (2024) in Related Work, used as general inspiration for gradient-based discrete sampling; it is not load-bearing for the paper's central claims. Comparisons use official MDLM/DUO/UDLM checkpoints and reported hyperparameters, so the reported gains are benchmarked rather than forced by the paper's own equations.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim depends on a modeling assumption that clean anchor tokens are necessary for learning discrete dependencies, plus a heuristic training objective. The schedule parameters r_min, r_max and λ are not specified, which makes the exact configuration underdetermined.

free parameters (3)
  • r_min
    Lower bound of the target continuous rank degradation r*(t) in the hybrid kernel (Sec 5.1, Eq 10). Value not stated in the paper; it controls the minimum noise level and thus the start of the schedule.
  • r_max
    Upper bound of the target continuous rank degradation r*(t). Value not stated; together with r_min it determines σ(t) via inversion of Eq 6 and affects the noise schedule at high t.
  • lambda (corruption bias interpolation)
    Interpolation coefficient for the corruption bias embedding in Appendix B.4. Not specified; determines how corrupted positions are encoded and the model's ability to distinguish clean from corrupted positions.
axioms (4)
  • domain assumption Learning discrete conditional dependencies requires clean anchor tokens during training
    Stated in Sec 4.2 and used to justify the masking component of CANDI. Self-conditioning in prior work may also provide anchors, but the necessity of clean tokens is not proven.
  • ad hoc to paper The reweighted cross-entropy loss (Eq 13) is a valid training objective for the hybrid forward process
    Adopted from masked diffusion for 'stable training dynamics' without deriving the ELBO for the hybrid kernel. The continuous score is then estimated from this model via Eq 4.
  • domain assumption The reverse ODE update (Eq 14) for corrupted positions correctly simulates the reverse of the Gaussian component conditioned on clean positions
    Used in exact and approximate inference; not rigorously derived for the hybrid marginal, but plausible from the standard score identity.
  • domain assumption ρ(t) controls the learnability of conditional structure and r(t) controls the learnability of the score function
    Conceptual mapping in Sec 4.2; intuitive but not formally established.

pith-pipeline@v1.3.0-alltime-deepseek · 33184 in / 10898 out tokens · 116910 ms · 2026-08-04T08:02:59.611896+00:00 · methodology

0 comments
read the original abstract

While continuous diffusion has shown remarkable success in continuous domains such as image generation, its direct application to discrete data has underperformed pure discrete formulations. To understand this gap, we introduce token identifiability, an analytical framework characterizing how Gaussian noise corrupts discrete data through two mechanisms: discrete identity corruption and continuous rank degradation. We reveal that these mechanisms scale differently with vocabulary size, creating a temporal dissonance that forces a tradeoff between learning continuous geometry and discrete structure. To address this, we propose CANDI (Continuous ANd DIscrete diffusion), a hybrid framework that decouples discrete and continuous corruption, enabling simultaneous learning of both. This unlocks the benefits of continuous diffusion for discrete spaces: on controlled generation, CANDI enables classifier-based guidance with off-the-shelf classifiers through simple gradient addition; on text generation, CANDI outperforms masked diffusion at low NFE, demonstrating the value of learning continuous gradients for discrete spaces. We include the code on the project page: https://patrickpynadath1.github.io/candi-lander.

Figures

Figures reproduced from arXiv: 2510.22510 by Jiaxin Shi, Patrick Pynadath, Ruqi Zhang.

Figure 1
Figure 1. Figure 1: We provide a visual comparison of discrete, continuous, and hybrid diffusion. Discrete [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: We visualize the two forms of corruption for token identifiability and visualize the asym [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: We plot ρ(σ) and r(σ) for different numbers of categories to illustrate the divergence be￾tween discrete identity corruption and continuous rank degradation. At small |V |, both mechanisms change at roughly the same rate through the forward process, enabling the model to learn both con￾ditional structure and continuous denoising simultaneously. At larger values of |V |, even when half of the positions are … view at source ↗
Figure 4
Figure 4. Figure 4: We demonstrate that our hybrid kernel eliminates the temporal dissonance induced by a large number of categories by decoupling discrete and continuous corruption. While the Gaussian kernel (blue lines) requires complete discrete cor￾ruption ρ = 1 to achieve meaningful continuous degradation at large vocabulary size, our hybrid approach maintains a linear relationship between ρ and r by explicitly controlli… view at source ↗
Figure 5
Figure 5. Figure 5: We demonstrate that single-point evaluations for perplexity and entropy can lead to mis [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: We include the generative frontier results on Text8 and OWT for One-hot diffusion (pink), [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: We show the entropy-perplexity frontiers for NFE=8, 16, 32, 64. Each subplot includes [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: We compare the frontiers for MDLM, UDLM, and CANDI at NFE [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The empirical results against the analytical expressions for both discrete and continuous [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: We show the throughput for autoregressive models and discrete diffusion models, using [PITH_FULL_IMAGE:figures/full_fig_p027_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: While not achieving the best frontier, CANDI maintains competitive performance with [PITH_FULL_IMAGE:figures/full_fig_p027_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: We show achievable frontier for MDLM, DUO, and our method for both small and large [PITH_FULL_IMAGE:figures/full_fig_p028_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: We show that despite using a single step Monte Carlo estimate of the expectation, the [PITH_FULL_IMAGE:figures/full_fig_p028_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: We include the different frontiers for each method obtained by applying grid search to [PITH_FULL_IMAGE:figures/full_fig_p029_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Sample Generation from CANDI at NFE=16 with temperature [PITH_FULL_IMAGE:figures/full_fig_p030_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Sample Generation from CANDI at NFE=32 with temperature [PITH_FULL_IMAGE:figures/full_fig_p030_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Sample Generation from CANDI at NFE=64 with temperature [PITH_FULL_IMAGE:figures/full_fig_p031_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Sample Generation from CANDI at NFE=128 with temperature [PITH_FULL_IMAGE:figures/full_fig_p031_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Sample Generation from CANDI at NFE=256 with temperature [PITH_FULL_IMAGE:figures/full_fig_p032_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Sample Generation from CANDI at NFE=512 with temperature [PITH_FULL_IMAGE:figures/full_fig_p032_20.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Self-conditioned Flow Map Language Models via Fixed-point Flows

    cs.CL 2026-07 unverdicted novelty 7.0

    Self-conditioned flow language models solve fixed-point iterations, enabling fixed-point flow maps that distill into FMLM* which outperforms SOTA in few-step generation on OpenWebText.

  2. Masked Language Flow Models

    cs.CL 2026-06 unverdicted novelty 7.0

    MLFMs combine masking with continuous flows to scale flow-based language models to reasoning and instruction-following tasks on GSM8K and MT-Bench.

  3. Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics

    cs.CL 2026-06 accept novelty 7.0

    Naive samplers beat published diffusion and flow models on gen-PPL with incoherent output, proving the metric unsound and motivating distributional evaluation suites.

  4. Discrete Stochastic Localization for Non-autoregressive Generation

    cs.LG 2026-05 unverdicted novelty 7.0

    Discrete Stochastic Localization provides a continuous-state framework with SNR-invariant denoisers on unit-sphere embeddings, enabling one network to support multiple per-token noise paths and improving MAUVE on OpenWebText.

  5. Flow Map Language Models: One-step Language Modeling via Continuous Denoising

    cs.CL 2026-02 unverdicted novelty 7.0

    Continuous flow language models match discrete diffusion baselines and their distilled one-step flow map versions exceed 8-step discrete diffusion quality on LM1B and OWT.

  6. Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics

    cs.CL 2026-06 conditional novelty 6.0

    Zero-parameter naive samplers achieve state-of-the-art generative perplexity while producing incoherent text, proving the metric is unsound; distributional divergences like MAUVE and energy distance correctly rank the...

  7. DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling

    cs.LG 2026-05 unverdicted novelty 6.0

    DiLaDiff augments masked diffusion LMs with latent space modeling and consistency distillation to improve token correlation capture and inference speed.

  8. Continuous Diffusion Scales Competitively with Discrete Diffusion for Language

    cs.CL 2026-05 conditional novelty 6.0

    RePlaid achieves a 20x compute gap to autoregressive models, new SOTA PPL of 22.1 among continuous DLMs on OpenWebText, and competitive scaling laws by aligning architecture with modern discrete DLMs.

  9. Understanding and Accelerating the Training of Masked Diffusion Language Models

    cs.LG 2026-05 conditional novelty 6.0

    Bell-shaped time sampling accelerates masked diffusion language model training by roughly 4x on LM1B by countering locality bias in language data.

  10. Understanding and Accelerating the Training of Masked Diffusion Language Models

    cs.LG 2026-05 conditional novelty 6.0

    Bell-shaped time sampling — drawing the corruption level near t=0.5 — accelerates masked diffusion language model training by up to ~4× without changing the final loss.

  11. Discrete Stochastic Localization for Non-autoregressive Generation

    cs.LG 2026-05 unverdicted novelty 6.0

    DSL provides a continuous embedding framework where one denoiser supports a family of SNR paths for discrete sequences, improving MAUVE scores on OpenWebText and allowing random-order and hybrid sampling from a fine-t...

  12. Coupling Models for One-Step Discrete Generation

    cs.LG 2026-05 unverdicted novelty 6.0

    Coupling Models enable single-step discrete sequence generation via learned couplings to Gaussian latents and outperform prior one-step baselines on text perplexity, biological FBD, and image FID metrics.

  13. Token Time Continuous Diffusion for Language Modeling

    cs.CL 2026-05 conditional novelty 6.0

    A continuous diffusion language model where each token denoises at its own rate—sure tokens first—improves few-step generation over discrete samplers and roughly matches global-time continuous models.

  14. Generative Frontiers: Why Evaluation Matters for Diffusion Language Models

    cs.LG 2026-04 conditional novelty 6.0

    Generative perplexity and entropy are shown to be the two additive components of KL divergence to a reference distribution, motivating generative frontiers as a principled evaluation method for diffusion language models.

  15. MetaState: Persistent Working Memory Enhances Reasoning in Discrete Diffusion Language Models

    cs.CL 2026-03 conditional novelty 6.0

    Adding a 0.6%-parameter recurrent working memory to frozen discrete diffusion LLMs yields an average 4.5-percentage-point gain on GSM8K, MATH-500, HumanEval, and MBPP.

  16. Flow Map Language Models: One-step Language Modeling via Continuous Denoising

    cs.CL 2026-02 conditional novelty 6.0

    Continuous flows on token embeddings with flow-map distillation produce one-step language models whose quality exceeds recent 8-step discrete diffusion baselines on LM1B and OpenWebText.

  17. Diffusion Language Model for Recommendation

    cs.IR 2026-07 conditional novelty 5.0

    DLMRec adapts a masked discrete diffusion language model to recommendation via collaborative tokenization, curriculum masking, and stability-aware voting, outperforming autoregressive LLM baselines on LastFM, MovieLen...

  18. HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models

    cs.LG 2026-06 unverdicted novelty 5.0

    HybridCodec combines discrete tokens with continuous residuals via a focal modulation codec and hybrid Transformer to improve speaker retention and reduce autoregressive steps in speech language models.

Reference graph

Works this paper leans on

48 extracted references · 25 linked inside Pith · cited by 14 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Block diffusion: Interpolating between autoregressive and diffusion language models

    Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2503.09573

    Pith/arXiv arXiv 2025

  3. [3]

    Structured denoising diffusion models in discrete state-spaces

    Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34: 0 17981--17993, 2021

    2021

  4. [4]

    Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg

    Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. 0 (arXiv:2107.03006), February 2023. doi:10.48550/arXiv.2107.03006. URL http://arxiv.org/abs/2107.03006. arXiv:2107.03006

    Pith/arXiv arXiv doi:10.48550/arxiv.2107.03006 2023

  5. [5]

    Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. `. 0 (arXiv:2205.14987), October 2022. doi:10.48550/arXiv.2205.14987. URL http://arxiv.org/abs/2205.14987. arXiv:2205.14987

    Pith/arXiv arXiv doi:10.48550/arxiv.2205.14987 2022

  6. [6]

    Large language models to diffusion finetuning

    Edoardo Cetin, Tianyu Zhao, and Yujin Tang. Large language models to diffusion finetuning. arXiv preprint arXiv:2501.15781, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Analog bits: Generating discrete data using diffusion models with self-conditioning

    Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv:2208.04202, 2022

    Pith/arXiv arXiv 2022

  8. [8]

    Diffusion is spectral autoregression, 2024

    Sander Dieleman. Diffusion is spectral autoregression, 2024. URL https://sander.ai/2024/09/02/spectral-autoregression.html

    2024

  9. [9]

    Continuous diffusion for categorical data

    Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.15089, 2022

    Pith/arXiv arXiv 2022

  10. [10]

    Empowering diffusion models on the embedding space for text generation

    Zhujin Gao, Junliang Guo, Xu Tan, Yongxin Zhu, Fang Zhang, Jiang Bian, and Linli Xu. Empowering diffusion models on the embedding space for text generation. arXiv preprint arXiv:2212.09412, 2022

    Pith/arXiv arXiv 2022

  11. [11]

    Openwebtext corpus

    Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  12. [12]

    Oops i took a gradient: Scalable sampling for discrete distributions

    Will Grathwohl, Kevin Swersky, Milad Hashemi, David Duvenaud, and Chris Maddison. Oops i took a gradient: Scalable sampling for discrete distributions. In International Conference on Machine Learning, pp.\ 3831--3841. PMLR, 2021

    2021

  13. [13]

    Likelihood-based diffusion language models

    Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models. Advances in Neural Information Processing Systems, 36: 0 16693--16715, 2023

    2023

  14. [14]

    Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control

    Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. 0 (arXiv:2210.17432), June 2023. doi:10.48550/arXiv.2210.17432. URL http://arxiv.org/abs/2210.17432. arXiv:2210.17432 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2210.17432 2023

  15. [15]

    Argmax flows and multinomial diffusion: Learning categorical distributions

    Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forr \'e , and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in neural information processing systems, 34: 0 12454--12465, 2021

    2021

  16. [16]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35: 0 26565--26577, 2022

    2022

  17. [17]

    Variational diffusion models

    Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34: 0 21696--21707, 2021

    2021

  18. [18]

    Diffusion-lm improves controllable text generation

    Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-lm improves controllable text generation. Advances in neural information processing systems, 35: 0 4328--4343, 2022

    2022

  19. [19]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    Pith/arXiv arXiv 2017

  20. [20]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. 0 (arXiv:2310.16834), June 2024. doi:10.48550/arXiv.2310.16834. URL http://arxiv.org/abs/2310.16834. arXiv:2310.16834

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.16834 2024

  21. [21]

    text8 dataset

    Matt Mahoney. text8 dataset. http://mattmahoney.net/dc/textdata, 2011

    2011

  22. [22]

    Improving vector-quantized image modeling with latent consistency-matching diffusion

    Bac Nguyen, Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Stefano Ermon, and Yuki Mitsufuji. Improving vector-quantized image modeling with latent consistency-matching diffusion. arXiv preprint arXiv:2410.14758, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Scaling up masked diffusion models on text

    Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.18514, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 4195--4205, 2023

    2023

  25. [25]

    Gradient-based discrete sampling with automatic cyclical scheduling

    Patrick Pynadath, Riddhiman Bhattacharya, Arun Hariharan, and Ruqi Zhang. Gradient-based discrete sampling with automatic cyclical scheduling. Advances in Neural Information Processing Systems, 37: 0 46728--46763, 2024

    2024

  26. [26]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  27. [27]

    Chiu, Alexander Rush, and Volodymyr Kuleshov

    Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. 0 (arXiv:2406.07524), November 2024. doi:10.48550/arXiv.2406.07524. URL http://arxiv.org/abs/2406.07524. arXiv:2406.07524

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.07524 2024

  28. [28]

    The diffusion duality

    Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin T Chiu, and Volodymyr Kuleshov. The diffusion duality. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=9P9Y8FOSOk

    2025

  29. [29]

    Simple guidance mechanisms for discrete diffusion models

    Yair Schiff, Subham Sekhar Sahoo, Hao Phung, Guanghan Wang, Sam Boshar, Hugo Dalla-torre, Bernardo P de Almeida, Alexander Rush, Thomas Pierrot, and Volodymyr Kuleshov. Simple guidance mechanisms for discrete diffusion models. arXiv preprint arXiv:2412.10193, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and generalized masked diffusion for discrete data. 0 (arXiv:2406.04329), June 2024. doi:10.48550/arXiv.2406.04329. URL http://arxiv.org/abs/2406.04329. arXiv:2406.04329

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.04329 2024

  31. [31]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp.\ 2256--2265. pmlr, 2015

    2015

  32. [32]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020 a

    Pith/arXiv arXiv 2010

  33. [33]

    Generative modeling by estimating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019

    2019

  34. [34]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020 b

    Pith/arXiv arXiv 2011

  35. [35]

    Entropic time schedulers for generative diffusion models

    Dejan Stancevic, Florian Handke, and Luca Ambrogioni. Entropic time schedulers for generative diffusion models. arXiv preprint arXiv:2504.13612, 2025

    arXiv 2025

  36. [36]

    Self-conditioned embedding diffusion for text generation

    Robin Strudel, Corentin Tallec, Florent Altch \'e , Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, et al. Self-conditioned embedding diffusion for text generation. arXiv preprint arXiv:2211.04236, 2022

    Pith/arXiv arXiv 2022

  37. [37]

    Score-based continuous-time discrete diffusion models

    Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. 0 (arXiv:2211.16750), March 2023. doi:10.48550/arXiv.2211.16750. URL http://arxiv.org/abs/2211.16750. arXiv:2211.16750

    Pith/arXiv arXiv doi:10.48550/arxiv.2211.16750 2023

  38. [38]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  39. [39]

    Dream 7b: Diffusion large language models

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Seqdiffuseq: Text diffusion with encoder-decoder transformers

    Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. Seqdiffuseq: Text diffusion with encoder-decoder transformers. 0 (arXiv:2212.10325), May 2023. doi:10.48550/arXiv.2212.10325. URL http://arxiv.org/abs/2212.10325. arXiv:2212.10325 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2212.10325 2023

  41. [41]

    A langevin-like sampler for discrete distributions

    Ruqi Zhang, Xingchao Liu, and Qiang Liu. A langevin-like sampler for discrete distributions. In International Conference on Machine Learning, pp.\ 26375--26396. PMLR, 2022

    2022

  42. [42]

    Continuously augmented discrete diffusion model for categorical generative modeling

    Huangjie Zheng, Shansan Gong, Ruixiang Zhang, Tianrong Chen, Jiatao Gu, Mingyuan Zhou, Navdeep Jaitly, and Yizhe Zhang. Continuously augmented discrete diffusion model for categorical generative modeling. arXiv preprint arXiv:2510.01329, 2025

    arXiv 2025

  43. [43]

    Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling

    Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409.02908, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Coevolutionary continuous discrete diffusion: Make your diffusion language model a latent reasoner

    Cai Zhou, Chenxiao Yang, Yi Hu, Chenyu Wang, Chubin Zhang, Muhan Zhang, Lester Mackey, Tommi Jaakkola, Stephen Bates, and Dinghuai Zhang. Coevolutionary continuous discrete diffusion: Make your diffusion language model a latent reasoner. arXiv preprint arXiv:2510.03206, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  46. [46]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  47. [47]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  48. [48]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...