Pith. sign in

REVIEW 3 major objections 7 minor 32 references

Four discrete-diffusion training losses are the same reverse-rate objective, so a pretrained autoregressive checkpoint can be adapted directly to uniform noise where every token stays editable.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 12:53 UTC pith:VI6DPXKK

load-bearing objection Solid reverse-rate unification and a real AR→uniform recipe; the abstract’s Pareto win is overstated against mismatched baselines. the 3 major comments →

arxiv 2607.24507 v1 pith:VI6DPXKK submitted 2026-07-27 cs.LG cs.AI

UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective

Xiaoyi Jiang , Jingyuan Li , Yixuan Jiang , Wei Liu , Yi Zhu , Zuoqiang Shi , Pipi Hu This is my paper
classification cs.LG cs.AI
keywords discrete diffusionautoregressive adaptationreverse ratesuniform noiseclean-token predictioncontinuous-time Markov chainslanguage modelsscore entropy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that several leading discrete diffusion language models are not optimizing different reverse processes. Once each method’s network output is rewritten as reverse jump rates, the conditional losses of SEDD, MDLM/GIDD, M2S, and Neural CTMC become one shared generalized KL objective. From that identity the authors derive exact conversions among clean-token, concrete-score, posterior-mean, and exit-rate heads, so a single clean-token interface can drive either mask or uniform corruption. They then continually pretrain GPT-2 checkpoints straight onto uniform diffusion—Unifusion—without a masked intermediate stage, annealing causal attention open and adding time conditioning. On 124M and 355M models the samplers improve both generative perplexity and unigram entropy as the step budget grows to 256; no same-scale model they evaluate beats them on both metrics at once, and they lead the compared diffusion models on WinoGrande, SIQA, and BBH.

Core claim

Under a shared continuous-time Markov forward process, the conditional losses of SEDD, MDLM/GIDD, M2S, and Neural CTMC are identical once expressed as a master Bregman divergence over model reverse rates. Clean-token predictions convert exactly into concrete-score, posterior-mean, and exit-rate/jump parameterizations, yielding a shared x0 interface that supports switching between absorbing-mask and uniform kernels and enabling direct autoregressive-to-uniform adaptation.

What carries the argument

The master reverse-rate objective L(θ): a time integral of pointwise Bregman divergences f(r,c)=r log(r/c)−r+c between true conditional reverse rates and model reverse rates. Theorems 3.3–3.5 show each published loss is this functional in different coordinates and give closed-form maps from an x0 prediction into the other three heads.

Load-bearing premise

That picking one rate-equivalent loss after a single shared training bake-off, and scoring samples with another model’s perplexity plus unigram entropy, is enough to rank adapted models even when baselines differ in data, size, and token budget.

What would settle it

From the same GPT-2 initialization, convert one clean-token head into the four native losses under matched data and batching; if reverse rates or GenPPL paths still diverge after conversion, or if any same-scale baseline simultaneously beats Unifusion’s reported 256-step GenPPL and entropy pair under the paper’s protocol, the central claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Autoregressive checkpoints can be reused for uniform diffusion without first training a masked diffusion model.
  • Prediction heads (concrete score, posterior mean, exit rate/jump) can be swapped without changing the reverse process being optimized.
  • Uniform corruption keeps every position editable at every sampling step, which the paper ties to a better fluency–diversity trade-off at higher step budgets.
  • The same x0 interface supports conversion in both directions between mask and uniform kernels.
  • At both tested scales, direct uniform adaptation yields the strongest WinoGrande, SIQA, and BBH scores among the compared diffusion models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If reverse-rate equivalence is general, many new discrete-diffusion objectives may be reparameterizations; reporting rate-space diagnostics beside native losses would make that testable.
  • Causal-to-bidirectional attention annealing may preserve as much AR knowledge as the objective unification; ablating anneal length would separate the two effects.
  • Mid-generation revision or controlled-edit benchmarks would test the claimed editability advantage of uniform over mask kernels more directly than unigram entropy alone.
  • Repeating the recipe above GPT-2 medium would show whether the GenPPL–entropy knee survives stronger autoregressive teachers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper makes two contributions. First, a theoretical unification: for a fixed forward CTMC, the conditional losses of SEDD, MDLM/GIDD, M2S, and Neural CTMC are shown to be the same generalized-KL (Bregman) objective over model reverse rates (Def. 3.1, Thm. 3.3, Cor. 3.4), with Proposition 3.2 bounding NLL by this objective up to a constant, and Theorem 3.5 giving exact conversions from a clean-token x0 prediction to the other three parameterizations. Second, a method: Unifusion shifts a pretrained GPT-2 AR softmax by one position to initialize an x0 predictor, then continual-pretrains under the GIDD loss with a linear uniform corruption kernel and causal-to-bidirectional attention annealing, yielding direct AR→uniform-noise diffusion adaptation. Experiments report a GenPPL/unigram-entropy frontier at 124M and 355M (97.783/5.2626 and 71.516/5.6669 at 256 steps), with no same-scale baseline beating Unifusion on both metrics simultaneously, plus best WinoGrande/SIQA/BBH accuracy among compared diffusion models at both scales.

Significance. The unification result is clean, useful, and well executed: Appendix A gives complete reductions of each published loss to the master rate divergence (including the boundary conventions for f(0,c) and the reachability restriction in Thm. 3.5), and Appendix B verifies formula-level identities to 1e-13 and code-level rate agreement of converted heads on a fixed checkpoint to 1e-11. The paper is commendably honest about scope: Cor. 3.4 is equivalence in rate space, not in finite-budget optimization, and Sec. 4.1 demonstrates the distinction empirically. The x0-interface observation (the shifted AR softmax is already a clean-token predictor, Fig. 1) is a simple but practically enabling insight for cross-kernel AR adaptation, and the direct AR→uniform path avoids the two-stage masked route used by prior conversions. Experiments use five seeds with mean±std. The empirical frontier claim, however, rests on a heterogeneous baseline set and a two-metric criterion whose second axis (within-sample unigram entropy) is a weak diversity proxy; the significance of the empirical half depends on resolving the attribution concern below.

major comments (3)
  1. [§4.3, Table 4 / C.3] The headline frontier claim ('no evaluated model at the same scale simultaneously outperforms Unifusion on both metrics') is confounded by initialization, data, and scale. Per Table 4 and Sec. C.3, Unifusion-S is the only small model that is both GPT2-initialized and FineWeb-trained: all uniform-kernel baselines (SEDD-uniform, GIDD variants, Neural CTMC, M2S, DOU) are ~170M models trained from scratch on OpenWebText with 131B–682B tokens, while Unifusion-S is 124M with 106B FineWeb tokens on top of GPT2. Two baselines already beat Unifusion-S's 256-step GenPPL (M2S 89.204, DOU 93.433 vs 97.783); the Pareto statement survives only on entropy gaps of 0.10–0.25 nats in a within-sample unigram statistic. Meanwhile Sec. 4.1 shows a GPT2-initialized GIDD run reaches min GenPPL 100.7 within 60B tokens, so AR initialization itself plausibly accounts for much of the frontier position. As written,
  2. [§4.1, Table 1] The choice of GIDD as Unifusion's native objective rests on a single 60B-token run per objective (Table 1), and the ranking may not be stable. The four runs reach their minima at different epochs (GIDD 1.92, SEDD 1.20, NCTMC 1.64, M2S 2.00), and the manuscript itself emphasizes that rate-equivalent losses differ in loss weighting and batch size — so the comparison conflates parameterization with optimizer hyperparameters that were not re-tuned per objective. Since this choice is load-bearing for all downstream results, the authors should either (i) report seed variability or a second budget point for the bake-off, or (ii) soften the conclusion from 'we therefore use GIDD' to a statement that this is a heuristic selection under a fixed shared recipe.
  3. [§3.2 vs. C.5 (Algorithm 1)] There is an apparent contradiction about time conditioning. §3.2 states 'we learn the t-conditioning parameters during adaptation,' but Algorithm 1 and the C.5 text state 'The implementation uses t in the corruption and loss, but does not pass an explicit time embedding to the Transformer.' If the x0 predictor never observes t, then xθ(·|zt,t) is a slight misnomer and the model must infer the noise level from zt statistics alone; this is a significant design detail that affects how readers interpret the rate-equivalence experiments (the four heads in Sec. 4.1 presumably share this property). Please reconcile the two statements and clarify exactly how t enters the network, if at all.
minor comments (7)
  1. [§4.3 / C.3] Unigram entropy differences of 0.10–0.25 nats (Table 4b) carry seed stds of ~0.005, so they are statistically real, but the metric is a within-sample unigram statistic on 512-token generations; it cannot distinguish lexical diversity from mild incoherence (cf. the DiffGPT-S 16-step sample, H=5.5106). Consider reporting a complementary diversity metric (e.g., distinct-n or MAUVE) or at least noting this limitation where the Pareto claim is made.
  2. [C.4, Table 5] UnifusionGemma (4.6B, Table 5) appears only in Appendix C.4 with no mention in the main text or abstract. Either introduce it in Sec. 4.4 or remove it; as is, it reads as an undeclared result.
  3. [§1, §4, D.1] Naming is inconsistent between DiffuGPT (Sec. 1, refs) and DiffGPT (Sec. 4, Table 2/4); likewise the sample section header uses lowercase 'm2s'. Please unify.
  4. [Table 2] Table 2 caption notes model sizes differ (124M vs 169–170M baselines); since several baselines are larger, a sentence acknowledging that Unifusion's leads come despite a parameter disadvantage would preempt misreading.
  5. [Figure 4] Figure 4's caption mentions panels (a) and (b) but the axis/annotation descriptions are incomplete in the extracted text; please ensure the figure is self-contained (inverted log axis, direction of 'better').
  6. [§4.3, Abstract] The entropy improvement for Unifusion-S across 16→256 steps (5.2461→5.2626, i.e., 0.017 nats) is described as entropy 'rising' and feeding the claim of a steadily improving trade-off; the improvement is essentially all in GenPPL. The wording in the abstract and §4.3 should be calibrated accordingly.
  7. [Prop. 3.2 / A.2] Prop. 3.2/Thm. A.2 numbering is confusing: the proposition is labeled 3.2 in the main text but appears as 'Theorem 3.2' in Appendix A.2, and the proof cites 'Theorem A.2' (a remark) for the endpoint argument. Please harmonize numbering.

Circularity Check

0 steps flagged

No significant circularity: reverse-rate equivalence is proved algebraically, and GenPPL/entropy wins are external measurements.

full rationale

The load-bearing theoretical chain (Def. 3.1 → Prop. 3.2 → Thm. 3.3–3.5 / Cor. 3.4) defines a master Bregman/KL objective over reverse rates, then derives that SEDD, MDLM/GIDD, M2S, and Neural CTMC conditional losses equal that objective once each head is mapped to off-diagonal rates, and that clean-token x0 logits convert exactly into concrete-score, posterior-mean, and exit-rate/jump coordinates. Appendix A supplies full proofs; Appendix B checks formula and checkpoint identities to ~1e-11–1e-13. Equivalence is scoped to rate space, not finite-budget trajectories (Sec. 4.1 explicitly shows native losses diverge under the same init). Self-citations to the authors’ Neural CTMC and M2S papers only name methods being unified alongside external SEDD/MDLM/GIDD; uniqueness is not imported from those citations. Empirical frontier and zero-shot numbers are GPT2-large GenPPL, sample unigram entropy, and task accuracy—external to the training loss—so they are not algebraic restatements of fitted inputs. Method choice of GIDD after a bake-off is ordinary model selection, not a prediction forced by construction. No step reduces a claimed first-principles result to its own definition or fit.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 2 invented entities

The theory rests on standard CTMC/Bregman machinery plus domain choices about discrete diffusion kernels and AR initialization. The empirical claim additionally depends on training hyperparameters, evaluation metrics, and baseline selection rather than on newly postulated physical entities.

free parameters (5)
  • Linear uniform schedule α_t=1-t, β_t=t = α_t=1-t after rescaling t∈(0,1)
    Corruption schedule is a design choice fixed by the authors; other schedules could change trainability and sampling quality.
  • Causal-to-bidirectional annealing horizon K_ann = 10,000 steps (GPT-2 S/M)
    Attention mask schedule (10k steps small/medium; 6k for Gemma experiment) is hand-chosen and load-bearing for stable AR transfer.
  • AdamW peak LR, warmup, batch, token budget = LR 3e-4; S ~240k steps / M ~255k steps
    Optimization recipe (3e-4 cosine, 3k warmup, global batch 864, ~106B/226B processed tokens) is tuned continual-pretraining configuration that affects reported minima.
  • GIDD loss-weight clip at 2.0 and dynamic weighting = clip 2.0
    Native GIDD weighting/clipping is retained after the bake-off and changes finite-budget trajectories versus other rate-equivalent losses.
  • Sampling temperature, step grid, GenPPL evaluator = T=1.0; steps ∈{16..256}; evaluator GPT-2-large
    Unconditional sampling at temperature 1.0 with GPT-2-large GenPPL and unigram entropy defines the headline Pareto claim.
axioms (6)
  • domain assumption Discrete corruption is a continuous-time Markov chain with well-defined forward rates and cumulative kernels (absorbing or uniform interpolating form).
    Section 2 builds all reverse rates and losses on this CTMC setup standard in discrete diffusion.
  • standard math Bregman divergence generated by φ(u)=u log u yields the master pointwise rate divergence f(r,c)=r log(r/c)-r+c with stated boundary conventions.
    Definition 3.1; standard information-geometry/KL calculus used to unify losses.
  • domain assumption Support/integrability conditions for the reverse model so L upper-bounds NLL up to a prior KL constant (Prop. 3.2).
    Likelihood bound requires prior support, positive model rates on true edges, and endpoint KL limits.
  • domain assumption After a one-position shift, a pretrained AR next-token softmax is a useful initialization for the diffusion clean-token predictor x_θ.
    Section 3.2 and Figure 1; central transfer assumption enabling Unifusion without training from scratch.
  • standard math Conditional and marginal rate objectives share gradients up to a θ-independent gap C_gap (Eqs. 5–6).
    Follows from Jensen on the Bregman divergence and Lemma A.1; used to treat reverse-rate matching as the learning target.
  • ad hoc to paper GenPPL–entropy Pareto dominance and selected zero-shot accuracies are adequate to claim a favorable uniform-diffusion adaptation.
    Evaluation protocol in §4.3–4.4; metric pair is author-chosen and drives the main empirical claim.
invented entities (2)
  • Master reverse-rate objective L(θ) / pointwise rate divergence D_t independent evidence
    purpose: Single functional that absorbs SEDD, GIDD/MDLM, M2S, and Neural CTMC conditional losses after reparameterization.
    Named organizational object for the unification; mathematically a standard Bregman/KL on rates rather than a new physical entity.
  • Unifusion (shared x0 interface + AR→uniform continual pretraining recipe) no independent evidence
    purpose: Practical method to initialize and fine-tune GPT-2 as uniform-noise discrete diffusion.
    Method bundle (shift, anneal attention, time conditioning, linear uniform schedule, GIDD loss); validated only by this paper’s runs unless externally reproduced.

pith-pipeline@v1.2.0-grok45-kimik3 · 39331 in / 4233 out tokens · 80275 ms · 2026-07-31T12:53:54.895320+00:00 · methodology

0 comments
read the original abstract

Existing methods mainly adapt pretrained autoregressive (AR) language models to masked diffusion, whereas we directly adapt them to uniform-noise diffusion, where every token remains editable during sampling. However, adapting AR checkpoints across corruption kernels remains challenging because existing DLMs use different objectives and prediction parameterizations. We establish connections among SEDD, MDLM/GIDD, M2S, and Neural CTMC by expressing their conditional losses as a single generalized Kullback--Leibler objective over model reverse rates. We further derive conversions from clean-token predictions to concrete-score, posterior-mean, and exit-rate/jump parameterizations, yielding a shared \(x_0\) interface that supports switching between mask and uniform kernels. Building on these connections, we propose \ours{}, a simple continual pre-training approach for directly adapting pretrained GPT2 checkpoints to uniform-noise diffusion. Through systematic evaluation of 124M- and 355M-parameter models, we show that \ours{} steadily improves the trade-off between generative perplexity (GenPPL) and unigram entropy as the sampling budget increases from 16 to 256 steps. At 256 steps, \ours{}-S and \ours{}-M achieve GenPPL/entropy pairs of \(97.783/5.2626\) and \(71.516/5.6669\), respectively; no evaluated model at the same scale simultaneously outperforms \ours{} on both metrics. At both scales, \ours{} also achieves the highest WinoGrande, SIQA, and BBH accuracy among the compared diffusion models.

Figures

Figures reproduced from arXiv: 2607.24507 by Jingyuan Li, Pipi Hu, Wei Liu, Xiaoyi Jiang, Yixuan Jiang, Yi Zhu, Zuoqiang Shi.

Figure 1
Figure 1. Figure 1: compares these two choices. We initialize two models from the same AR checkpoint (GPT2-small) and train them with the same uniform-kernel M2S loss (11), differing only in how the shifted AR output is used. The x0 model has 48.6% lower average loss during the first 2k updates and 27.9% lower average loss during updates 5k–10k, before the loss of the µ model catches up after roughly 10k updates. This compari… view at source ↗
Figure 2
Figure 2. Figure 2: Finite-budget training trajectories. GenPPL trajectories for the four rate-equivalent ob￾jectives [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Conversion paths. All four paths started from the same GPT2-small AR checkpoint and were optimized with GIDD under the same data budget and optimization settings. Full training and evaluation details are provided in Section C.2 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: (a) GenPPL–entropy frontier. (b) Selected samples. Across 16–256 steps, Unifusion-S improves monotonically from 143.944±0.872 to 97.783±1.901 GenPPL while its mean sample-level entropy rises from 5.2461±0.0022 to 5.2626±0.0041 nats. Its 64-step point already Pareto-dominates the 256-step endpoints of SEDD-uniform, Neural CTMC, and every evaluated GIDD variant. At 256 steps, Unifusion-S occupies a favorable… view at source ↗
Figure 5
Figure 5. Figure 5: Training-loss trajectories. We report zero-shot accuracy on HellaSwag [28], WinoGrande [20], SocialIQA [21], PIQA [2], and BBH [24]. For a fair comparison, all models use identical prompts, GPT2 tokenization, candi￾date construction, clean contexts, target-only corruption and scoring, a maximum sequence length of 512, and the same 32-point time grid tk = 1 − (k − 1)/32, with each candidate token selected i… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 6 linked inside Pith

  1. [1]

    Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg

    Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Struc- tured denoising diffusion models in discrete state-spaces. InAdvances in Neural Information Processing Systems, 34:17981–17993, 2021. 9

    2021

  2. [2]

    PIQA: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. InProceedings of the AAAI Conference on Artificial Intel- ligence, 34(5):7432–7439, 2020

    2020

  3. [3]

    A continuous time framework for discrete denoising models

    Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. InAdvances in Neural Information Processing Systems, 35:28266–28279, 2022

    2022

  4. [4]

    Train- ing verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Train- ing verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  5. [5]

    Discrete flow matching

    ItaiGat, TalRemez, NetaShaul, FelixKreuk, RickyT.Q.Chen, GabrielSynnaeve, YossiAdi, andYaron Lipman. Discrete flow matching. InAdvances in Neural Information Processing Systems, 37:133345– 133385, 2024

    2024

  6. [6]

    Masked diffusion models are secretly learned-order autoregressive models.arXiv preprint arXiv:2511.19152, 2025

    Prateek Garg, Bhavya Kohli, and Sunita Sarawagi. Masked diffusion models are secretly learned-order autoregressive models.arXiv preprint arXiv:2511.19152, 2025

    arXiv 2025

  7. [7]

    Scaling diffusion language models via adaptation from autoregressive models

    Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling diffusion language models via adaptation from autoregressive models. InInternational Conference on Learning Representations, 2025

    2025

  8. [8]

    Hashimoto

    Ishaan Gulrajani and Tatsunori B. Hashimoto. Likelihood-based diffusion language models. InAdvances in Neural Information Processing Systems, 36:16693–16715, 2023

    2023

  9. [9]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. InAdvances in Neural Information Processing Systems, 33:6840–6851, 2020

    2020

  10. [10]

    Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans

    Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. InInternational Conference on Learning Representations, 2022

    2022

  11. [11]

    Neural continuous-time Markov chain: Discrete diffusion via decoupled jump timing and direction

    Jingyuan Li, Xiaoyi Jiang, Fukang Wen, Wei Liu, Renqian Luo, Yi Zhu, Zuoqiang Shi, and Pipi Hu. Neural continuous-time Markov chain: Discrete diffusion via decoupled jump timing and direction. arXiv preprint arXiv:2604.15694, 2026

    Pith/arXiv arXiv 2026

  12. [12]

    Mean-to-score discrete diffusion: Posterior-mean denoisers for score entropy.arXiv preprint arXiv:2607.21372, 2026

    Jingyuan Li, Xiaoyi Jiang, Yixuan Jiang, Wei Liu, Yi Zhu, Zuoqiang Shi, and Pipi Hu. Mean-to-score discrete diffusion: Posterior-mean denoisers for score entropy.arXiv preprint arXiv:2607.21372, 2026

    Pith/arXiv arXiv 2026

  13. [13]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 32819–32848, 2024

    2024

  14. [14]

    Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data

    Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. InInternational Conference on Learning Representations, 2025

    2025

  16. [16]

    The FineWeb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. InAdvances in Neural Information Processing Systems, 37:30811–30849, 2024

    2024

  17. [17]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019. 10

    2019

  18. [18]

    Chiu, Alexander Rush, and Volodymyr Kuleshov

    Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. InAdvances in Neural Information Processing Systems, 37:130136–130184, 2024

    2024

  19. [19]

    Chiu, and Volodymyr Kuleshov

    Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin T. Chiu, and Volodymyr Kuleshov. The diffusion duality. InForty-second International Conference on Machine Learning, 2025

    2025

  20. [20]

    WinoGrande: Anadversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    KeisukeSakaguchi, RonanLeBras, ChandraBhagavatula, andYejinChoi. WinoGrande: Anadversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    2021

  21. [21]

    Social IQa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. InProceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4463–4473, 2019

    2019

  22. [22]

    Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and generalized masked diffusion for discrete data. InAdvances in Neural Information Processing Systems, 37:103131– 103167, 2024

    2024

  23. [23]

    Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole

    Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. InInternational Conference on Learning Representations, 2021

    2021

  24. [24]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging BIG-Bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261, 2022

    Pith/arXiv arXiv 2022

  25. [25]

    A deep and tractable density estimator

    Benigno Uria, Iain Murray, and Hugo Larochelle. A deep and tractable density estimator. InProceed- ings of the 31st International Conference on Machine Learning, volume 32 ofProceedings of Machine Learning Research, pages 467–475, 2014

    2014

  26. [26]

    Generalized interpolating discrete diffusion

    Dimitri von Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf, and Thomas Hof- mann. Generalized interpolating discrete diffusion. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, pages 61810–61843, 2025

    2025

  27. [27]

    Dream 7B: Diffusion large language models.arXiv preprint arXiv:2508.15487, 2025

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7B: Diffusion large language models.arXiv preprint arXiv:2508.15487, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    HellaSwag: Can a ma- chine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a ma- chine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019. 11 Appendix: contents A Proofs........................................................................................

    2019

  29. [29]

    2.fork= 1,...,Kdo x0∼p data, t∼Unif[10 −3,1), z t∼q t|0(·|x 0)

    Setθ←θ AR. 2.fork= 1,...,Kdo x0∼p data, t∼Unif[10 −3,1), z t∼q t|0(·|x 0). SampleA (k) and compute the shifted clean-token predictionxθ(·|z t,t). θ←AdamW(θ,∇ θLobj(xθ,zt,x 0,t)). 3.end for Return:θ. The implementation usestin the corruption and loss, but does not pass an explicit time embedding to the Transformer. Fort>s, Bayesian sampling uses qt|s(b|a) ...

  30. [30]

    8ton D"

    Sett n = (1−ε)(1−n/N)and drawz i t0∼π,i= 1,...,L. 2.forn= 0,...,N−1do Sett=t n,s=t n+1, and predictxθ(·|z t,t)with fully bidirectional attention. For each positioni, samplezi s∼ˆpi s|t(·|z i t). 3.end for Return:the generated sequencez 1:L 0 . C.6 Inference Speed We measure inference speed on a single NVIDIA H100 80GB GPU with batch size1, bf16 precision,...

    2010

  31. [31]

    - Results of liquids are applied into the blood so that sensors are sensed not with notable local overlay

    Laser Ever Crip Change Record - Save the Digital Way - Dr Ltderve shows the count of patients recorded with an oximeter. - Results of liquids are applied into the blood so that sensors are sensed not with notable local overlay. Put on your new device. It must come with a following service plan that medication you are receiving to take care of your patient...

    2014

  32. [32]

    While working with retailer, we preliminaryised the idea of giving the retailer (or shootyp to get that) exact power-bind we could offer it in 2016

    ( )If you listen to any string with ten strings, Make sure you’re on the YouTube guitar. Stank is a 2011 non-linux mine. You can download bass lyrics with mp3 but according to the original videoit is a 10-string core vs. guitar strings.)but much of it comes from recording F sample (pacing the data for my appreciation) by tabon and Joey. V8 chord scored or...

    2011