Pith. sign in

REVIEW 3 major objections 6 minor 18 references

Reducing Pretraining-Generation Mismatch in Diffusion Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Diffusion language models can recover a measurable part of their prompt-continuation gap by training with a clean prefix and a denoised suffix, without changing inference.

desk verdict A useful objective-level fix for dLLM prompt continuation, but the attribution to clean-prefix conditioning is confounded by the simultaneous addition of an AR prefix loss. read the letter →

arxiv 2608.09424 v1 pith:5D3SMP2A submitted 2026-08-10 cs.CL

Xiaocheng Lu , Huabin Liu , Song Guo , Jianguo Li This is my paper · ORCID
classification cs.CL
keywords diffusionlanguagemodelspromptcontinuationpretraining-generationmismatchprefix-conditionedblockmaskedmodelingwarmup-stable-decay
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Diffusion language models trail autoregressive models on prompt-continuation tasks partly, the paper argues, because pretraining corrupts prompt and continuation tokens together while evaluation hands the model a clean prompt. The paper proposes Prefix-Conditioned Diffusion (PCD), a continued-pretraining objective that keeps the left prefix clean and trains it with an autoregressive next-token loss while applying masked-diffusion denoising only to the unknown suffix. On matched continued-pretraining runs, PCD improves the native diffusion stable baseline by +2.56 points on the LLaDA2-Mini six-benchmark average and +4.86 points on the Qwen-1.7B mechanism comparison, with gains at every completed budget from 100B to 400B. These results support the claim that aligning the pretraining context distribution with prompt-conditioned generation can recover a measurable part of the diffusion language model's continuation gap without changing inference. The paper's variational view makes this precise: replacing corrupted-prefix context with the clean prefix removes a non-negative conditional-mutual-information term from the suffix-denoising objective.

What carries the argument

The load-bearing object is the hybrid attention/corruption mask of PCD: for a sampled prefix boundary $k$, prefix positions stay clean, use causal attention, and are trained with an AR next-token loss, while suffix positions are corrupted by the native diffusion mask and reconstructed by same-position MDM under the clean-prefix context. The effective mask $m^{\mathrm{pcd}}_i = s_i m_i$ and the corrupted input $\tilde{x}^{\mathrm{pcd}}_i = x_i$ for $a_i=1$ enforce that only suffix tokens can become denoising targets. Two orthogonal knobs control the recipe: intra-sample mixing (AR prefix plus MDM suffix inside one sequence) and inter-sample mixing (mixing PCD and native diffusion examples in a minibatch via $z \sim \mathrm{Bernoulli}(p_{\mathrm{pcd}})$). The variational identity $V^*_{\mathrm{nat}} - V^*_{\mathrm{pcd}} = I(Y; C \mid \tilde{C}, \tilde{Y}, M)$ formalizes the removed corrupted-prefix context shift.

What would settle it

Train a variant that keeps the prefix clean and denoises the suffix with the same no-shift MDM objective but removes the AR prefix loss (prefix tokens observed clean but not supervised), and compare Avg.6 against both native stable and PCD at the same budget. If the clean-prefix-only variant retains the gain, clean-prefix conditioning carries the effect; if it falls back to native level, the added AR prefix supervision is the operative ingredient.

Watch

Extended reading notes

Core claim

The central claim is that prompt-continuation performance of diffusion language models is limited by a training-interface mismatch, not by the absence of an autoregressive decoder. Native stable-phase denoising can corrupt the very prefix that prompt-continuation evaluation keeps clean; PCD changes the attention mask, corruption mask, and label construction so that each training sequence contains a clean, autoregressively supervised prefix and a no-shift masked-diffusion suffix. In controlled same-family comparisons, PCD stays above the native stable baseline at every completed budget, reaches +2.56 Avg.6 at the matched 400B LLaDA2-Mini stable stage, and remains +1.67 after matched decay continuation; Qwen-1.7B ablations attribute the strongest signal to intra-sample clean-prefix suffix denoising rather than to batch-level objective mixing.

Load-bearing premise

Every PCD variant trains the clean prefix with an added autoregressive next-token loss, and no experiment isolates a clean prefix without that AR loss, so the reported gains could in principle come from the extra AR supervision rather than from matching the evaluation context distribution.

Editorial extensions

If this is right

  • Every completed continued-pretraining budget from 100B to 400B shows PCD above the matched native stable baseline, with the gap growing to +2.56 Avg.6 at 400B on LLaDA2-Mini.
  • The benefit survives WSD decay continuation (+1.67 over the matched native decay row) and transfers to chat-SFT, where the PCD-derived checkpoint reaches 67.39 versus 66.53 for the best native-SFT run.
  • Intra-sample clean-prefix suffix denoising, not inter-sample mixing, is the primary alignment signal: the intra-only Qwen row gains +4.86 versus +4.70 for inter-only and +4.66 for mixed.
  • PCD is a training-objective change only; it requires no autoregressive decoder, verifier, or new inference mode, so it slots into existing WSD-style conversion pipelines.
  • The 12-benchmark stable-stage average improves on all 12 benchmarks by +2.48 points, indicating the effect is broad across QA, math, code, and reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because every PCD variant adds an AR next-token loss on the prefix, the paper's design does not isolate clean-prefix conditioning from the extra supervision; a control with a clean prefix but no AR prefix loss would determine whether the interface or the added loss drives the gain.
  • The 96.8% median prefix-ratio diagnostic suggests benchmark difficulty is dominated by long clean prompts, so gains may concentrate on tasks with longer prompts; a prompt-length-stratified evaluation would test this directly.
  • PCD's principle extends beyond block diffusion: leaving any prefix uncorrupted and denoising the rest with full-sequence diffusion is the same interface fix, and the same reasoning could be applied at SFT time rather than only during continued pretraining.
  • The optimal prefix-ratio range depends on backbone and budget in the Qwen sweep, so a validation-driven prefix schedule may beat any fixed range; the paper does not claim a universal optimum.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Prefix-Conditioned Diffusion (PCD), a continued-pretraining objective for diffusion language models that keeps prompt prefixes clean and autoregressively supervised while training the unknown suffix with a no-shift masked-diffusion (MDM) objective. The authors argue that native diffusion pretraining corrupts prompt and continuation tokens together, creating a mismatch with prompt-conditioned evaluation, and that PCD reduces this mismatch. They report matched-budget comparisons against same-family native dLLM stable baselines on LLaDA2-Mini and Qwen-1.7B, with a +2.56 Avg.6 gain at the 400B stable budget and a +4.86 gain in the Qwen mechanism comparison, plus additional results on decay continuation, chat-SFT transfer, prompt-prefix statistics, and intra-sample versus inter-sample objective mixing.

Significance. The matched-budget experimental design and consistent gains across budgets and backbones are strengths: the paper ships a clear controlled comparison on the same infrastructure, data, and evaluation harness, and it reports a falsifiable prediction. If the effect is genuinely due to clean-prefix conditioning, PCD would be a practical training-interface fix compatible with WSD-style conversion. The paper also provides a clean variational identity and useful prompt-statistics diagnostics. However, the central attribution is not yet established, because every PCD variant adds an autoregressive next-token loss on the prefix, and no control isolates the clean-prefix interface from this additional AR supervision. The variational argument in Eq. (4) addresses only the suffix-MDM term and omits the AR term, so it cannot rule out the alternative explanation that the gains come from adding AR supervision to continued pretraining. The phenomena are plausible and worth publishing, but the mechanism claim needs additional controls.

major comments (3)
  1. [§4.4, Figure 6, Eq. (8)] The paper attributes the reported gains to clean-prefix conditioning, but every PCD variant combines an AR prefix loss with no-shift suffix MDM, while the native baseline has neither. In Eq. (8), L_intra always contains the λar term; the intra-sample-only, inter-only, and mixed rows in Figure 6 all use this objective, and the inter-only row at ppcd=0.50 still uses PCD examples with their AR prefix loss. No row sets λar=0 while keeping clean-prefix suffix MDM, and no row adds an AR loss to the native corruption baseline. The variational identity in Eq. (4) compares only the suffix-MDM terms under clean versus corrupted prefixes and omits the AR term entirely, so it does not address this confound. Please add a clean-prefix-without-AR control (e.g., λar=0) and, ideally, a native-corruption-plus-AR control, and revisit the mechanism claims in light of those results.
  2. [§4.1, §4.3, Limitations] The headline numbers come from single training runs, and the paper provides no variance estimates. The matched-budget gains start at +0.99 Avg.6 at 100B and reach +2.56 at 400B; without seed variance or repeated evaluations, it is difficult to assess whether the consistent ordering is statistically meaningful or within run-to-run noise. The Limitations section acknowledges that most large-scale rows are single runs, but this issue is load-bearing for the claim that PCD consistently improves over the native baseline. Please provide seed variance at smaller scale (e.g., repeated Qwen-1.7B runs), and where full repeats are infeasible, report at least bootstrap confidence intervals over benchmark examples or multiple evaluation seeds.
  3. [§4.3, §5] The text states that PCD improves the native baseline on all 12 benchmarks, but Table 1 reports only the six Avg.6 benchmarks, and Figure 8 reports domain aggregates rather than per-benchmark scores. The breadth claim is not verifiable from the displayed data. Please include the full 12-benchmark table in the main text or appendix, with matched native-baseline scores for each benchmark, so that the 'all 12' claim can be checked.
minor comments (6)
  1. [§1] There is a missing space in 'interfacePrefix-Conditioned Diffusion'; it should read 'interface Prefix-Conditioned Diffusion'.
  2. [§3.3, Eq. (4)] The quantities V*_nat and V*_pcd are introduced without a formal definition; please define them explicitly (e.g., as conditional entropies or as optimal suffix-denoising losses over a model family) and state the independence assumptions needed for the Markov condition Y ⊥ C~ | (C, Y~, M).
  3. [§3.2 and §3.4] The text says 'We use λar = λmdm = 1 unless otherwise specified,' but no experiment varies these weights. Either add a λ sweep or remove the phrase, since the current wording implies an ablation that is not reported.
  4. [Table 1] The row label 'LLaDA2-MiniPCD stable' is missing a space; it should be 'LLaDA2-Mini PCD stable'.
  5. [Figure 3] The AR-decoding diagnostic is not described: please specify how PCD and native checkpoints are decoded autoregressively, since the models are block-diffusion checkpoints and the AR-decoding procedure is not obvious from the main text.
  6. [§4.2 and Figure 6] The Qwen mechanism suite is said to use six benchmarks, but the specific composition is not listed; please provide the benchmark list in the caption or table so readers can interpret the Qwen Avg.6 numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the PCD derivation is an exact identity, and the central empirical claim rests on matched external-benchmark comparisons rather than on a fitted parameter renamed as a prediction.

full rationale

The paper's formal derivation (Eq. 4) is V*_nat - V*_pcd = H(Y|Ctilde,Ytilde,M) - H(Y|C,Ytilde,M) = I(Y;C|Ctilde,Ytilde,M) >= 0, an identity from conditional mutual information under the stated Markov structure. The claim that PCD removes a corrupted-prefix context-shift term is therefore a transparent consequence of the definitions, not a hidden assumption. The paper does not use this identity to predict the magnitude of benchmark gains; it explicitly states, 'This does not by itself prove that PCD must improve accuracy' (Section 5), and instead relies on matched native dLLM stable baselines at the same budgets (Figure 4: +0.99 at 100B, +1.33 at 200B, +2.60 at 300B, +2.56 at 400B). The Qwen ablations likewise compare same-initialization, same-data, same-harness runs. No fitted parameter is renamed a prediction: prefix ratio and ppcd are design choices swept in Figure 9, not optimized against the test benchmarks. No load-bearing self-citation appears: the paper cites external prior systems such as LLaDA2, WSD, Dream, and block diffusion, and I find no invocation of the present authors' own prior work to force a conclusion. The one legitimate concern is a confound rather than circularity: every PCD variant adds an AR prefix loss alongside clean-prefix conditioning, and no control isolates the two. That threatens causal attribution, but it is openly acknowledged in the Limitations section (single runs, seed sensitivity), and no equation forces the reported gains. Thus the derivation is self-contained and the empirical claim is externally evaluated.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The ledger shows a modest number of hand-set hyperparameters (loss weights, prefix ratio range, inter-sample probability) and four domain assumptions. No new physical entities or ad hoc mathematical axioms are introduced. The main unstated assumption is not in the ledger but in the experimental interpretation: the gains are ascribed to clean-prefix conditioning despite the AR-loss confound.

free parameters (3)
  • lambda_ar and lambda_mdm = 1 (default)
    Loss weights for the AR prefix and MDM suffix in Eq 3 and Eq 8. Set equal by hand, not fitted.
  • prefix ratio distribution q(rho) = e.g., uniform over [0, 0.75] for the best Qwen configuration
    The prefix boundary k is sampled from q; the sweep in Figure 9 selects an intermediate range, so the reported gains use a distribution chosen from the same benchmarks.
  • inter-sample probability p_pcd = e.g., 0.5 for the mixed recipe
    Controls the fraction of PCD examples per minibatch (Eq 9-10). Selected via sweep; the best mixed configuration is used for LLaDA2.
assumptions (4)
  • standard math Standard masked diffusion training objective (Eq 1) as the base for native dLLM stable training.
    The paper defines the native diffusion loss in Eq 1 and uses it as the baseline objective.
  • domain assumption Conditional independence Y perpendicular to C_tilde given (C, Y_tilde, M) for the corruption process (Eq 4).
    The variational view in Eq 4 relies on this Markov structure to derive the mutual information gap. It assumes prefix and suffix corruption are independent and that suffix corruption is independent of the clean prefix given the corrupted suffix.
  • domain assumption Prompt-continuation benchmarks present long clean prefixes (aggregate 96.8% prefix ratio in the diagnostic).
    Section 5 uses a tokenizer-based diagnostic on five public benchmarks to support this. KorBench is omitted, and the corruption rate r=0.3 is illustrative.
  • domain assumption The WSD conversion recipe produces a stable block-diffusion interface during the stable phase.
    PCD is inserted after AR-to-BD warmup; the paper assumes this yields a stable base model for continued pretraining.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reducing Pretraining-Generation Mismatch in Diffusion Language Models." pith.science (2026). https://pith.science/paper/5D3SMP2A

@misc{pith2026260809424,
  author       = {Pith},
  title        = {Pith review of: Reducing Pretraining-Generation Mismatch in Diffusion Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5D3SMP2A}},
  note         = {Machine review of arXiv:2608.09424}
}
read the original abstract

Autoregressive language models align training and use: generation conditions on a clean prompt, and training predicts future tokens from clean left context. Diffusion language models offer parallel denoising, but native dLLM pretraining can randomly corrupt prompt and continuation tokens together, weakening the clean-prefix interface needed for prompt-conditioned generation. We identify this mismatch for prompt continuation and propose PCD (Prefix-Conditioned Diffusion), a pretraining objective that combines AR prefix supervision with no-shift suffix denoising. At the training-objective level, PCD changes the attention mask, corruption mask, and label construction in continued pretraining; it does not require an autoregressive decoder, verifier, or new inference mode. By supervising the clean-prefix side autoregressively and applying diffusion only to the unknown continuation, PCD makes the local training interface resemble how block-diffusion models are queried at evaluation time. We further separate intra-sample prefix conditioning from inter-sample objective mixing, allowing us to identify the local alignment signal separately from the optional batch-level mixing knob. Across LLaDA2-Mini and Qwen-1.7B backbones, PCD consistently improves over same-family native dLLM stable baselines, reaching a 4.2% relative gain on the main LLaDA2-Mini six-benchmark average (+2.56 points) and a 14.2% relative gain in the primary Qwen mechanism comparison (+4.86 points). These results suggest that aligning the pretraining context distribution with prompt-conditioned generation can recover a measurable part of the dLLM continuation gap without changing inference.

Figures

Figures reproduced from arXiv: 2608.09424 by the authors.

Figure 1
Figure 1. Motivation for Prefix-Conditioned Diffusion (PCD). Native diffusion pretraining can corrupt the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Prefix-conditioned diffusion training framework. PCD combines inter-sample objective mixing, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. AR-interface diagnostic under AR decod￾ing. Orange labels report Avg.6 gains over matched native stable checkpoints. 100B 200B 300B 400B +50B 59 60 61 62 63 64 65 +0.99 +1.33 +2.60 +2.56 +1.67 decay Continued-pretraining budget 6-bench average Matched budget gaps Native dLLM PCD [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Chat-SFT transfer on LLaDA2-Mini. HumanEval-FIM is excluded because chat tuning changes the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Qwen-1.7B mechanism ablation at 50B continued pretraining. Bars show Avg.6 gains over the same [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Benchmark prompt-prefix statistics. At the illustrative rate [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: 12-benchmark domain averages at the final [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 7 canonical work pages

  1. [3]

    URLhttps://arxiv.org/abs/2512.15745. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. InAdvances in Neural Information Processing Systems,

    arXiv

  2. [8]

    Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer

    URLhttps://arxiv.org/abs/2604.03677. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 6112–6121. Ass...

    arXiv 2019

  3. [9]

    URL https: //aclanthology.org/D19-1633/

    doi: 10.18653/v1/D19-1633. URL https: //aclanthology.org/D19-1633/. Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. DiffuSeq: Sequence to sequence text generation with diffusion models. InInternational Conference on Learning Representations,

    doi:10.18653/v1/d19-1633

  4. [10]

    Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling

    URL https://proceedings.neurips.cc/paper/2020/hash/ 4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html. Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. InAdvances in Neu- ral Information Processing Systems,

    work page 2020

  5. [11]

    Jason Lee, Elman Mansimov, and Kyunghyun Cho

    URL https://papers.neurips.cc/paper/2021/hash/ 67d96d458abdef21792e6d8e590244e7-Abstract.html. Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1173–1182. Association for Computationa...

    work page 2021

  6. [13]

    Aaron Lou, Chenlin Meng, and Stefano Ermon

    URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/ 1be5bc25d50895ee656b8c2d9eb89d6a-Abstract-Conference.html. Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. InProceedings of the 41st International Conference on Machine Learning,

    work page 2022

  7. [15]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J

    URL https://arxiv.org/abs/2502.09992. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67,

    arXiv

  8. [16]

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli

    URLhttps://arxiv.org/abs/2406.07524. Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InProceedings of the 32nd International Conference on Machine Learning, pp. 2256–2265. PMLR,

    arXiv

Show all 18 references
  1. [19]

    Fengqi Zhu, Xinyue Wen, and Yankai Lin

    URL https://arxiv.org/abs/ 2508.15487. Fengqi Zhu, Xinyue Wen, and Yankai Lin. LLaDA-MoE: Efficient mixture-of-experts diffusion language models.arXiv preprint arXiv:2509.24389,

    arXiv

  2. [20]

    URLhttps://arxiv.org/abs/2509.24389. 12

  3. [2017]

    Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma

    URL https://proceedings.neurips.cc/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html. Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma. Understanding warmup- stable-decay learning rates: A river valley loss landscape view. InInternationa...

    2017

  4. [2018]

    URLhttps://aclanthology.org/D18-1149/

    doi: 10.18653/v1/ D18-1149. URLhttps://aclanthology.org/D18-1149/. Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion-LM improves controllable text generation. InAdvances in Neural Information Pro- cessing Systems,

    doi

  5. [2019]

    URL https://aclanthology.org/N19-1423/

    doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423/. Yao Fu, Daniel Fathi, Gantavya Bhatt, Peiyuan Zhang, Yang Su, Senbin Zhu, Xianren Zeng, Hang Wang, Chao Xu, Shigeki Saito, Juan Carlos Niebles, et al. Nemotron-Labs-Diffusion: A tri-mode language model unifying...

    doi

  6. [2020]

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T

    URL https://proceedings.neurips.cc/paper/ 2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html. Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. MaskGIT: Masked generative image transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and ...

    2020

  7. [2022]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

    URL https://openaccess.thecvf.com/content/CVPR2022/html/Chang_MaskGIT_ Masked_Generative_Image_Transformer_CVPR_2022_paper.html. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. ...

    2019

  8. [2024]

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li

    URL https://arxiv.org/abs/2310.16834. Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models.arXiv preprint arXiv:2502.09992,

    arXiv

  9. [2025]

    Jacob Austin, Daniel D

    URL https://arxiv.org/ abs/2503.09573. Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. InAdvances in Neural Information Processing Systems,

    arXiv

  10. [2026]

    Yoshinari Fujinuma and Keisuke Sakaguchi

    URL https://research.nvidia.com/publication/2026-05_ nemotron-labs-diffusion-tri-mode-language-model-unifying-autoregressive. Yoshinari Fujinuma and Keisuke Sakaguchi. Unlocking prompt infilling capability for diffusion language models,

    2026

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.