Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

This paper claims that conditioning a parallel decoder on Gumbel noise extracted from an autoregressive teacher converts the intractable joint-distribution matching problem into a supervised reconstruction task, and that this plug-in improv

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 17:39 UTC pith:24N2IJF6

load-bearing objection Gumbel Distillation is a genuinely new distillation recipe with consistent gains, but the headline numbers need a τ=0.85 ablation and error bars before they should be taken at face value. the 5 major comments →

arxiv 2603.22216 v2 pith:24N2IJF6 submitted 2026-03-23 cs.CL cs.LG

Gumbel Distillation for Parallel Text Generation

Chi Zhang , Xixi Hu , Bo Liu , Qiang Liu This is my paper
classification cs.CL cs.LG
keywords Gumbel-Max trickparallel decodingknowledge distillationnon-autoregressive language modelsmasked diffusionmulti-token predictionlatent conditioningtext generation quality
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Parallel (non-autoregressive) language models decode quickly but treat tokens in a block as conditionally independent, which degrades coherence. Gumbel Distillation uses the Gumbel-Max trick to turn an autoregressive teacher's stochastic sampling into a deterministic noise-to-token mapping, then trains the parallel student to reconstruct target tokens from that noise blueprint. This reframes the hard problem of matching the teacher's joint sequence distribution as a supervised learning problem, and it works as a plug-in for masked diffusion and multi-token-prediction architectures. On a large web-text corpus, the paper reports a 30.0% improvement in MAUVE and a 10.5% reduction in generative perplexity over a masked diffusion baseline, at unchanged parallel decoding speed.

Core claim

The central discovery is that the Gumbel-Max trick provides a deterministic link between a vector of standard Gumbel noises and the teacher's sampled token: once the logits are fixed, token = argmax(logits + noise). For any already-written text, the paper derives an algorithm to sample the posterior Gumbel noise that would have produced it, in a single parallel forward pass through the teacher, so every training pair (context, noise, target) encodes the teacher's joint decision. A student that learns the mapping p(x_block | context, noise_block) therefore learns the joint distribution of the block without enumerating the exponentially many possible combinations. The paper shows this conditio

What carries the argument

The Gumbel-Max trick as reparameterization: x = argmax_k(l_k + xi_k) with xi_k i.i.d. standard Gumbel. This makes categorical sampling deterministic given noise, so a teacher's output can be paired with a latent 'blueprint' noise vector. The load-bearing component is Algorithm 1, posterior Gumbel sampling, which constructs noise for all tokens of a sequence in parallel from one forward pass, preserving the argmax condition and (per the paper) standard Gumbel marginals. In the student, the noise is softmax-normalized and linearly projected into the token-embedding space, replacing mask embeddings or conditioning prediction heads.

Load-bearing premise

The method rests on the assumption that the posterior Gumbel noise extracted by Algorithm 1 for a fixed corpus token faithfully matches the distribution of noise an autoregressive teacher would have used to sample that token; the paper's appendix proof of that equivalence contains a mis-centered location for the maximum-score posterior (it states G(l_x,1) where the correct location is log sum_k exp(l_k)), so the guarantee as written does not go through.

What would settle it

For a small vocabulary with known logits, draw x from the softmax, sample the posterior noise by Algorithm 1 across many trials, and check two empirical distributions: the post-noise scores max_k(l_k + xi_k) should follow a Gumbel centered at log sum exp(l_k), and each xi_k should be standard Gumbel conditional on x. A mismatch in either would falsify the blueprint's fidelity; a broader test would compare acceptance/MAUVE of students trained on Algorithm-1 noise versus sequentially drawn teacher noise on identical prompts.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Parallel decoding can gain a share of autoregressive coherence without losing parallelism, since the Gumbel condition enters only at training time and through a small input projection.
  • With Gumbel conditioning, quality holds at fewer sampling steps: the reported curves show lower perplexity at a given number of function evaluations.
  • Multi-token prediction heads accept longer blocks; the relative acceptance gain grows at later heads (up to roughly 37.6% on a 7B-scale backbone), suggesting the conditioning supplies the missing sequential dependencies.
  • Zero-shot common-sense accuracy improves on several benchmarks, evidence that the teacher's knowledge transfers along with its fluency.
  • The noise distribution itself matters: ablations show Gaussian noise degrades performance and uniform noise collapses to low diversity, while Gumbel noise works.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: The same noise-as-blueprint idea could enable controllable generation—editing or steering the latent Gumbel vector should predictably steer content, a direction the paper names only as future work.
  • Inference: Because parallel extraction pairs clean corpus text with posterior noise while sequential extraction pairs teacher-generated text with prior noise, the observed advantage of parallel extraction suggests the student benefits most from high-quality ground-truth targets; this could be isolated by training on clean text with teacher-generated noise.
  • Inference: The paper's theoretical guarantee, as written, is not fully established: the appendix's posterior-maximum statement (M given the winning category is x ~ G(l_x, 1)) should be centered at log-sum-exp of the logits, so the winning-coordinate marginal proof needs correction. A reader should treat the method's empirical results as the primary evidence until the posterior proof is fixed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Gumbel Distillation, a distillation procedure for parallel text generators. Using the Gumbel-Max trick, an autoregressive teacher's sampling decisions are encoded as Gumbel noise; the parallel student is trained to reconstruct a block of target tokens conditioned on that noise. For offline corpora, Algorithm 1 samples from the posterior p(ξ|X=x) of the teacher's Gumbel-Max process in a single forward pass. The method is plugged into MDLM, BD3-LM, and Medusa. On LM1B and OpenWebText, the authors report large MAUVE and generative-perplexity gains over MDLM and BD3-LM, and higher Medusa acceptance rates. Ablations compare the method with token-level/sequence-level knowledge distillation and with adaptive parallel decoding, and a maze toy demonstrates the idea in a controlled setting.

Significance. The core idea is attractive and, if validated, practically useful: it converts a difficult joint-distribution matching problem into a supervised noise-conditioned learning problem, is architecture-agnostic, and incurs only a per-block projection overhead. Strengths include a concrete posterior-sampling algorithm, public code, an independent MAUVE evaluation against human text, and ablations showing that the specific Gumbel noise matters. However, the current empirical validation is confounded by an inference-time Gumbel temperature τ=0.85 that is applied only to the distilled models and is not ablated, and the written proof of the central theorem contains an erroneous duplicate formulation. The headline claim is therefore not yet established in the present form.

major comments (5)
  1. [Appendix C.1; Tables 1 and 4] All main diffusion results use inference-time Gumbel scaling ξ←τξ with τ=0.85, while training and the theory (Section 4.1, Algorithm 1) use ξ∼G(0,1). The paper reports no τ=1.0 ablation, and the baselines are not given an analogous calibrated inference parameter. Since the text states that this calibration brings substantial improvement in generative perplexity, the reported 30.0% MAUVE and 10.5% GenPPL gains are not cleanly attributable to the distillation mechanism. Please report τ=1.0 results and, ideally, apply a matched inference-time variance reduction to the baselines.
  2. [Appendix B (Eqs. 2 vs 4; B.2)] The theorem is stated twice with two different constructions. Eq. (2) gives ξ'_k=-log(exp(-ζ_k)/p_k + exp(-ζ0)) for losing k, whereas Algorithm 1 and Eq. (4) use -log(exp(-ζ_k)+p_k exp(-ζ0)); only the latter satisfies marginal preservation. The B.2 proof also asserts M|argmax=x∼G(l_x,1); the correct conditional location is logZ, and ξ_x|X=x is G(-log p_x,1), not G(0,1). The B.3-B.4 version correctly proves the unconditional marginal property. As printed, the appendix does not consistently establish Algorithm 1; please consolidate and correct it.
  3. [Section 5.1 and D.1, Table 1] GenPPL is computed with GPT-2-Large, the same model used as the distillation teacher. Thus the reported generative-perplexity improvement partly measures how closely the student imitates the evaluator, not an independent fluency measure. MAUVE, computed against human text, is independent and is the stronger evidence, but the paper should either add GenPPL under a different AR evaluator or explicitly de-emphasize the teacher-evaluated number in the headline.
  4. [Table 3] The Medusa acceptance-rate improvements are reported as point estimates over 500 samples, without confidence intervals or significance tests. The smallest gain (GPT-2-Small, Head 1: 0.445 vs 0.465) is a 0.020 absolute difference and may be sampling noise. Please provide error bars or a significance test, especially for the claim of monotonically increasing relative gains with head index.
  5. [Sections 4.1 and 5.3, Table 5] The paper states that parallel and sequential Gumbel extraction are theoretically equivalent, but no theorem establishes this. Theorem 4.1 only guarantees that Algorithm 1 samples from p(ξ|x). Equivalence of the resulting training distribution to the forward Gumbel-Max coupling requires the corpus to be drawn from the teacher distribution p*; for OpenWebText and LM1B this is an assumption and is not directly tested. The ablation in Table 5 shows parallel extraction works better empirically, but it does not close the theoretical gap. Please state the assumption explicitly and, if possible, test it (e.g., by comparing the marginal of ξ under extracted data to G(0,1)).
minor comments (5)
  1. [Figure 2 and Table 7] The caption numbers do not match the table: e.g., 'Success: 64%, NFE=3' for MDLM and 'Success: 94%, NFE=3' for Gumbel are not in Table 7, which reports NFE=2 and NFE=4.
  2. [Table 5] The checkmark layout is hard to read. Label the columns (Parallel, Sequential, Gumbel, Gaussian, Uniform) and mark entries explicitly rather than relying on row alignment.
  3. [Appendix E.2] The likelihood table is acknowledged as 'heavily biased' because the Gumbel-conditioned models see the exact Gumbel vectors for validation sequences. Such results should be presented as a diagnostic only, not as a comparative evaluation, and ideally moved out of the main appendix flow.
  4. [Appendix C.1] The phrase 'calibrated Gumbel distribution' is imprecise: if ξ∼G(0,1), then τξ∼G(0,τ), not a Gumbel with 'reduced variance' and the same location. Also, the paper should explain why the same calibration is not used during training.
  5. [General] Tables 1, 2, and 4 report no variance or number of seeds. Reporting standard errors or at least stating the number of seeds would improve reproducibility. There are also minor typos (e.g., 'Consdier' in Figure 7) and the duplicate theorem block in Appendix B should be removed after the correction described above.

Circularity Check

2 steps flagged

Central derivation is self-contained; minor evaluation-level circularities in GenPPL and zero-shot perplexity are flagged but not load-bearing.

specific steps
  1. other [Appendix D.1 / Section 5.1 (Table 1)]
    "our primary metric is Generative Perplexity (Gen. PPL), evaluated using a pre-trained GPT-2-Large model. ... For all distillation experiments, the autoregressive teacher is GPT-2-Large."

    The student is trained to imitate GPT-2-Large's conditional distributions via Gumbel distillation, and GenPPL is the negative log-likelihood under the same GPT-2-Large model. Thus the reported 10.5% GenPPL improvement partly measures how closely the student imitates the teacher/evaluator itself, rather than an external notion of fluency. This is a partial evaluation circularity, though MAUVE against human text is independent.

  2. other [Appendix E.2, Table 8]
    "Note that the results here are heavily biased towards models trained with Gumbel Distillation as they are conditioned on the Gumbel vectors corresponding to the ground truth sequences."

    The zero-shot validation perplexities are computed with the model conditioned on Gumbel vectors that are themselves derived from the ground-truth validation sequences via the teacher. The conditioning input therefore leaks the target tokens, making low perplexity partly an artifact of the evaluation protocol rather than a genuine prediction on unseen text. The paper explicitly disclaims this, and the table is not used in the main conclusions.

full rationale

The core derivation chain is not circular: the Gumbel-Max posterior sampler (Algorithm 1 / Theorem 4.1) is a standard mathematical construction with independent content, and training the student on posterior Gumbel noise while sampling prior Gumbel noise at inference is a well-defined latent-conditioned scheme, not an equivalence between input and output by definition. The main MAUVE results are computed against human text, and the zero-shot reasoning benchmarks are external, providing independent support for the headline claims. The circularity concerns that do exist are confined to evaluation protocols: GenPPL is measured with the same GPT-2-Large model used as distillation teacher, and Appendix E.2 openly states that its validation perplexities are conditioned on ground-truth-derived Gumbel vectors. These are genuine but partial, self-flagged issues; the inference-time tau=0.85 calibration is a confound (no tau=1.0 ablation) but belongs under correctness risk rather than circularity. Overall the paper's method is self-contained and does not reduce to its own fitted values or to a self-citation chain.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The central method depends on the Gumbel-Max reparameterization, a posterior-sampling procedure (Theorem 4.1), the corpus-as-teacher assumption for parallel extraction, and the assumption that posterior training noise transfers to prior noise at inference. Only one tunable scalar (τ) is introduced by the paper.

free parameters (1)
  • τ (inference Gumbel temperature) = 0.85
    Appendix C.1 introduces a calibrated Gumbel scale at inference and sets τ=0.85 for all Gumbel-distilled models; no equivalent tuning is reported for baselines.
axioms (4)
  • standard math Gumbel-Max trick: argmax_k(l_k + ξ_k) with ξ_k ∼ G(0,1) is an exact categorical sample from softmax(l).
    Section 3 uses this as the foundation for the noise-to-token mapping.
  • ad hoc to paper Algorithm 1 outputs a sample from the posterior p(ξ|X=x) of the Gumbel-Max process.
    Stated as Theorem 4.1. The appendix proof is not reliable as written: it asserts M|argmax=x ∼ G(lx,1), which is false in general (correct location is log Z); Eq (2) and Eq (4) of the theorem are inconsistent.
  • domain assumption The training corpus is treated as if drawn from the teacher's distribution p*.
    Section 4.1, Parallel Gumbel Extraction: 'assuming its data was drawn from the teacher's distribution p*'. GPT-2-Large was not used to generate LM1B/OWT.
  • domain assumption The student, trained on posterior noise, will work when conditioned on prior Gumbel noise at inference.
    Training conditions on posterior samples ξ|x; inference samples ξ ∼ G(0,1)^V. No explicit mechanism in the paper guarantees this transfer; the posterior sampling property alone does not.

pith-pipeline@v1.3.0-alltime-deepseek · 30019 in / 33106 out tokens · 318157 ms · 2026-08-02T17:39:21.757199+00:00 · methodology

0 comments
read the original abstract

The slow, sequential nature of autoregressive (AR) language models has driven the adoption of parallel decoding methods. However, these non-AR models often sacrifice generation quality as they struggle to model the complex joint distribution of token sequences. To narrow this performance gap, we introduce Gumbel Distillation, a novel distillation technique that enables parallel decoders to learn this distribution effectively. Our method leverages the Gumbel-Max trick to create a deterministic mapping from a latent Gumbel noise space to the output tokens of a high-performing AR teacher. As a model-agnostic technique, Gumbel Distillation seamlessly integrates with diverse parallel decoding architectures, including MDLM and BD3-LM. Experiments on LM1B and OpenWebText show that Gumbel Distillation substantially improves the generation quality of parallel language models, achieving a 30.0% improvement in MAUVE score and 10.5% in generative perplexity over MDLM trained on OpenWebText dataset. Code available at https://github.com/hxixixh/gumbel-distill.

Figures

Figures reproduced from arXiv: 2603.22216 by Bo Liu, Chi Zhang, Qiang Liu, Xixi Hu.

Figure 1
Figure 1. Figure 1: A conceptual overview of Gumbel Distillation. The distillation process consists of two steps: (1) Data Generation: An autoregressive teacher model’s sampling process is combined with Gumbel noise to deterministically generate pairs of token sequences and their corresponding Gumbel noise. In practice, one could alternatively extract {ξ} in parallel based on an offline training corpus (Section 4.1); (2) Stud… view at source ↗
Figure 2
Figure 2. Figure 2: A toy maze problem illustrating how Gumbel Distillation helps a student model learn a structured task. The goal is to generate a valid path from a start (<bos>) to a target (<eos>) using a simple vocabulary (up, down, left, right). The figure visualizes 100 generated paths from each model, slightly jittered for visualization. The baseline MDLM frequently fails, produc￾ing incoherent paths, indicating its d… view at source ↗
Figure 3
Figure 3. Figure 3: Generative Perplexity and MAUVE Score vs. Number of Function Evaluations (NFE) of MDLM on LM1B and OpenWebText. Our method (Gumbel Distillation) consistently outper￾forms the baselines, achieving lower perplexity for a given number of evaluations. MDLM MDLM + Gumbel Distillation BD3-LM BD3-LM + Gumbel Distillation [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Zero-shot performance on common-sense reasoning and question answering bench￾marks. Our Gumbel Distillation consistently improves accuracy over the baseline models across a suite of eight tasks. All scores are percentages (%). Performance vs. Number of Sampling Steps In [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Gumbel Distillation Integrated with MDLM. Left: original MDLM architecture, masked [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Gumbel Distillation integrated with Medusa. The base backbone model is frozen when [PITH_FULL_IMAGE:figures/full_fig_p024_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt used for LLM-based evaluation using Gemini-2.5-pro. [PITH_FULL_IMAGE:figures/full_fig_p025_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Sample from MDLM of length 1024 and T = 1024 steps. The generative perplexity of the [PITH_FULL_IMAGE:figures/full_fig_p027_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Sample from MDLM + Gumbel Distillation of length 1024 and T = 1024 steps. The [PITH_FULL_IMAGE:figures/full_fig_p028_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Sample from BD3-LM of length 1024 and T = 1024 steps. The generative perplexity of [PITH_FULL_IMAGE:figures/full_fig_p029_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Sample from BD3-LM + Gumbel Distillation of length 1024 and T = 1024 steps. The [PITH_FULL_IMAGE:figures/full_fig_p030_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Sample from AR of length 1024 and T = 1024 steps. The generative perplexity of the [PITH_FULL_IMAGE:figures/full_fig_p031_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. NAVIRA: Decoupled Stochastic Remasking for Masked Diffusion Language Models

    cs.CL 2026-06 unverdicted novelty 6.0

    NAVIRA decouples quality scoring from regeneration via stochastic remasking in masked diffusion LMs, improving fluency and LLM-judge scores on a 170M model.

Reference graph

Works this paper leans on

88 extracted references · 30 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Block diffusion: Interpolating between autoregressive and diffusion language models

    Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. In International Conference on Learning Representations, 2025

    2025

  2. [2]

    Structured denoising diffusion models in discrete state-spaces

    Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 2021

    2021

  3. [3]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020

    2020

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in neural information processing systems, volume 33, pp.\ 1877--1901, 2020

    1901

  5. [5]

    Medusa: Simple llm inference acceleration framework with multiple decoding heads

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads. In International Conference on Machine Learning, pp.\ 5209--5235. PMLR, 2024

    2024

  6. [6]

    A continuous time framework for discrete denoising models

    Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35: 0 28266--28279, 2022

    2022

  7. [7]

    One billion word benchmark for measuring progress in statistical language modeling

    Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. 2014

    2014

  8. [8]

    Diffusion forcing: Next-token prediction meets full-sequence diffusion

    Boyuan Chen, Diego Mart \' Mons \'o , Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 37: 0 24081--24125, 2024

    2024

  9. [9]

    Accelerating large language model decoding with speculative sampling

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    Analog bits: Generating discrete data using diffusion models with self-conditioning

    Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv:2208.04202, 2022

    Pith/arXiv arXiv 2022

  11. [11]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL-HLT, 2019

    2019

  12. [12]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  13. [13]

    A discourse-aware attention model for abstractive summarization of long documents

    Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Paper...

    2018

  14. [14]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Gemini diffusion

    DeepMind . Gemini diffusion . https://deepmind.google/models/gemini-diffusion/, 2025

    2025

  16. [16]

    Continuous diffusion for categorical data

    Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.15089, 2022

    Pith/arXiv arXiv 2022

  17. [17]

    Unifying autoregressive and diffusion-based sequence generation

    Nima Fathi, Torsten Scholak, and Pierre-Andr \'e No \"e l. Unifying autoregressive and diffusion-based sequence generation. arXiv preprint arXiv:2504.06416, 2025

    arXiv 2025

  18. [18]

    The language model evaluation harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The languag...

    arXiv 2024

  19. [19]

    Discrete flow matching

    Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. Advances in Neural Information Processing Systems, 37: 0 133345--133385, 2024

    2024

  20. [20]

    Mask-predict: Parallel decoding of conditional masked language models

    Marjan Ghazvininejad, Omer Levy, Yinhan Yin, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019

    2019

  21. [21]

    Better & faster large language models via multi-token prediction

    Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozi\` e re, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. In International Conference on Machine Learning, 2024

    2024

  22. [22]

    Openwebtext corpus

    Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  23. [23]

    Diffuseq: Sequence to sequence text generation with diffusion models

    Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. In The Eleventh International Conference on Learning Representations, 2023

    2023

  24. [24]

    Scaling diffusion language models via adaptation from autoregressive models

    Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusion language models via adaptation from autoregressive models. In International Conference on Learning Representations, 2025

    2025

  25. [25]

    Non-autoregressive neural machine translation

    Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoregressive neural machine translation. arXiv preprint arXiv:1711.02281, 2017

    Pith/arXiv arXiv 2017

  26. [26]

    Levenshtein transformer

    Jiatao Gu, Chang Wang, and Jianbo Zhao. Levenshtein transformer. In Advances in Neural Information Processing Systems (NeurIPS), 2019

    2019

  27. [27]

    Minillm: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    E.J. Gumbel. Statistical Theory of Extreme Values and Some Practical Applications: A Series of Lectures. Applied mathematics series. U.S. Government Printing Office, 1954. URL https://books.google.com/books?id=SNpJAAAAMAAJ

    1954

  29. [29]

    Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control

    Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. arXiv preprint arXiv:2210.17432, 2022

    Pith/arXiv arXiv 2022

  30. [30]

    Diffusionbert: Improving generative masked language models with diffusion models

    Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. Diffusionbert: Improving generative masked language models with diffusion models. arXiv preprint arXiv:2211.15029, 2022

    Pith/arXiv arXiv 2022

  31. [31]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  32. [32]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 2020

    2020

  33. [33]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020

    2020

  34. [34]

    Argmax flows and multinomial diffusion: Learning categorical distributions

    Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forr \'e , and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in neural information processing systems, 34: 0 12454--12465, 2021

    2021

  35. [35]

    Accelerating diffusion llms via adaptive parallel decoding

    Daniel Israel, Guy Van den Broeck, and Aditya Grover. Accelerating diffusion llms via adaptive parallel decoding. arXiv preprint arXiv:2506.00413, 2025

    arXiv 2025

  36. [36]

    Any-order flexible length masked diffusion

    Jaeyeon Kim, Lee Cheuk-Kit, Carles Domingo-Enrich, Yilun Du, Sham Kakade, Timothy Ngotiaoco, Sitan Chen, and Michael Albergo. Any-order flexible length masked diffusion. arXiv preprint arXiv:2509.01025, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016

    2016

  38. [38]

    Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement

    Wouter Kool, Herke van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), International Conference on Machine Learning, 2019

    2019

  39. [39]

    Cllms: Consistency large language models

    Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. Cllms: Consistency large language models. In Forty-first International Conference on Machine Learning, 2024

    2024

  40. [40]

    Mercury: Ultra-fast language models based on diffusion

    Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, et al. Mercury: Ultra-fast language models based on diffusion. arXiv preprint arXiv:2506.17298, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, 2023

    2023

  42. [42]

    Diffusion-lm improves controllable text generation

    Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion-lm improves controllable text generation. In Advances in Neural Information Processing Systems, 2022

    2022

  43. [43]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    Pith/arXiv arXiv 2022

  44. [44]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024 a

    Pith/arXiv arXiv 2024

  45. [45]

    Discrete copula diffusion

    Anji Liu, Oliver Broadrick, Mathias Niepert, and Guy Van den Broeck. Discrete copula diffusion. arXiv preprint arXiv:2410.01949, 2024 b

    Pith/arXiv arXiv 2024

  46. [46]

    Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals

    Lang Liu, Krishna Pillutla, Sean Welleck, Sewoong Oh, Yejin Choi, and Zaid Harchaoui. Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals . In NeurIPS, 2021

    2021

  47. [47]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022

    Pith/arXiv arXiv 2022

  48. [48]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning, 2024

    2024

  49. [49]

    Maddison, Daniel Tarlow, and Tom Minka

    Chris J. Maddison, Daniel Tarlow, and Tom Minka. A* sampling. In Advances in Neural Information Processing Systems. MIT Press, 2014

    2014

  50. [50]

    Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz

    Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of E nglish: The P enn T reebank. Computational Linguistics, 19 0 (2): 0 313--330, 1993. URL https://aclanthology.org/J93-2004/

    1993

  51. [51]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    Pith/arXiv arXiv 2016

  52. [52]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the Annual Conference on Empirical Methods in Natural Language Processing, 2018

    2018

  53. [53]

    Scaling up masked diffusion models on text

    Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.18514, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    Large language diffusion models

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025

    Pith/arXiv arXiv 2025

  55. [55]

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data

    Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In International Conference on Learning Representations, 2025

    2025

  56. [56]

    The lambada dataset: Word prediction requiring a broad discourse context

    D Paperno, G Kruszewski, A Lazaridou, QN Pham, Raffaella Bernardi, S Pezzelle, M Baroni, G Boleda, and R Fern \'a ndez. The lambada dataset: Word prediction requiring a broad discourse context. In 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016-Long Papers, volume 3, pp.\ 1525--1534. Association for Computational Linguistics...

    2016

  57. [57]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 4195--4205, 2023

    2023

  58. [58]

    Mauve: Measuring the gap between neural text and human text using divergence frontiers

    Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 2021

    2021

  59. [59]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  60. [60]

    Simple and effective masked diffusion language models

    Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 2024

    2024

  61. [61]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 2021

    2021

  62. [62]

    Your llm knows the future: Uncovering its multi-token prediction potential

    Mohammad Samragh, Arnav Kundu, David Harrison, Kumari Nishu, Devang Naik, Minsik Cho, and Mehrdad Farajtabar. Your llm knows the future: Uncovering its multi-token prediction potential. arXiv preprint arXiv:2507.11851, 2025

    Pith/arXiv arXiv 2025

  63. [63]

    Accelerating transformer inference for translation via parallel decoding

    Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodol \`a . Accelerating transformer inference for translation via parallel decoding. arXiv preprint arXiv:2305.10427, 2023

    Pith/arXiv arXiv 2023

  64. [64]

    Simplified and generalized masked diffusion for discrete data

    Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37: 0 103131--103167, 2024

    2024

  65. [65]

    Diffusionblocks: Blockwise training for generative models via score-based diffusion

    Makoto Shing and Takuya Akiba. Diffusionblocks: Blockwise training for generative models via score-based diffusion. arXiv preprint arXiv:2506.14202, 2025

    arXiv 2025

  66. [66]

    Ideas in inference-time scaling can benefit generative pre-training algorithms

    Jiaming Song and Linqi Zhou. Ideas in inference-time scaling can benefit generative pre-training algorithms. arXiv preprint arXiv:2503.07154, 2025

    Pith/arXiv arXiv 2025

  67. [67]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021

    2021

  68. [68]

    Seed diffusion: A large-scale diffusion language model with high-speed inference

    Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, et al. Seed diffusion: A large-scale diffusion language model with high-speed inference. arXiv preprint arXiv:2508.02193, 2025

    Pith/arXiv arXiv 2025

  69. [69]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  70. [70]

    Score-based continuous-time discrete diffusion models

    Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. arXiv preprint arXiv:2211.16750, 2022

    Pith/arXiv arXiv 2022

  71. [71]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  72. [72]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, volume 30, 2017

    2017

  73. [73]

    u tte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Sch \

    Dimitri von R \"u tte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Sch \"o lkopf, and Thomas Hofmann. Generalized interpolating discrete diffusion. In International Conference on Machine Learning, 2025

    2025

  74. [74]

    Remasking discrete diffusion models with inference-time scaling

    Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. In ICLR Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2025

    2025

  75. [75]

    Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding

    Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.22618, 2025

    Pith/arXiv arXiv 2025

  76. [76]

    Ar-diffusion: Auto-regressive diffusion model for text generation

    Tong Wu, Zhihao Fan, Xiao Liu, Hai-Tao Zheng, Yeyun Gong, Jian Jiao, Juntao Li, Jian Guo, Nan Duan, Weizhu Chen, et al. Ar-diffusion: Auto-regressive diffusion model for text generation. Advances in Neural Information Processing Systems, 36: 0 39957--39974, 2023

    2023

  77. [77]

    A survey on non-autoregressive generation for neural machine translation and beyond

    Yisheng Xiao, Lijun Wu, Furu Meng, Tao Qin, and Tie-Yan Liu. A survey on non-autoregressive generation for neural machine translation and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

    2022

  78. [78]

    Energy-based diffusion language models for text generation

    Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovec, Stefano Ermon, and Arash Vahdat. Energy-based diffusion language models for text generation. arXiv preprint arXiv:2410.21357, 2024

    Pith/arXiv arXiv 2024

  79. [79]

    Dream 7b: Diffusion large language models

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487, 2025

    Pith/arXiv arXiv 2025

  80. [80]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2019

    2019

Showing first 80 references.