Pith. sign in

REVIEW 4 major objections 4 minor 168 references

DiffusionGemma claims that discrete diffusion, obtained by fine-tuning instead of pretraining, can decode about 20 tokens per forward pass and reach roughly 1,500 tokens per second on a single H100, establishing a new Pareto frontier for sp

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:09 UTC pith:XMEY7REV

load-bearing objection A well-engineered open-weights diffusion LLM with a plausible but over-stated Pareto claim — the speed is measured, the loop-free assumption is not. the 4 major comments →

arxiv 2608.00146 v1 pith:XMEY7REV submitted 2026-07-31 cs.CL cs.AI

DiffusionGemma Technical Report

DiffusionGemma Team: Adrien Ali Ta\"iga , James Assiene , Daniele Calandriello , Rahma Chaabouni , Jo\~ao Gante , Tamara von Glehn , Nate Keating , Chris Knutsen
show 35 more authors
Martin Kukla Tianlin Liu Ivan Lobov Ofir Nabati Jo\~ao Gabriel Oliveira Nicolas Perez-Nieves Nastasia Prutianova Bobak Shahriari Jean Tarbouriech Pavel Tyletski \c{C}a\u{g}lar \"Unl\"u Cindy Wu Glenn Cameron Jerome Connor Sertan Girgin Maarten Grootendorst Alon Levkovitch Eliya Nachmani Omar Sanseviero Piotr Stanczyk Quentin Berthet Andrew Campbell Cl\'ement Crepy Valentin De Bortoli Arnaud Doucet Romuald Elie Alexandre Galashov Klaus Greff Alexis Jacq David Ruhe Yu-Han Wu Sebastian Flennerhag Brendan O'Donoghue George Scrivener Shantanu Thakoor
This is my paper
classification cs.CL cs.AI
keywords discrete diffusiontext diffusion language modelblock-autoregressive decodingentropy-bounded samplingadaptive stoppingsampler distillationreinforcement learninginference speed
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DiffusionGemma is a bid to move text generation from token-at-a-time decoding to parallel block refinement without sacrificing the intelligence of a modern model. The authors claim that fine-tuning an existing 25B-parameter mixture-of-experts autoregressive model rather than pretraining a diffusion model from scratch produces a discrete diffusion model that emits about 20 tokens per forward pass and reaches roughly 1,500 output tokens per second on a single H100 at batch size one. That is several times faster than the same model served autoregressively, even with speculative decoding, with a modest benchmark-quality penalty and retained multimodal, long-context, and thinking capabilities. A sympathetic reader should read this as the first open-weight evidence that text diffusion can be a practical serving alternative to autoregression in the low-concurrency regime.

Core claim

The central claim is that a text diffusion model can be obtained by fine-tuning an existing autoregressive checkpoint, and that the resulting model establishes a new Pareto frontier between generation speed and capability. The report argues this is achieved by keeping a causal encoder for context and key-value cache appends, denoising 256-token canvases with bidirectional attention, and then running a second training stage that combines sampler distillation with reinforcement learning, simultaneously raising reward and compressing the denoising trajectory to about 12 effective steps. The empirical assertion is concrete: roughly 20 tokens per forward pass and about 1,500 tokens per second on

What carries the argument

The mechanism that carries the argument is block-autoregressive discrete diffusion. The model encodes the prompt and already-written text with causal attention into a key-value cache, then iteratively denoises a 256-token canvas of uniformly random tokens using bidirectional attention, self-conditioning on its own previous predictions; each finished canvas is appended to the cache. Two named components matter: the entropy-bounded sampler accepts the lowest-entropy tokens and re-noises the rest, capping mutual information at a budget of 0.1, and adaptive stopping ends denoising when mean entropy falls below 0.005 and two consecutive argmax predictions agree. Together these turn the number of

Load-bearing premise

The strongest speed and quality numbers rest on one premise: the second training stage has made few-step generation safe, so the adaptive stop fires because the canvas has converged rather than because the model fell into a repetitive loop; and the report itself admits occasional token stuttering still occurs, so that premise is only approximately true.

What would settle it

Take a set of open-ended prompts, run DiffusionGemma with its default sampler, and log the step at which adaptive stopping fires together with the final output. If any substantial fraction of outputs show a coherent prefix that collapses into repetition while mean entropy falls below 0.005, the speed figure is measuring loop collapse rather than convergence. A complementary check is to rerun the same prompts with the maximum 48 denoising steps: wherever the 48-step output is clearly better than the default-stop output, the effective-step claim is not holding for that input.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • At batch size 1 on an H100, DiffusionGemma reaches about 1,500 tokens per second, versus 204 for the same model autoregressively and 303 with multi-token-prediction speculative decoding, roughly 7x and 5x faster.
  • The same weights can still decode left-to-right, with quality close to the original autoregressive model, so requests could be routed between diffusion and AR modes by latency and task difficulty.
  • The two-stage recipe uses less than 10% of the starting autoregressive model's training token budget, so converting an existing checkpoint into a diffusion model is cheap relative to pretraining.
  • Structured or highly constrained outputs such as JSON, code edits, or Sudoku grids can converge in two or three denoising steps, and even sequentially dependent outputs take only about seven steps for 40 tokens.
  • The joint reinforcement-learning and sampler-distillation stage also makes the model produce roughly half as many tokens as the supervised-fine-tuning checkpoint, which compounds the speed gain.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the dual diffusion/AR capability suggests a serving system could route short, latency-critical generations to diffusion and difficult or open-ended reasoning to autoregressive mode, or interleave both within one response; the paper does not implement such a router.
  • Editorial extension: the fixed entropy thresholds are global, but the report's own step-count plots show domain-dependent convergence, so a likely improvement is learned or per-prompt stopping thresholds, which the paper does not explore.
  • Editorial extension: the single-request speed advantage comes partly from shifting memory-bound work to compute-bound work, so the same recipe should benefit most on hardware with high compute-to-bandwidth ratios; the paper notes this but does not quantify it across GPUs.
  • Editorial extension: because the sampler re-noises high-entropy tokens, the model has a built-in opportunity for constrained decoding: a schema or grammar could force low-entropy structure and let the sampler explore only the uncertain slots; the paper demonstrates the phenomenon qualitatively without building such an interface.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. DiffusionGemma converts the pretrained Gemma 4 26B A4B mixture-of-experts language model into a discrete diffusion language model via a two-stage pipeline: supervised fine-tuning to teach bidirectional 256-token canvas denoising, followed by an online sampler-distillation and reinforcement-learning stage (SD·RL) that jointly improves quality and compresses the denoising trajectory. The resulting model generates text in block-autoregressive fashion, with an entropy-bounded sampler, temperature annealing, and adaptive stopping. The report claims roughly 20 tokens per forward pass (TPF) and about 1,500 output tokens per second on a single H100, establishing a new Pareto frontier for speed-versus-quality, while retaining AR decoding, thinking mode, multimodal inputs, and long-context support. The paper includes detailed architecture descriptions, training details, latency breakdowns, benchmark tables, and qualitative examples of bidirectional reasoning and self-correction.

Significance. If the central claims hold, this is a significant systems contribution: it demonstrates that a discrete diffusion model fine-tuned from an AR MoE backbone can deliver large low-batch decoding speedups while remaining competitive on reasoning and coding benchmarks, and the open-weights release with HuggingFace/vLLM reference implementations would provide a strong reproducible baseline for the community. The paper's strengths include measured hardware timings, a transparent per-kernel breakdown, a clear two-stage training recipe, and explicit acknowledgment of known limitations. However, the headline Pareto and speed claims rest on the entropy-bounded adaptive-stopping heuristic not being fooled by degenerate repetitive loops. The paper's own evidence shows that this exact failure mode occurs in the SFT checkpoint, and Section 10 concedes that the final model still exhibits occasional stuttering and repetitive loops. Because TPF/TPS are defined through adaptive-stopping trajectories, the empirical robustness of the stopping trigger is load-bearing and must be established before the central claim can be accepted. The comparison to Mercury 2 also rests on a black-box API speed estimate

major comments (4)
  1. [§5, §10, Algorithm 1, Eq. (10)] The headline TPF/TPS figures are computed along adaptive-stopping trajectories, and the stopping condition (Algorithm 1: mean entropy ≤ e_stop and stable argmax) can be triggered by the repetitive-loop degeneration documented for the SFT checkpoint in Section 5 and Appendix B (Figures 16–17). Section 10 concedes that the final SD·RL checkpoint still exhibits "occasional token stuttering... repetitive loops." The manuscript does not report the per-canvas rate of loop-induced early stops on the final checkpoint, nor the correlation between early stopping and correctness. Since Eq. (10) defines TPF using the adaptive-stopping trajectory, a non-negligible loop rate would inflate TPF/TPS while deflating quality. This is load-bearing for the abstract's "~20 TPF / ~1,500 TPS" and the Pareto claim. Please report loop/degeneration rates on the final checkpoint across the evaluation suite and show
  2. [Figure 1, §7, Table 3] The "new Pareto frontier" claim is established on a two-benchmark average (GPQA-Diamond and LiveCodeBench-v6) for both quality and speed, while the full evaluation suite contains 19 benchmarks. A two-task average is a thin basis for a frontier claim, especially because both tasks are reasoning/coding oriented and may favor thinking mode. In addition, Table 3 reports no error bars or seed variance; Figure 9 mentions 3-seed averages only for the SD·RL frontier. Please report the Pareto comparison over the full suite or provide a principled justification for the restricted subset, and include variance estimates or significance tests for the headline quality and speed comparisons.
  3. [Appendix E, §7] The speed comparison to Mercury 2 is based on a black-box API latency estimate using NNLS on Eq. (14). The appendix explicitly states that corrupted outputs exhausting the 50k-token maximum produce outliers that give a "slightly favorable bias" to the speed estimate, yet the paper uses this estimate for the "roughly 2.5× speedup over Mercury 2" claim and for Figure 1. Please report the estimates with outliers excluded (for example, capping output tokens at a plausible limit) and state how the comparison changes; otherwise the 2.5× speedup claim is not robust.
  4. [Abstract, §3, §10] The paper claims that DiffusionGemma "retains the starting model's support for ... long contexts," but no long-context evaluation is presented anywhere in the manuscript. Given the block-autoregressive KV-cache mechanism is a novel architectural modification, this claim needs empirical support, for example a long-context QA or summarization experiment at increasing input lengths, before it can be presented as a retained capability.
minor comments (4)
  1. [Figure 1 and Table 3 caption] The speed measurements for LLaDA 2.1 Flash (8× B200) and Nemotron 14B (1× H100, bfloat16) are made on different hardware than the DiffusionGemma H100 FP8 measurements, yet they are plotted on the same axes without a normalization caveat. Please add a clear note that cross-hardware TPS comparisons are indicative only.
  2. [§6, Eq. (12), Table 4] Eq. (12) defines TPS using a single t_fwd (13.56 ms for 4096-input/1024-output tokens), while Table 4 reports per-benchmark TPS that vary with context length. Please clarify whether the headline "~1,500 TPS" uses the fixed t_fwd or the per-benchmark estimates, and report the distribution of t_fwd across benchmarks.
  3. [Section 10, Table 3] The MMMU-Pro thinking score (54.3) dropping below the no-think score (66.0) is attributed to missing closing thought tags. This is a significant qualitative effect and should be discussed in the main experimental section, not only in the limitations list, so that readers do not misinterpret the thinking-mode benchmark as a capability limitation.
  4. [General] The first paragraph of the PDF/marked-up text contains missing spaces between words ("WeintroduceDiffusionGemma..."). Please ensure the arXiv source formatting is fixed in the final version.

Circularity Check

0 steps flagged

No significant circularity; the paper's central claims are empirical measurements.

full rationale

DiffusionGemma is an empirical systems report. The headline numbers (TPF≈20, TPS≈1500) are measured values: TPF is defined in Eq. 10 as Total Tokens/(Total Denoising Steps + K − 1), and each quantity is reported per benchmark in Table 4; Eq. 12 then converts TPF to TPS using a measured t_fwd. No parameter is fitted to the headline speed and then re-presented as a prediction. The adaptive-stopping heuristic (Algorithm 1) and hyperparameters (b=0.1, e_stop=0.005, N=48) are engineering choices, and the paper explicitly shows that without SD·RL the SFT checkpoint degenerates into loops that collapse the entropy signal, which is a benchmark/deployment validity concern rather than an equation-level circularity. Self-citations are present (e.g., the block-AR provenance note and interpretability citations) but none is load-bearing: the block-AR strategy is also credited to independent concurrent work (Wu et al. 2025, Arriola et al. 2025, Deschenaux and Gulcehre 2026a), and the interpretability citations support a non-essential remark. The residual-loop limitation in Section 10 could affect the robustness of the Pareto claim, but that is an unverified assumption about deployment, not a reduction of the derivation to its inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

No new physical or conceptual entities are introduced. The self-conditioning MLP is an architectural module adopted from prior self-conditioning methods; it is not a new entity. Free parameters are the tuned sampler hyperparameters and canvas size, which are not derived and directly influence the reported efficiency metrics. The central derivation uses standard discrete diffusion results from the cited literature.

free parameters (5)
  • Canvas length C = 256 tokens
    Architectural choice for block-AR decoding; determines parallelization and per-forward cost; not derived from first principles.
  • Entropy budget b = 0.1
    Sampler acceptance threshold controlling token commitment; tuned for quality/latency; central to TPF.
  • Adaptive stopping threshold e_stop = 0.005
    Entropy threshold to trigger early stopping; tuned; directly determines the average number of denoising steps.
  • Temperature schedule endpoints tau_max/tau_min = 0.8 / 0.4
    Linear annealing schedule for softmax over denoising time; tuned; affects sampling diversity and convergence behavior.
  • Maximum denoising steps N = 48
    Inference-time budget; average effective ~12 via adaptive stopping; chosen to balance quality and latency.
axioms (4)
  • standard math Continuous-time Markov chain / discrete flow matching theory (Campbell et al., Gat et al.) correctly describes the denoising process, and the neural network can learn the posterior ℙ(X0|Xt).
    Invoked throughout Section 2; the whole training objective (Eq. 11) rests on this.
  • domain assumption Warm-starting from the post-trained Gemma 4 26B A4B checkpoint transfers capabilities (reasoning, multimodal, long context) into the diffusion model.
    Section 3: inheritance of features is a stated motivation; no ablation shows how much capability comes from initialization vs. diffusion training.
  • domain assumption Entropy-bounded sampling with adaptive stopping reliably signals convergence (low entropy = high-quality tokens) across tasks, not degenerate loops.
    Section 3.3; the paper shows this fails for the SFT model at N=48 (Figures 16-17) and assumes SD·RL fixes it robustly across the eval suite.
  • domain assumption SD·RL online teacher trajectories serve as high-quality references for distillation, and joint reward + entropy minimization does not collapse diversity.
    Section 5; no analysis of diversity collapse beyond conciseness observed.

pith-pipeline@v1.3.0-alltime-deepseek · 39228 in / 13646 out tokens · 150064 ms · 2026-08-04T01:09:24.282760+00:00 · methodology

0 comments
read the original abstract

We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models. Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2B total parameters. Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget. The first stage uses supervised fine-tuning to teach bidirectional denoising, while the second stage combines reinforcement learning with sampler distillation to jointly improve generation quality and inference efficiency. DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability. Averaged across our full evaluation suite, it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU, which is substantially faster than AR models even with state-of-the-art speculative decoding. DiffusionGemma also retains the starting model's support for thinking mode, multimodal inputs, and long contexts. Despite diffusion fine-tuning, it remains capable of AR generation with only minor performance degradation, suggesting a path toward hybrid diffusion-AR decoding.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

168 extracted references · 25 linked inside Pith

  1. [1]

    Mastering the game of

    Silver, David and Huang, Aja and Maddison, Chris J and Guez, Arthur and Sifre, Laurent and Van Den Driessche, George and Schrittwieser, Julian and Antonoglou, Ioannis and Panneershelvam, Veda and Lanctot, Marc and others , journal=. Mastering the game of. 2016 , publisher=

    2016

  2. [2]

    arXiv preprint arXiv:2503.19786 , year=

    Gemma 3 technical report , author=. arXiv preprint arXiv:2503.19786 , year=

    Pith/arXiv arXiv

  3. [3]

    Beyond autoregression: Fast

    Deschenaux, Justin and Gulcehre, Caglar , booktitle=iclr, year=. Beyond autoregression: Fast

  4. [4]

    Song, Yang and Dhariwal, Prafulla and Chen, Mark and Sutskever, Ilya , title =

  5. [5]

    Progressive Distillation for Fast Sampling of Diffusion Models , author=

  6. [6]

    Learnable Sampler Distillation for Discrete Diffusion Models , author=

  7. [7]

    Liu, Jie and Liu, Gongye and Liang, Jiajun and Li, Yangguang and Liu, Jiaheng and Wang, Xintao and Wan, Pengfei and Zhang, Di and Ouyang, Wanli , booktitle=neurips, year=

  8. [8]

    Zheng, Kaiwen and Chen, Huayu and Ye, Haotian and Wang, Haoxiang and Zhang, Qinsheng and Jiang, Kai and Su, Hang and Ermon, Stefano and Zhu, Jun and Liu, Ming-Yu , booktitle=icml, year=

  9. [9]

    Training diffusion models with reinforcement learning , author=

  10. [10]

    d1: Scaling reasoning in diffusion large language models via reinforcement learning , author=

  11. [11]

    Reinforcement learning with discrete diffusion policies for combinatorial action spaces , author=

  12. [12]

    Accelerated sampling from masked diffusion models via entropy bounded unmasking , author=

  13. [13]

    Discrete flow matching , author=

  14. [14]

    2026 , howpublished=

    2026

  15. [15]

    Zadouri, Ted and Hoehnerbach, Markus and Shah, Jay and Thakkar, Vijay and Dao, Tri , booktitle =

  16. [16]

    Language models are few-shot learners , author =

  17. [17]

    arXiv preprint arXiv:2312.11805 , year=

    Pith/arXiv arXiv

  18. [18]

    arXiv preprint arXiv:2403.08295 , year=

    Gemma: Open models based on. arXiv preprint arXiv:2403.08295 , year=

    Pith/arXiv arXiv

  19. [19]

    Deep unsupervised learning using nonequilibrium thermodynamics , author=

  20. [20]

    Denoising diffusion probabilistic models , author=

  21. [21]

    Li, Xiang Lisa and Thickstun, John and Gulrajani, Ishaan and Liang, Percy and Hashimoto, Tatsunori B , booktitle=neurips, year=

  22. [22]

    arXiv preprint arXiv:2211.04236 , year=

    Self-conditioned Embedding Diffusion for Text Generation , author=. arXiv preprint arXiv:2211.04236 , year=

    Pith/arXiv arXiv

  23. [23]

    2022 , primaryClass=

    Continuous diffusion for categorical data , author=. 2022 , primaryClass=

    2022

  24. [24]

    arXiv preprint arXiv:2506.02070 , year =

    Holderrieth, Peter and Erives, Ezra , title =. arXiv preprint arXiv:2506.02070 , year =

    arXiv

  25. [25]

    PeerJ Computer Science , year=

    Diffusion models in text generation: a survey , author =. PeerJ Computer Science , year=

  26. [26]

    Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall , author=

  27. [27]

    Shabalin, Alexander and Meshchaninov, Viacheslav and Chimbulatov, Egor and Lapikov, Vladislav and Kim, Roman and Bartosh, Grigory and Vetrov, Dmitry , booktitle=aaai, year=

  28. [28]

    Cosmos: Compressed and Smooth Latent Space for Text Diffusion Modeling , author=

  29. [29]

    Hu, Keya and Qiu, Linlu and Lu, Yiyang and Zhao, Hanhong and Li, Tianhong and Kim, Yoon and Andreas, Jacob and He, Kaiming , journal=

  30. [30]

    Journal of Machine Learning Research , year =

    Lingxiao Zhao and Xueying Ding and Lijun Yu and Leman Akoglu , title =. Journal of Machine Learning Research , year =

  31. [31]

    Structured denoising diffusion models in discrete state-spaces , author=

  32. [32]

    Simple and Effective Masked Diffusion Language Models , author=

  33. [33]

    Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution , author=

  34. [34]

    2026 , howpublished =

    2026

  35. [35]

    2026 , note =

    Gemma 4 Model Card , url =. 2026 , note =

    2026

  36. [36]

    arXiv preprint arXiv:2607.02770 , year=

    Gemma 4 technical report , author=. arXiv preprint arXiv:2607.02770 , year=

    Pith/arXiv arXiv

  37. [37]

    A Continuous Time Framework for Discrete Denoising Models , author=

  38. [38]

    Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design , author=

  39. [39]

    Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspective , author=

  40. [40]

    Mask-Predict: Parallel Decoding of Conditional Masked Language Models , author=

  41. [41]

    arXiv preprint arXiv:2602.23225 , year=

    Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding? , author=. arXiv preprint arXiv:2602.23225 , year=

    arXiv

  42. [42]

    Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning , author=

  43. [43]

    The Curious Case of Neural Text Degeneration , author=

  44. [44]

    Zhang, Shimao and Bao, Yu and Huang, Shujian , journal=

  45. [45]

    How Transparent is

    Engels, Joshua and McDougall, Callum and Chughtai, Bilal and Kramar, Janos and Rajamanoharan, Senthoran and Wu, Cindy and Conmy, Arthur and Chen, Asic Q and Tarbouriech, Jean and Ma, Min and O'Donoghue, Brendan and de Oliveira, Jo. How Transparent is. arXiv preprint arXiv:2606.20560 , year=

    Pith/arXiv arXiv

  46. [46]

    Fast inference from transformers via speculative decoding , author=

  47. [47]

    Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding

    Xia, Heming and Yang, Zhe and Dong, Qingxiu and Wang, Peiyi and Li, Yongqi and Ge, Tao and Liu, Tianyu and Li, Wenjie and Sui, Zhifang. Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding. Findings of the Association for Computational Linguistics: ACL 2024. 2024

    2024

  48. [48]

    2025 , booktitle =

    Zhu, Kan and Gao, Yufei and Zhao, Yilong and Zhao, Liangyu and Zuo, Gefei and Gu, Yile and Xie, Dedong and Tang, Tian and Xu, Qinyu and Ye, Zihao and Kamahori, Keisuke and Lin, Chien-Yu and Wang, Ziren and Wang, Stephanie and Krishnamurthy, Arvind and Kasikci, Baris , title =. 2025 , booktitle =

    2025

  49. [49]

    Jin, Qiao and Dhingra, Bhuwan and Liu, Zhengping and Cohen, William and Lu, Xinghua , booktitle=emnlp, year=

  50. [50]

    Zhu, Ruidong and Jiang, Ziheng and Jin, Chao and Wu, Peng and Stuardo, Cesar A. and Wang, Dongyang and Zhang, Xinlei and Zhou, Huaping and Wei, Haoran and Cheng, Yang and Xiao, Jianzhe and Zhang, Xinyi and Liu, Lingjun and Lin, Haibin and Chang, Li-Wen and Ye, Jianxi and Yu, Xiao and Liu, Xuanzhe and Jin, Xin and Liu, Xin , title =. 2025 , booktitle =

    2025

  51. [51]

    Quest: Query-Aware Sparsity for Efficient Long-Context

    Tang, Jiaming and Zhao, Yilong and Zhu, Kan and Xiao, Guangxuan and Kasikci, Baris and Han, Song , booktitle=icml, year=. Quest: Query-Aware Sparsity for Efficient Long-Context

  52. [52]

    arXiv preprint arXiv:2110.14168 , year=

    Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

    Pith/arXiv arXiv

  53. [53]

    arXiv preprint arXiv:2108.07732 , year=

    Program synthesis with large language models , author=. arXiv preprint arXiv:2108.07732 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

    Pith/arXiv arXiv

  55. [55]

    Language models are multilingual chain-of-thought reasoners , author=

  56. [56]

    Terry Yue Zhuo and Vu Minh Chien and Jenny Chim and Han Hu and Wenhao Yu and Ratnadira Widyasari and Imam Nur Bani Yusuf and Haolan Zhan and Junda He and Indraneil Paul and Simon Brunner and Chen GONG and James Hoang and Armel Randy Zebaze and Xiaoheng Hong and Wen-Ding Li and Jean Kaddour and Ming Xu and Zhihan Zhang and Prateek Yadav and Naman Jain and ...

  57. [57]

    Bowman , booktitle=colm, year=

    David Rein and Betty Li Hou and Asa Cooper Stickland and Jackson Petty and Richard Yuanzhe Pang and Julien Dirani and Julian Michael and Samuel R. Bowman , booktitle=colm, year=

  58. [58]

    Tsoukalas, George and Lee, Jasper and Jennings, John and Xin, Jimmy and Ding, Michelle and Jennings, Michael and Thakur, Amitayush and Chaudhuri, Swarat , booktitle=neurips, year=

  59. [59]

    On leakage of code generation evaluation datasets , author=

  60. [60]

    Jain, Naman and Gu, Alex and Li, Wen-Ding and Yan, Fanjia and Zhang, Tianjun and Wang, Sida and Solar-Lezama, Armando and Sen, Koushik and Stoica, Ion , booktitle=iclr, year=

  61. [61]

    Yue, Xiang and Zheng, Tianyu and Ni, Yuansheng and Wang, Yubo and Zhang, Kai and Tong, Shengbang and Sun, Yuxuan and Yu, Botao and Zhang, Ge and Sun, Huan and Su, Yu and Chen, Wenhu and Neubig, Graham , booktitle=acl, year=

  62. [62]

    Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with

    Jasper Dekoninck and Nikola Jovanovi. Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with. 3rd AI for Math Workshop at the International Conference on Machine Learning (ICML) , year=

  63. [63]

    Attention is All you Need , year =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser,. Attention is All you Need , year =

  64. [64]

    Gemini Diffusion , year =

  65. [65]

    Accelerating Gemma 4: faster inference with multi-token prediction drafters , year =

  66. [66]

    arXiv preprint arXiv:2508.02193 , year=

    Seed diffusion: A large-scale diffusion language model with high-speed inference , author=. arXiv preprint arXiv:2508.02193 , year=

    Pith/arXiv arXiv

  67. [67]

    2026 , note =

    Yonggan Fu and Lexington Whalen and Abhinav Garg and Chengyue Wu and Maksim Khadkevich and Nicolai Oswald and Enze Xie and Daniel Egert and Sharath Turuvekere Sreenivas and Shizhe Diao and Chenhan Yu and Ye Yu and Weijia Chen and Sajad Norouzi and Jingyu Liu and Shiyi Lan and Ligeng Zhu and Jin Wang and Jindong Jiang and Morteza Mardani and Mehran Maghoum...

    2026

  68. [68]

    arXiv preprint arXiv:2506.17298 , year=

    Pith/arXiv arXiv

  69. [69]

    Large language diffusion models , author=

  70. [70]

    Bie, Tiwei and Cao, Maosong and Chen, Kun and Du, Lun and Gong, Mingliang and Gong, Zhuochen and Gu, Yanmei and Hu, Jiaqi and Huang, Zenan and Lan, Zhenzhong and others , journal=

  71. [71]

    Li, Yuhui and Wei, Fangyun and Zhang, Chao and Zhang, Hongyang , booktitle=neurips, year=

  72. [72]

    Chen, Jian and Liang, Yesheng and Liu, Zhijian , booktitle=icml, year=

  73. [73]

    Cheng, Xin and Yu, Xingkai and Shao, Chenze and Li, Jiashi and Xiong, Yunfan and Qian, Yi and Zhu, Jiaqi and Ma, Shirong and Zhang, Xiaokang and Ye, Jiasheng and Chen, Qinyu and Deng, Chengqi and Yu, Jiping and Dai, Damai and Zhang, Zhengyan and Wei, Yixuan and Tan, Yixuan and Yang, Wenkai and Xu, Runxin and Wu, Yu and Xu, Zhean and Wang, Xuanyu and Chen,...

  74. [74]

    Liu, Jingyu and Dong, Xin and Ye, Zhifan and Mehta, Rishabh and Fu, Yonggan and Singh, Vartika and Zhang, Ce and Molchanov, Pavlo , booktitle =

  75. [75]

    Journal of Machine Learning Research , volume=

    A neural probabilistic language model , author=. Journal of Machine Learning Research , volume=

  76. [76]

    Interspeech , volume=

    Recurrent neural network based language model , author=. Interspeech , volume=

  77. [77]

    arXiv preprint arXiv:1308.0850 , year=

    Generating sequences with recurrent neural networks , author=. arXiv preprint arXiv:1308.0850 , year=

    Pith/arXiv arXiv

  78. [78]

    Sequence to sequence learning with neural networks , author=

  79. [79]

    Non-autoregressive neural machine translation , author=

  80. [80]

    arXiv preprint arXiv:2302.01318 , year=

    Accelerating Large Language Model Decoding with Speculative Sampling , author=. arXiv preprint arXiv:2302.01318 , year=

    Pith/arXiv arXiv

Showing first 80 references.