Pith. sign in

REVIEW 3 major objections 6 minor 25 cited by

Dream-Coder 7B: An Open Diffusion Language Model for Code

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Dream-Coder 7B is a discrete diffusion language model for code that matches autoregressive baselines while generating code in task-adaptive, any-order patterns.

desk verdict First open diffusion code model to match AR 7B on LiveCodeBench – worth refereeing, but the RL objective is misspecified and the 'emergent patterns' are only anecdotal. read the letter →

arxiv 2509.01142 v1 pith:UWBHT3CV submitted 2025-09-01 cs.CL

Zhihui Xie , Jiacheng Ye , Lin Zheng , Jiahui Gao , Jingwei Dong , Zirui Wu , Xueliang Zhao , Shansan Gong
show 3 more authors
Xin Jiang Zhenguo Li Lingpeng Kong
This is my paper
classification cs.CL
keywords discretediffusioncodegenerationany-orderlanguagemodelreinforcementlearningverifiablerewardsLiveBenchmask-predict
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a discrete diffusion language model, trained entirely on public data, can match state-of-the-art autoregressive code models on standard benchmarks while also generating code in flexible, non-left-to-right orders. It introduces Dream-Coder 7B, adapted from an autoregressive checkpoint, and shows that after supervised fine-tuning and reinforcement learning with unit-test rewards, the model reaches 21.4% pass@1 on LiveCodeBench (2410–2505), on par with stronger autoregressive baselines of similar size. The paper's distinctive claim is that three generation patterns—sketch-first scaffolding, left-to-right completion, and interleaved reasoning—emerge naturally from diffusion training rather than being programmed. If true, this makes non-autoregressive generation a practical alternative to next-token prediction for code, not just a research curiosity.

What carries the argument

The central mechanism is the shift operation, which lets a Qwen2.5-Coder-style autoregressive transformer be reused as a diffusion denoiser by predicting tokens at shifted positions rather than original masked positions, preserving pretrained knowledge while gaining bidirectional context. Training uses continuous-time absorbing-state masked diffusion with a context-adaptive token-level noise rescheduling loss that assigns higher noise rates to harder-to-predict tokens. The post-training pipeline—random truncation, a padding penalty, and a GRPO reinforcement-learning recipe with verifiable rewards, coupled sampling, and informative substitution—converts the base model into the instruction-fol

What would settle it

Train the same supervised SFT checkpoint with a diffusion-native policy gradient—or with random mask orders—while keeping the same reward data and compute budget. If pass@1 on LiveCodeBench stays at 21.4%, the claim that the paper's GRPO recipe drives the gains is falsified. A second check: force decoding order to be strictly left-to-right on LiveCodeBench and compare with the model's free-form adaptive order; if the adaptive order does not outperform the fixed order, the 'emergent any-order generation' claim is weakened.

Watch

Extended reading notes

Core claim

The central claim is that adapting a pretrained autoregressive code model into a continuous-time absorbing-state diffusion model, then post-training it with supervised fine-tuning and verifiable-reward reinforcement learning, yields a code generator whose benchmark performance is competitive with autoregressive models of similar size and whose decoding order is task-adaptive. Because the model denoises masked tokens in parallel, it is not bound to left-to-right order: on complex algorithmic prompts it first produces a structural sketch, on simple completions it behaves almost left-to-right, and on reasoning-heavy tasks it interleaves logical components. The LiveCodeBench result of 21.4% pass

Load-bearing premise

The reinforcement-learning objective assumes that token-level importance ratios computed as if the model generated left-to-right are a valid way to optimize a diffusion denoiser, but a diffusion policy generates tokens in parallel and not in left-to-right order; if that surrogate is not valid, the reported post-RL gains may not be caused by the stated reinforcement-learning mechanism.

Editorial extensions

If this is right

  • Diffusion models can serve as practical code generators, not just research prototypes: Dream-Coder 7B matches autoregressive baselines on HumanEval, MBPP, BigCodeBench, and LiveCodeBench.
  • Any-order generation opens a new control knob: decoding can be biased toward sketch-first, left-to-right, or interleaved patterns depending on the task's structure.
  • The post-training recipe—SFT padding fixes plus a diffusion-tailored GRPO variant—is transferable to other diffusion language models and other verifiable-reward domains.
  • Iterative denoising allows adjustable quality-speed trade-offs at inference time, complementing test-time compute scaling.
  • Releasing checkpoints, training recipes, and preprocessing pipelines lets others reproduce and extend the approach without proprietary data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • In my reading, the paper's GRPO objective uses token-level importance ratios derived from a left-to-right policy factorization, yet a diffusion denoiser does not factorize generation left-to-right; if that surrogate does not hold, the post-RL gains could stem from the SFT data, reward overfitting, or the coupled-sampling trick rather than from policy optimization—a controlled comparison with a dif
  • The three observed generation patterns could be exploited further: if a lightweight controller selects the mask schedule per prompt, it might push pass@1 above the free-form adaptive behavior.
  • The same adaptation recipe may transfer to non-code domains such as mathematical reasoning or agentic planning, where arbitrary-order generation and iterative refinement are also valuable.
  • The 'emergent' patterns are likely learned associations between prompt structure and decoding order; measuring how predictably the pattern correlates with task category would show whether they are truly adaptive or simply dataset-driven.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Dream-Coder 7B, a 7B discrete diffusion language model for code initialized from the Qwen2.5-Coder checkpoint and trained with a continuous-time absorbing-state masked-diffusion objective. Post-training consists of supervised fine-tuning on 5M instruction examples with random truncation and a logit-level padding penalty, followed by GRPO-style reinforcement learning with unit-test verifiable rewards on a curated 17K-prompt set. The model is evaluated on HumanEval, MBPP, EvalPlus, BigCodeBench, LiveCodeBench, CRUXEval, and several general reasoning benchmarks. The paper's central claims are that Dream-Coder 7B Instruct reaches 21.4% pass@1 on LiveCodeBench (2410–2505), matching open autoregressive baselines, and that the model exhibits three emergent any-order generation patterns: sketch-first scaffolding, left-to-right completion, and interleaved reasoning.

Significance. If the empirical results hold, the contribution is meaningful: it would be the first open diffusion code model to reach parity with 7B-class autoregressive models on competitive coding benchmarks, and the release of weights, recipes, and code would be a useful community asset. The benchmark results are externally evaluated against held-out unit tests, so the headline numbers are not fitted constants; this is a genuine strength. However, the paper's most distinctive claim—emergent, task-adaptive any-order generation—is currently supported only by qualitative inspection of three examples, and the RL objective as written uses an autoregressive policy ratio that does not match the parallel masked-denoising distribution. Both issues bear directly on the paper's central contributions, so the significance is conditional until those are resolved.

major comments (3)
  1. [§4.2, GRPO objective] The displayed GRPO objective uses ρ_{i,t}(θ) = π_θ(o_{i,t} | q, o_{i,<t}) / π_old(o_{i,t} | q, o_{i,<t}), which is a token-level importance ratio for a left-to-right autoregressive policy. The model defined in §2.1 is a discrete diffusion model: masked tokens are predicted in parallel from the full masked context x_t, and the generation distribution is not factorized as a product of left-to-right conditionals. As written, the objective is therefore not a likelihood-ratio surrogate for the diffusion policy. Additionally, t is used both for the continuous diffusion time sampled from U(0,1) and for the token index in the inner sum t=1…|o_i|, making the displayed equation internally inconsistent. The authors need to (i) write a policy-gradient surrogate adapted to masked-diffusion policies, or justify why the AR ratio is a valid approximation; and (ii) provide an ablation that isolates the c
  2. [§5.4 and Figure 3] The headline claim of 'emergent any-order generation' is supported only by one hand-picked example per pattern. The paper does not describe how the generation order was extracted from decoding traces, how many samples were inspected, how often each pattern occurs, or how 'order' is quantified. There is no comparison to fixed decoding schedules (e.g., strict left-to-right or random-order unmasking) and no comparison to the SFT model before RL. Consequently, the assertion that 'these patterns are not pre-programmed but emerge naturally' is not falsifiable from the presented evidence. Please provide a quantitative methodology—such as the distribution of first-unmasked positions, order statistics, or a classifier with inter-annotator agreement—and ablations over decoding schedules. If the patterns are simply consequences of the confidence-based unmasking schedule, the novelty claim should be
  3. [§5.3, Tables 4–5] The paper attributes part of the final performance to reinforcement learning, but the tables do not report Dream-Coder-7B-SFT numbers on the same benchmarks. Figure 2 appears to be a schematic rather than a reproducible numeric comparison. Since the post-training recipe is a central contribution, the authors should report SFT and Instruct results under identical decoding settings (including temperature, denoising steps, penalty coefficient, and mask schedule) so that the marginal effect of the RL stage is visible. Without this, the reader cannot verify that the stated recipe—rather than data curation or decoding choices—produces the reported gains.
minor comments (6)
  1. [§2.2] The 'shift operation' is only cited to Gong et al. (2025a). Since the architecture adaptation is central to the method, please include a formal definition or an appendix description so the paper is self-contained.
  2. [§4.1] The description 'randomly truncate each response to the length of a randomly selected example in the batch' is ambiguous. Please specify whether all sequences are truncated to the minimum length, the selected example's length, or some other rule.
  3. [§4.2] The symbol t is overloaded: it denotes continuous diffusion time in the expectation and token position in the inner sum. Rename one of them to avoid confusion.
  4. [§5.1] The evaluation setup does not report the exact inference configuration used for all benchmark numbers: temperature, number of denoising steps, mask schedule, and the padding penalty coefficient/decay. Since §4.1 shows that pass@k depends on these settings, a single reproducible configuration should be stated.
  5. [Tables 1–3] The 'Average' column is not defined. Please state whether it is an unweighted arithmetic mean over the preceding columns and list which columns are included.
  6. [Figure 3] The caption says colors encode generation order, but there is no legend in the figure and no description of how the order was computed. The color scale is also difficult to read in grayscale; please add a quantitative legend and a methodology note.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: headline benchmark results are externally evaluated, and cited same-group methods function as building blocks rather than as forced inputs; the main weaknesses are evidentiary, not circular.

full rationale

Dream-Coder 7B's central quantitative claims (21.4% pass@1 on LiveCodeBench, HumanEval/MBPP/BigCodeBench/CRUXEval scores) are measured against held-out external benchmarks with unit-test-based evaluation, so they are not derived from fitted constants or from the paper's own definitions. The paper does lean heavily on prior same-group work: the shift operation is adopted from Gong et al. (2025a) and Ye et al. (2025b), the pretraining objective uses Context-Adaptive Token-Level Noise Rescheduling (Ye et al., 2025b), and the RL recipe is 'inspired by' Yu et al. (2025), Gong et al. (2025b), and An et al. (2025). These are self-citations, but they are not load-bearing in a circular sense: the cited methods are used as architectural/training components whose formulas are partially restated in the paper (Eq. 2), and the final benchmark evaluation does not reduce to those citations. The Section 5.4 claim of 'emergent any-order generation patterns' rests only on qualitative inspection of Figure 3, without quantitative order statistics or baselines; that is an evidentiary weakness, not a circular derivation. Similarly, the AR-style importance ratio in the GRPO objective (Section 4.2) may be a policy-mis-specification concern for a diffusion model, but it is not a case of the paper fitting a parameter and then renaming it a prediction. Overall, no step in the derivation chain is equivalent to its own input by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the absorption of several substantial frameworks from prior work (continuous-time absorbing diffusion, shift operation, context-adaptive noise rescheduling, GRPO with clip-higher). The most fragile assumption is that the RL objective, written in autoregressive token form, is valid for the diffusion policy. The padding penalty and RL clip bounds are hand-set hyperparameters.

free parameters (5)
  • Padding penalty coefficient = 1e-12
    Applied to [PAD] token logits at inference; chosen from pass@K inspection in Figure 1b.
  • GRPO asymmetric clip bounds = epsilon_low=0.2, epsilon_high=0.28
    Hand-selected 'Clip-Higher' range in the RL objective.
  • Difficulty calibration thresholds = >=5 unit tests; 8 candidates; exclude all-perfect
    Data curation thresholds for the 17k RL prompt set.
  • Evaluation temperature = 0.1 (shown in Figure 1b)
    Temperature used for pass@K; the paper does not specify the temperature for Tables 4-5.
  • Inference denoising steps = not stated
    Number of diffusion denoising steps materially affects output quality and is not reported.
assumptions (5)
  • standard math Continuous-time absorbing discrete diffusion with weighted cross-entropy (Eq. 1) is a tractable and valid training objective for language models.
    Inherited from Kingma et al., Campbell et al., Shi et al.; this is background math the paper relies on without new proof.
  • domain assumption The shift operation adapted from DiffuLLaMA preserves the pretrained knowledge of the AR checkpoint while enabling bidirectional denoising.
    Referenced to Gong et al. 2025a; no analysis in this paper verifies the knowledge preservation claim.
  • domain assumption Context-adaptive token-level noise rescheduling improves training for code by assigning more noise to harder tokens.
    Referenced to Ye et al. 2025b; the paper reuses this without new validation.
  • ad hoc to paper Token-level importance ratios under an autoregressive factorization correctly optimize the diffusion policy in RL.
    The GRPO objective in Section 4.2 uses pi_theta(o_t | q, o_<t) ratios, which assumes sequential token generation; for a diffusion policy that generates tokens in parallel under masks, this factorization is not the model's generation distribution.
  • domain assumption The three benchmark families (HumanEval, MBPP, LiveCodeBench, BigCodeBench, CRUXEval) measure the code generation capabilities the paper claims.
    Benchmarks are standard in the field; no new evaluation methodology is introduced.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dream-Coder 7B: An Open Diffusion Language Model for Code." pith.science (2026). https://pith.science/paper/UWBHT3CV

@misc{pith2026250901142,
  author       = {Pith},
  title        = {Pith review of: Dream-Coder 7B: An Open Diffusion Language Model for Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UWBHT3CV}},
  note         = {Machine review of arXiv:2509.01142}
}
read the original abstract

We present Dream-Coder 7B, an open-source discrete diffusion language model for code generation that exhibits emergent any-order generation capabilities. Unlike traditional autoregressive (AR) models that decode strictly left-to-right, Dream-Coder 7B adaptively determines its decoding strategy based on the coding task: sketch-first generation for complex algorithms, left-to-right generation for straightforward completions, and interleaved reasoning generation for code understanding tasks. We adapt a pretrained AR checkpoint to a discrete diffusion frameworks with a continuous-time weighted cross-entropy objective. Our post-training recipe comprises (i) supervised fine-tuning, where we mitigate padding pathologies via random truncation and a padding penalty to improve sample efficiency and stabilize generation; and (ii) reinforcement learning with verifiable rewards over a curated high-quality prompt set drawn from open-source datasets, using a tailored reinforcement learning recipe for diffusion language models. The resulting Dream-Coder 7B Instruct attains 21.4\% pass@1 on LiveCodeBench (2410--2505) and demonstrates competitive performance on HumanEval, MBPP, BigCodeBench, and CRUXEval. We release Dream-Coder-7B and Dream-Coder-7B-Instruct checkpoints, training recipes, preprocessing pipelines, and inference code to facilitate reproducibility and further research.

Figures

Figures reproduced from arXiv: 2509.01142 by the authors.

Figure 2
Figure 2. Overview of Dream-Coder 7B post-training pipeline showing supervised fine-tuning on [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Generation patterns exhibited by Dream-Coder 7B Instruct across different coding tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 25 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MaskForge: Structure-Aware Adaptive Attacks for Jailbreaking Diffusion Large Language Models

    cs.CR 2026-06 unverdicted novelty 7.0 of 10

    MaskForge reaches 79.3% average attack success rate on five dLLMs by adaptively searching and accumulating structural attack patterns with a UCB bandit, improving 17.6% over baselines and transferring to 88.2% on AdvBench.

  2. Constrained Code Generation with Discrete Diffusion

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    Constrained Diffusion for Code (CDC) integrates constraint satisfaction into the reverse denoising process of discrete diffusion models via constraint-aware operators that use optimization and program analysis to stee...

  3. Trajectory as the Teacher: Few-Step Discrete Flow Matching via Energy-Navigated Distillation

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Energy-navigated trajectory shaping during training produces 8-step discrete flow matching students that achieve 32% lower perplexity than 1024-step teachers on 170M language models with unchanged inference cost.

  4. DARE: Diffusion Language Model Activation Reuse for Efficient Inference

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    DARE reuses up to 87% of attention activations in diffusion LLMs through KV caching and output reuse, delivering 1.2x per-layer latency gains with average performance drops of 1.2-2.0%.

  5. Discrete Tilt Matching

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    Discrete Tilt Matching recasts dLLM fine-tuning as state-level matching of tilted local unmasking posteriors, producing a stable weighted cross-entropy loss that improves Sudoku and Countdown performance when applied ...

  6. Discrete Tilt Matching

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    DTM recasts dLLM fine-tuning as weighted cross-entropy matching of tilted local posteriors, with demonstrated gains on Sudoku and math tasks.

  7. DMax: Aggressive Parallel Decoding for dLLMs

    cs.LG 2026-04 conditional novelty 7.0 of 10

    DMax uses On-Policy Uniform Training and Soft Parallel Decoding to enable aggressive parallelism in dLLMs, raising TPF on GSM8K from 2.04 to 5.47 and on MBPP from 2.71 to 5.86 while preserving accuracy.

  8. Attention-Based Sampler for Diffusion Language Models

    cs.CL 2026-03 conditional novelty 7.0 of 10

    Attn-Sampler decodes diffusion language models by selecting tokens in descending order of attention column sums, yielding higher quality and more parallel generation than token-level greedy baselines.

  9. The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models

    cs.CL 2026-01 conditional novelty 7.0 of 10

    Arbitrary-order generation lowers the reasoning Pass@k of diffusion LLMs; constraining RL to autoregressive order (JustGRPO) improves accuracy and preserves parallel decoding.

  10. Sangam: Efficiently Serving Diffusion LLMs with the AR Stack

    cs.DC 2026-07 conditional novelty 6.5 of 10

    Deficit token-budget scheduling plus hybrid prefill overflow makes AR-style colocated and disaggregated serving work for cached diffusion LLMs with block-sized, recurring prefills.

  11. Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

    cs.CL 2026-07 accept novelty 6.0 of 10

    Joint AR–diffusion training yields one tri-mode LM that switches AR, diffusion, and self-speculation, beating open AR/diffusion models on accuracy and tokens-per-forward.

  12. DiPOD: Diffusion Policy Optimization without Drifting Apart

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    DiPOD stabilizes diffusion policy optimization by interleaving self-distillation with gradient updates via an on-policy ELBO regularizer, yielding more stable training and higher rewards than prior methods.

  13. SimSD: Simple Speculative Decoding in Diffusion Language Models

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    SimSD adds a masking strategy to enable speculative decoding in diffusion LLMs, delivering up to 7.46x throughput gains on SDAR models while preserving generation quality.

  14. Understanding and Accelerating the Training of Masked Diffusion Language Models

    cs.LG 2026-05 conditional novelty 6.0 of 10

    Bell-shaped time sampling accelerates masked diffusion language model training by roughly 4x on LM1B by countering locality bias in language data.

  15. Understanding and Accelerating the Training of Masked Diffusion Language Models

    cs.LG 2026-05 conditional novelty 6.0 of 10

    Bell-shaped time sampling — drawing the corruption level near t=0.5 — accelerates masked diffusion language model training by up to ~4× without changing the final loss.

  16. FlowLM: Few-Step Language Modeling via Diffusion-to-Flow Adaptation

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    FlowLM converts diffusion LMs to flow matching via fine-tuning, achieving few-step generation that rivals or beats 2000-step diffusion and saturates faster than training flow models from scratch.

  17. UnMaskFork: Test-Time Scaling for Masked Diffusion via Deterministic Action Branching

    cs.LG 2026-02 conditional novelty 6.0 of 10

    UnMaskFork applies MCTS with deterministic cross-model unmasking actions and node caching to outperform existing test-time scaling baselines for masked diffusion language models on code and math tasks.

  18. Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed

    cs.CL 2025-12 unverdicted novelty 6.0 of 10

    Efficient-DLM converts AR models to dLMs via block-wise causal attention and position-dependent masking, yielding higher accuracy and 2.7-4.5x throughput than Dream 7B and Qwen3 4B.

  19. LLaDA2.0: Scaling Up Diffusion Language Models to 100B

    cs.LG 2025-12 conditional novelty 6.0 of 10

    LLaDA2.0 scales discrete diffusion language models to 100B parameters via systematic conversion from autoregressive models using a 3-phase WSD training scheme and releases open-source 16B and 100B MoE variants.

  20. TACG: Trajectory-Aware Commit Gating for Diffusion Language Model Decoding

    cs.CL 2026-07 conditional novelty 5.5 of 10

    Trajectory-aware commit gating (TILG + History Gate + capped extra promotion) improves or preserves DLLM accuracy while reducing steps and raising tokens-per-forward without retraining.

  21. Attention-Based Sampler for Diffusion Language Models

    cs.CL 2026-03 conditional novelty 5.5 of 10

    Sampling diffusion LMs by descending attention column sums optimally approximates NP-hard log-likelihood maximization and yields a training-free Attn-Sampler with better quality and parallelism.

  22. DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    DACA-GRPO adds denoising-aware credit assignment and bias-reduced likelihood estimation to GRPO, delivering consistent gains up to 36.3pp on math, code, constraint, and schema benchmarks for diffusion LLMs.

  23. DMax: Aggressive Parallel Decoding for dLLMs

    cs.LG 2026-04 unverdicted novelty 5.0 of 10

    DMax enables faster parallel decoding in diffusion language models by using on-policy training to recover from errors and soft embedding interpolations for iterative revision, boosting tokens per forward pass roughly ...

  24. Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Code

    cs.SE 2026-04 accept novelty 5.0 of 10

    A survey of diffusion, world-model, and state-space architectures for code that frames them as routes past autoregressive bottlenecks toward System-2 coding agents.

  25. Beyond Execution: Static-Analysis Rewards and Hint-Conditioned Diffusion RL for Code Generation

    cs.SE 2026-05 unverdicted novelty 4.0 of 10

    Static checking rewards and moderate AST-based hints improve diffusion RL performance for code generation, with effectiveness varying by task difficulty across HumanEval, MBPP, and LiveCodeBench.

Reference graph

Works this paper leans on

64 extracted references · 15 canonical work pages · cited by 21 Pith papers

  1. [1]

    Smollm2: When smol goes big -- data-centric training of a small language model, 2025

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werr...

    arXiv 2025

  2. [2]

    Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models

    Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models. 2025. URL https://hkunlp.github.io/blog/2025/Polaris

    work page 2025

  3. [3]

    System card: Claude opus 4 & claude sonnet 4, 2025

    Anthropic. System card: Claude opus 4 & claude sonnet 4, 2025. URL https://www-cdn.anthropic.com/07b2a3f9902ee19fe39a36ca638e5ae987bc64dd.pdf. System Card

    work page 2025

  4. [4]

    Structured denoising diffusion models in discrete state-spaces

    Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34: 0 17981--17993, 2021 a

    work page 2021

  5. [5]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021 b

    arXiv 2021

  6. [6]

    The pitfalls of next-token prediction

    Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction. In International Conference on Machine Learning, pages 2296--2318. PMLR, 2024

    work page 2024

  7. [7]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, 2020

    2020

  8. [8]

    A continuous time framework for discrete denoising models

    Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35: 0 28266--28279, 2022

    2022

Show all 64 references
  1. [9]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  2. [10]

    Killian, Mikhail Yurochkin, Zhengzhong Liu, Eric P

    Zhoujun Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Yonghao Zhuang, Nilabjo Dey, Yuheng Zha, Yi Gu, Kun Zhou, Yuqi Wang, Yuan Li, Richard Fan, Jianshu She, Chengqian Gao, Abulhair Saparov, Haonan Li, Taylor W. Killian, Mikhail Yurochkin, Zhengzh...

    2025 arXiv

  3. [11]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  4. [12]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  5. [13]

    Every sample matters: Leveraging mixture-of-experts and high-quality data for efficient and accurate code llm, 2025

    Codefuse and Ling Team. Every sample matters: Leveraging mixture-of-experts and high-quality data for efficient and accurate code llm, 2025. URL https://arxiv.org/abs/2503.17793

    2025 arXiv

  6. [14]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabil...

    2025 arXiv

  7. [15]

    Gemini diffusion

    DeepMind. Gemini diffusion. 2025. URL https://deepmind.google/models/gemini-diffusion/

    2025

  8. [16]

    Competitive programming with large reasoning models

    Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, et al. Competitive programming with large reasoning models. arXiv preprint arXiv:2502.06807, 2025

    2025 arXiv

  9. [17]

    DiffuSeq : Sequence to sequence text generation with diffusion models

    Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. DiffuSeq : Sequence to sequence text generation with diffusion models. In International Conference on Learning Representations, ICLR, 2023

    2023

  10. [18]

    Scaling diffusion language models via adaptation from autoregressive models

    Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling diffusion language models via adaptation from autoregressive models. International Conference on Learning Representati...

    2025

  11. [19]

    Diffucoder: Understanding and improving masked diffusion models for code generation

    Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. Diffucoder: Understanding and improving masked diffusion models for code generation. arXiv preprint arXiv:2506.20639, 2025 b

    2025 arXiv

  12. [20]

    Cruxeval: A benchmark for code reasoning, understanding and execution

    Alex Gu, Baptiste Rozi \`e re, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065, 2024

    2024 arXiv

  13. [21]

    Likelihood-based diffusion language models

    Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models. Advances in Neural Information Processing Systems, 36: 0 16693--16715, 2023

    2023

  14. [22]

    Deepseek-coder: When the large language model meets programming--the rise of code intelligence

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  15. [23]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  16. [24]

    Argmax flows and multinomial diffusion: Learning categorical distributions

    Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forr \'e , and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in neural information processing systems, 34: 0 12454--12465, 2021

    2021

  17. [25]

    Accelerating diffusion language model inference via efficient kv caching and guided diffusion

    Zhanqiu Hu, Jian Meng, Yash Akhauri, Mohamed S Abdelfattah, Jae-sun Seo, Zhiru Zhang, and Udit Gupta. Accelerating diffusion language model inference via efficient kv caching and guided diffusion. arXiv preprint arXiv:2505.21467, 2025

    2025

  18. [26]

    Siming Huang, Tianhao Cheng, J. K. Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, and Wei Chu. Opencoder: The open cookbook for top-tier co...

    2025 arXiv

  19. [27]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  20. [28]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  21. [29]

    Mercury: Ultra-fast language models based on diffusion

    Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, Stefano Ermon, et al. Mercury: Ultra-fast language models based on diffusion. arXiv e-prints, pages arXiv--2506, 2025

    2025

  22. [30]

    Variational diffusion models

    Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34: 0 21696--21707, 2021

    2021

  23. [31]

    Race: Large-scale reading comprehension dataset from examinations

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683, 2017

    2017 arXiv

  24. [32]

    Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan ...

    2024

  25. [33]

    Codei/o: Condensing reasoning patterns via code input-output prediction

    Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. Codei/o: Condensing reasoning patterns via code input-output prediction. arXiv preprint arXiv:2502.07316, 2025

    2025 arXiv

  26. [34]

    Diffusion-lm improves controllable text generation

    Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B Hashimoto. Diffusion-lm improves controllable text generation. In Conference on Neural Information Processing Systems, NeurIPS, 2022

    2022

  27. [35]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36: 0 21558--21572, 2023

    2023

  28. [36]

    Fullstack bench: Evaluating llms as full stack coder

    Siyao Liu, He Zhu, Jerry Liu, Shulin Xin, Aoyan Li, Rui Long, Li Chen, Jack Yang, Jinxiang Xia, ZY Peng, et al. Fullstack bench: Evaluating llms as full stack coder. arXiv preprint arXiv:2412.00535, 2024

    2024 arXiv

  29. [37]

    dllm-cache: Accelerating diffusion large language models with adaptive caching

    Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. dllm-cache: Accelerating diffusion large language models with adaptive caching. arXiv preprint arXiv:2506.06295, 2025

    2025 arXiv

  30. [38]

    Discrete diffusion language modeling by estimating the ratios of the data distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning, ICML, 2024

    2024

  31. [39]

    Starcoder 2 and the stack v2: The next generation, 2024

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Z...

    2024

  32. [40]

    Deepcoder: A fully open-source 14b coder at o3-mini level

    Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open-source 14b coder at o3-mini level. https://pretty-radio-b75.notion.si...

    2025

  33. [41]

    dkv-cache: The cache for diffusion language models

    Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models. arXiv preprint arXiv:2505.15781, 2025

    2025 arXiv

  34. [42]

    Llm critics help catch llm bugs

    Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Llm critics help catch llm bugs. arXiv preprint arXiv:2407.00215, 2024

    2024 arXiv

  35. [44]

    Large language diffusion models

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025 b

    2025 arXiv

  36. [45]

    Openai o3 and o4-mini system card, 2025

    OpenAI. Openai o3 and o4-mini system card, 2025. URL https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf. System Card

    2025

  37. [46]

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data

    Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. International Conference on Learning Representations, 2025

    2025

  38. [47]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydl \' c ek, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37: 0 30811--30849, 2024

    2024

  39. [48]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  40. [49]

    Code llama: Open foundation models for code

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  41. [50]

    Simple and effective masked diffusion language models

    Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Marroquin, Alexander M Rush, Yair Schiff, Justin T Chiu, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In The Thirty-eighth Annual Conference on Neural Information Processing ...

    2024

  42. [51]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  43. [52]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  44. [53]

    Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and generalized masked diffusion for discrete data. Neural Information Processing Systems, 2024

    2024

  45. [54]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  46. [55]

    Kimi k2: Open agentic intelligence

    Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025

    2025 arXiv

  47. [56]

    Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution

    Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502.18449, 2025

    2025 arXiv

  48. [57]

    Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding, 2025

    Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding, 2025. URL https://arxiv.org/abs/2505.22618

    2025 arXiv

  49. [58]

    Teaching language models to critique via reinforcement learning

    Zhihui Xie, Liyu Chen, Weichao Mao, Jingjing Xu, Lingpeng Kong, et al. Teaching language models to critique via reinforcement learning. arXiv preprint arXiv:2502.03492, 2025

    2025

  50. [59]

    Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding

    Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding. arXiv preprint arXiv:2503.02951, 2025

    2025 arXiv

  51. [60]

    Beyond autoregression: Discrete diffusion for complex reasoning and planning

    Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning. International Conference on Learning Representations, 2025 a

    2025

  52. [61]

    Dream 7b, 2025 b

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025 b . URL https://hkunlp.github.io/blog/2025/dream

    2025

  53. [62]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  54. [63]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  55. [64]

    A reparameterized discrete diffusion model for text generation

    Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.05737, 2023

    2023 arXiv

  56. [65]

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877, 2024

    2024 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.