Pith. sign in

REVIEW 26 cited by

d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2504.12216 v2 pith:ULCPVUMB submitted 2025-04-16 cs.CL cs.LG

Siyan Zhao , Devaansh Gupta , Qinqing Zheng , Aditya Grover This is my paper · ORCID
classification cs.CLcs.LG
keywords dllmsreasoninglanguagemodelsperformancelargemaskedrecent
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent large language models (LLMs) have demonstrated strong reasoning capabilities that benefits from online reinforcement learning (RL). These capabilities have primarily been demonstrated within the left-to-right autoregressive (AR) generation paradigm. In contrast, non-autoregressive paradigms based on diffusion generate text in a coarse-to-fine manner. Although recent diffusion-based large language models (dLLMs) have achieved competitive language modeling performance compared to their AR counterparts, it remains unclear if dLLMs can also leverage recent advances in LLM reasoning. To this end, we propose d1, a framework to adapt pre-trained masked dLLMs into reasoning models via a combination of supervised finetuning (SFT) and RL. Specifically, we develop and extend techniques to improve reasoning in pretrained dLLMs: (a) we utilize a masked SFT technique to distill knowledge and instill self-improvement behavior directly from existing datasets, and (b) we introduce a novel critic-free, policy-gradient based RL algorithm called diffu-GRPO, the first integration of policy gradient methods to masked dLLMs. Through empirical studies, we investigate the performance of different post-training recipes on multiple mathematical and planning benchmarks. We find that d1 yields the best performance and significantly improves performance of a state-of-the-art dLLM. Our code is released at https://dllm-reasoning.github.io/.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 26 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models

    cs.CL 2026-01 conditional novelty 7.0 of 10

    Arbitrary-order generation lowers the reasoning Pass@k of diffusion LLMs; constraining RL to autoregressive order (JustGRPO) improves accuracy and preserves parallel decoding.

  2. AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A continuous-latent diffusion language model that keeps a full-width decodable latent and adapts the denoiser (low-rank noisy input, width-calibrated noise, trajectory consistency) beats compared diffusion/continuous ...

  3. Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMs

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Evolutionary Decoding, a selection-and-mutation test-time strategy, improves LLaDA 2.0 mathematical reasoning on AIME and AMC benchmarks.

  4. From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Masked diffusion language models fail to exploit their any-order interface because of positional uncertainty; insertion-based (FlexMDM) and latent-segment (LatentMDM) variants recover distinct any-order inference beha...

  5. Trace-Based On-Policy Distillation for Masked Diffusion Language Models

    cs.CL 2026-07 conditional novelty 6.0 of 10

    TOPD distills a stronger diffusion LLM into a weaker one on the student's own denoising trajectories and matches a reward-trained 4B model on MATH500 with 4x fewer rollouts.

  6. Hierarchical Domain Generalization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Over infinite domains, hierarchy-uniform domain generalization is impossible for every nontrivial hypothesis class; a length-generalization bound is a property of the length hierarchy, not a hierarchy-free guarantee.

  7. Concurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Processes

    cs.LG 2026-07 conditional novelty 6.0 of 10

    CO2Jump couples text and image denoising through cross-modal attention and remasking, achieving best joint accuracy on three concurrent-generation tasks.

  8. Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

    cs.CL 2026-07 accept novelty 6.0 of 10

    Joint AR–diffusion training yields one tri-mode LM that switches AR, diffusion, and self-speculation, beating open AR/diffusion models on accuracy and tokens-per-forward.

  9. LightningRL: Breaking the Accuracy-Parallelism Trade-off of Block-wise dLLMs via Reinforcement Learning

    cs.LG 2026-03 conditional novelty 6.0 of 10

    A GRPO-based post-training method for block-wise diffusion LLMs improves average tokens-per-forward from ~3 to 7.3 while preserving accuracy.

  10. ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs

    cs.LG 2025-10 conditional novelty 6.0 of 10

    Parallel decoding in diffusion LLMs degrades quality on token-dependent tasks, and existing strategies cannot adapt parallelism to preserve accuracy, as shown by the new 17-task ParallelBench benchmark.

  11. Inpainting-Guided Policy Optimization for Diffusion Large Language Models

    cs.LG 2025-09 reject novelty 6.0 of 10

    IGPO injects partial ground-truth reasoning hints into masked diffusion LLM generation during RL sampling, restoring non-zero advantages and improving math benchmarks, but as written the RL training set overlaps the e...

  12. Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models

    cs.CL 2025-09 conditional novelty 6.0 of 10

    TraceRL aligns the RL objective with the DLM's actual step-by-step decoding, producing TraDo-4B/8B models that beat autoregressive baselines on math reasoning.

  13. Any-Order Flexible Length Masked Diffusion

    cs.LG 2025-08 conditional novelty 6.0 of 10

    FlexMDM is a discrete diffusion model that provably supports any-order generation over variable-length sequences by learning an insertion expectation alongside the unmasking posterior, validated by length-fidelity, ma...

  14. ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Compute

    cs.CL 2025-08 conditional novelty 6.0 of 10

    ParaThinker trains LLMs for native parallel reasoning and reports 7 to 12 percent higher accuracy on math benchmarks over sequential thinking with modest latency overhead.

  15. DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 7B masked-diffusion code model plus complementary-mask GRPO (coupled-GRPO) improves benchmark scores and shifts decoding away from strict left-to-right order.

  16. On a few pitfalls in KL divergence gradient estimation for RL

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Differentiating KL estimates as losses gives biased or reversed KL gradients; the paper derives and tests unbiased sequence-level estimators.

  17. SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards

    cs.AI 2025-06 conditional novelty 6.0 of 10

    SUDER uses the likelihood of reconstructing the original input from a sampled output as a self-reward, improving both understanding and generation in unified multimodal models without external supervision.

  18. Decomposing Elements of Problem Solving: What "Math" Does RL Teach?

    cs.AI 2025-05 conditional novelty 6.0 of 10

    Reinforcement learning (GRPO) on math LLMs primarily increases execution robustness on already-solvable problems, not planning or coverage of new problems.

  19. TACG: Trajectory-Aware Commit Gating for Diffusion Language Model Decoding

    cs.CL 2026-07 conditional novelty 5.5 of 10

    Trajectory-aware commit gating (TILG + History Gate + capped extra promotion) improves or preserves DLLM accuracy while reducing steps and raising tokens-per-forward without retraining.

  20. Can I Have Your Order? Monte-Carlo Tree Search for Slot Filling Ordering in Diffusion Language Models

    cs.AI 2026-02 reject novelty 5.0 of 10

    Monte Carlo tree search can pick better slot-filling orders for masked diffusion language models, but the claimed advantage over autoregressive models depends on truncating the autoregressive baselines.

  21. Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding

    cs.LG 2026-01 conditional novelty 5.0 of 10

    A training-free inference framework prunes suffix masks, adapts confidence thresholds, and early-exits at EOS to speed up diffusion LLM decoding by up to 68×.

  22. Simple Policy Gradients for Reasoning with Diffusion Language Models

    cs.LG 2025-10 reject novelty 5.0 of 10

    AGRPO makes GRPO-style policy gradients tractable for diffusion LLMs by Monte-Carlo sampling denoising timesteps, but the unbiasedness claim only holds for a step-level objective, not the token-level GRPO objective.

  23. Review, Remask, Refine (R3): Process-Guided Block Diffusion for Text Generation

    cs.CL 2025-07 conditional novelty 5.0 of 10

    R3 uses process reward model scores to decide which blocks of a masked diffusion language model's output to remask and regenerate, improving MATH accuracy from 29% to 43% without retraining.

  24. Boundary-Guided Policy Optimization for Memory-efficient RL of Diffusion Large Language Models

    cs.LG 2025-10 conditional novelty 4.0 of 10

    BGPO makes RL for diffusion LLMs memory-efficient by decomposing the ELBO-based objective into per-sample terms, enabling larger Monte Carlo sample sizes and better performance.

  25. A Survey on Latent Reasoning

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A survey that organizes latent reasoning methods into vertical recurrence, horizontal recurrence, and infinite-depth diffusion, arguing that silent reasoning can beat explicit chain-of-thought.

  26. A Survey on Diffusion Language Models

    cs.CL 2025-08 unverdicted novelty 3.0 of 10

    A comprehensive survey of diffusion language models covering taxonomy, training and inference techniques, and comparisons with autoregressive models.

Pith tools