Pith. sign in

REVIEW 3 major objections 6 minor 70 references

One model trained jointly for AR and diffusion can switch among autoregressive, parallel diffusion, and self-speculation decoding, matching strong open baselines while decoding about six tokens per forward.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 03:02 UTC pith:KA42B26C

load-bearing objection Solid systems packaging of joint AR–diffusion into a real tri-mode family with device numbers and a useful SOL ceiling; SOTA accuracy claims are only partly isolated from Ministral3 provenance. the 3 major comments →

arxiv 2607.05722 v1 pith:KA42B26C submitted 2026-07-07 cs.CL

Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

Yonggan Fu , Lexington Whalen , Abhinav Garg , Chengyue Wu , Maksim Khadkevich , Nicolai Oswald , Enze Xie , Daniel Egert
show 18 more authors
Sharath Turuvekere Sreenivas Shizhe Diao Chenhan Yu Ye Yu Weijia Chen Sajad Norouzi Jingyu Liu Shiyi Lan Ligeng Zhu Jin Wang Jindong Jiang Morteza Mardani Mehran Maghoumi Song Han Ante Juki\'c Nima Tajbakhsh Jan Kautz Pavlo Molchanov
This is my paper
classification cs.CL
keywords tri-mode language modeljoint AR-diffusion trainingblock diffusionself-speculation decodingtokens per forwardspeed-of-light analysismulti-token prediction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that autoregressive and diffusion language modeling need not compete. A single network trained with a weighted joint next-token and block-diffusion objective can run in three inference modes: ordinary left-to-right decoding, block-wise parallel diffusion, and self-speculation in which diffusion drafts and AR verifies. The authors claim the two losses are complementary—diffusion strengthens lookahead planning while AR supplies left-to-right linguistic priors—and that self-speculation already beats multi-token-prediction heads on acceptance length and measured throughput. A speed-of-light construction further claims that an ideal diffusion sampler could still deliver roughly 76 percent more real tokens per forward than today’s best self-speculation path. The resulting 3B/8B/14B base, instruct, and vision-language family is reported to match or exceed open AR and diffusion peers on accuracy while substantially raising tokens-per-forward and system throughput, so one set of weights can serve high-concurrency cloud and low-concurrency personal inference without architecture changes.

Core claim

Joint AR–diffusion training with a carefully chosen diffusion weight (α = 0.3), global loss averaging, and a two-stage AR-then-joint schedule produces a single model that fully preserves AR accuracy, supports native block diffusion, and enables high-acceptance self-speculation without auxiliary prediction heads; at 8B instruct scale this yields roughly 6× tokens per forward versus a comparable AR baseline at matched accuracy, with diffusion’s theoretical upper bound still substantially higher.

What carries the argument

The joint objective (AR next-token loss plus α times a block-wise diffusion denoising loss) together with a dual-stream attention pattern that keeps the clean stream strictly causal. That pattern lets both losses be computed in one forward–backward pass and, at inference, lets the same weights act as AR decoder, diffusion denoiser, or diffusion drafter plus AR verifier.

Load-bearing premise

That the reported gains over open AR, diffusion, and multi-token-prediction baselines come mainly from the joint objective and tri-mode inference rather than from differences in pretraining data volume, recipe, and evaluation harness.

What would settle it

A controlled experiment that continuous-pretrains and SFT-matches an otherwise identical AR-only baseline on the exact same token budget, data mixture, and evaluation harness as the joint model; if the joint model then loses its accuracy or tokens-per-forward advantage, the complementarity claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents Nemotron-Labs-Diffusion, a family of 3B/8B/14B language models (base, instruct, and VLM) trained with a joint AR–diffusion objective (Eq. 3, α=0.3) under a two-stage recipe and a dual-stream attention pattern that keeps the clean stream strictly causal. The resulting checkpoint supports three inference modes—standard AR, block-wise diffusion denoising (with optional learned sampler), and self-speculation (diffusion draft + AR verify; linear and quadratic variants, with optional LoRA draft alignment)—without architectural forks. Empirically, the 8B instruct model matches or exceeds Qwen3-8B accuracy while reporting ~6× tokens-per-forward under linear self-speculation and ~4× SPEED-Bench throughput vs Qwen3-8B-Eagle3 on GB200/SGLang; progressive ablations (Tab. 1–3), multi-scale tables (Tab. 5–9), acceptance-length comparisons to Eagle3/MTP (Tab. 10), and multi-GPU device measurements (Fig. 1, 9) support complementarity of the two losses and practical efficiency of self-speculation. A speed-of-light (SOL) analysis via recursive dynamic compaction estimates up to 76.5% more real TPF than linear self-speculation under an optimal diffusion sampler.

Significance. If the results hold under the stated training and evaluation conditions, the work is a substantial systems contribution: it packages AR, parallel diffusion, and self-speculation into one drop-in checkpoint that adapts across concurrency regimes, and it shows self-speculation can beat auxiliary-head MTP (Eagle3) in acceptance length and real-device throughput. The controlled 25B-token ablations and AR-with/without-diffusion SFT controls give credible evidence that AR and diffusion losses are complementary rather than zero-sum. The SOL construction and multi-GPU SPEED-Bench measurements are concrete, falsifiable artifacts that clarify headroom beyond current samplers. Model family release and Megatron Bridge pipeline further raise the work’s utility to the community.

major comments (3)
  1. [Sec. 5.1–5.2, Tab. 5–8] Sec. 5.1–5.2 and Tab. 5–8: Full-scale SOTA accuracy claims (e.g., NLD-8B vs Qwen3-8B / LLaDA / Dream / SDAR) rest on continuous pretraining from Ministral3 (1T pure-AR + 300B joint tokens) plus proprietary-style SFT, while baselines differ in data mixture, total tokens, and often evaluation harness (NeMo-Skills vs official diffusion pipelines). The 25B-token ablations (Tab. 1–3) and matched AR-only vs joint SFT (Tab. 3) isolate complementarity under controlled data, but they do not quantify what fraction of the headline accuracy/TPF/throughput gains at full scale is due to the joint objective and tri-mode inference versus stronger provenance. The manuscript should explicitly bound causal attribution for the strongest claim and, where possible, add a matched-data or matched-checkpoint comparison at a scale closer to the released models.
  2. [Sec. 4.2, Fig. 7, Abstract] Sec. 4.2 and Fig. 7: The 76.5% real-TPF advantage of SOL over linear self-speculation mixes two different correctness targets (SOL matches the diffusion mode’s own serial highest-confidence path; linear SS matches AR verification) and two different cost models (one vs two forwards, plus prefix-only acceptance). The paper notes this, but the abstract and intro still present 76.5% as a clean headroom figure for “diffusion under an optimal sampler.” Please restate the claim so that acceptance-rate proximity to SOL (~10% gap) and real-TPF gap (two-forward + prefix truncation) are separated, and clarify that SOL is not an AR-accuracy ceiling.
  3. [Sec. 6.1, Tab. 5] Tab. 5 and Sec. 6.1: Diffusion baselines are evaluated with their official pipelines while NLD and AR baselines use NeMo-Skills; decoding hyperparameters (block size, confidence thresholds, thinking vs non-thinking) are only partially aligned. For load-bearing accuracy comparisons against LLaDA/Dream/SDAR, report a sensitivity check under a single harness or document that residual gaps survive re-evaluation under the authors’ pipeline.
minor comments (6)
  1. [Fig. 1] Fig. 1(b) and caption: Symbol sizes for diffusion block sizes (8/16/32) are hard to read; add an explicit legend entry for block size and for Linear vs Quad SS.
  2. [Sec. 2.1, Eq. (2)] Eq. (2): The 1/t reweighting and its interaction with global vs sequence averaging (Eq. 4–5) is well motivated in text; a short note on whether t is continuous or discretized in practice would help reproducibility.
  3. [Tab. 6, Sec. 6.1] Tab. 6: Small accuracy differences between AR and self-speculation are attributed to “kernel mismatches between 1-token decoding and multi-token prefilling”; quantify or cite the kernel path so readers can judge whether this is numerical noise or a systematic bias.
  4. [Appendix A] Appendix A: Sampler feature list (144-d, PCA top-3, entropy, etc.) is useful; state the PCA basis source and whether features are frozen across model scales.
  5. [Sec. 7] Related work (Sec. 7): Cite and briefly position against concurrent joint AR–diffusion / set-block decoding lines already mentioned in Sec. 2.2 ([14], [7]) so novelty of tri-mode inference and SOL is sharper.
  6. [Front matter, Sec. 2.4] Typos / polish: “Y onggan F u”, “W u”, “T uruvekere”, “Y e Y u” spacing in author list; “AsshowninTab.3” missing spaces (Sec. 2.4); consistent “tokens per forward” vs “TPF” on first use in abstract.

Circularity Check

0 steps flagged

No load-bearing circularity; empirical claims and SOL bound are self-contained against external benchmarks and the model’s own serial path, with only routine self-cites of prior methods.

full rationale

The paper’s central claims (joint AR–diffusion complementarity at α=0.3, self-speculation acceptance/TPF gains over MTP/Eagle3, 76.5% SOL headroom, and SOTA accuracy–speed on 3B/8B/14B base/instruct/VLM variants) are established by direct measurement on external suites (HumanEval/MBPP/LCB, GSM8K/Math500/AIME, GPQA/IFEval/MMLU, SPEED-Bench, VLMEvalKit) and by controlled 25B-token ablations (Tabs. 1–3) that isolate global loss averaging, two-stage training, and the AR loss term. The SOL construction (Sec. 4) is an intentional oracle relative to the diffusion model’s own serial-denoising target t; the 76.5% figure is simply (6.02−3.41)/3.41 from measured TPF and is not presented as a first-principles derivation. α=0.3 and LoRA hyperparameters are selected by magnitude alignment and ablation, not re-labeled as predictions. Self-citations ([10], [13], [20]) appear only as related-method context for block diffusion or quadratic decoding and do not supply uniqueness theorems or force the headline results. Data-provenance differences versus Qwen3/LLaDA/Dream/SDAR/Eagle3 affect causal attribution of the full-scale SOTA numbers but do not constitute circular reduction of any equation or claim to its own inputs. Hence only a minimal residual score for ordinary self-reference.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

This is an empirical systems paper. Load-bearing choices are training hyperparameters (especially diffusion weight α), initialization from Ministral3, block-diffusion modeling assumptions, and the SOL oracle definition. No new physical entities; ‘invented’ items are algorithmic constructs introduced or specialized here.

free parameters (5)
  • diffusion loss coefficient α = 0.3
    Fixed to 0.3 after magnitude-alignment and a discrete sweep {0.1,0.2,0.3,0.5,1.0}; both modes peak there (Tab. 2). Central complementarity claim depends on this operating point.
  • two-stage token budgets (Stage1 AR / Stage2 joint) = 1T + 300B
    1T pure AR then 300B joint continuous pretraining from Ministral3; ablation shows two-stage and AR loss dominate gains (Tab. 1).
  • LoRA rank/α on o_proj for draft alignment = r=128, α_LoRA=512
    rank 128, α=512 (~0.4% params); drives reported linear-SS TPF (Tab. 6).
  • LK-hybrid / CE temperatures and top-K = τ=3, K=200
    τ=3.0, K=200, η=0.5, λ_KL=λ_CE=1, 90% sampled drafts—hand-set training recipe for acceptance matching.
  • diffusion block length B and confidence/sampler thresholds = B up to 32 (default SS drafts)
    B∈{4,8,16,32} and commit thresholds trade accuracy vs TPF (Fig. 1b, Tab. 4, Fig. 8); deployment numbers depend on these knobs.
axioms (5)
  • domain assumption Natural language has a strong left-to-right inductive bias that pure any-order diffusion wastes capacity on.
    Stated in Introduction and Sec. 2.1 as motivation for joint AR loss and causal clean-stream attention.
  • domain assumption Block-wise diffusion (bidirectional within block, causal across blocks) is a valid training/inference factorization that preserves KV-cache reuse.
    Adopted from [9,10]; Fig. 3 and Eq. 2.
  • domain assumption Strictly causal Clean→Clean attention prevents AR label leakage while allowing joint AR+diffusion loss in one pass.
    Sec. 2.2; follows [14] with stated differences.
  • ad hoc to paper Serial highest-confidence denoising defines the diffusion model’s ‘converged’ block target for SOL.
    Sec. 4.1 oracle construction; SOL percentages are relative to this definition.
  • domain assumption Matching AR-mode verification (or serial diffusion target) is a sufficient correctness criterion for multi-token commits.
    Self-speculation and SOL evaluation; Sec. 3.3 and 4.2.
invented entities (3)
  • Tri-mode Nemotron-Labs-Diffusion family (AR / diffusion / self-speculation in one checkpoint) independent evidence
    purpose: Unify deployment modes without separate draft heads or models.
    Core product of the paper; independent evidence is released weights and external-benchmark tables, not a new physical object.
  • Recursive dynamic compaction SOL procedure no independent evidence
    purpose: Estimate maximum safe parallel commits matching serial diffusion targets.
    Sec. 4.1–4.2 method used to claim 76.5% TPF headroom; exists only as analysis algorithm here.
  • Lightweight diffusion commit sampler (4-layer Transformer on PCA/top-k features) independent evidence
    purpose: Learn which masked positions are safe to commit vs confidence thresholding.
    Appendix A; improves Pareto frontier in Fig. 8.

pith-pipeline@v1.1.0-grok45 · 35057 in / 3869 out tokens · 44632 ms · 2026-07-11T03:02:27.210887+00:00 · methodology

0 comments
read the original abstract

We introduce Nemotron-Labs-Diffusion, a tri-mode language model (LM) that unifies AR, diffusion, and self-speculation decoding within a single architecture. Trained with a joint AR-diffusion objective, Nemotron-Labs-Diffusion can switch modes to sustain high throughput across deployment settings and concurrency levels. Our study shows that (1) AR and diffusion objectives are complementary: diffusion improves lookahead planning, while AR provides left-to-right linguistic priors. (2) In self-speculation mode, diffusion drafts while AR verifies, outperforming multi-token prediction (MTP) methods in both acceptance rate and real-device efficiency. (3) A speed-of-light analysis further demonstrates diffusion's long-term potential, with up to 76.5% more tokens per forward pass than self-speculation under an optimal sampler. Scaling to 3B, 8B, and 14B parameters, our Nemotron-Labs-Diffusion family, including base, instruct, and vision-language models, consistently outperforms state-of-the-art open-source AR and diffusion LMs in both accuracy and speed. For example, Nemotron-Labs-Diffusion-8B decodes 6x more tokens per forward than Qwen3-8B with comparable accuracy, translating to 4x higher throughput on SPEED-Bench with SGLang on a GB200 GPU.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

70 extracted references · 70 canonical work pages · 41 internal anchors

  1. [1]

    SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding

    Talor Abramovich, Maor Ashkenazi, Benjamin Chislett, Tiyasa Mitra, Bita Darvish Rouhani, Ran Zilberstein, Yonatan Geifman, et al. Speed-bench: A unified and diverse benchmark for speculative decod- ing.arXiv preprint arXiv:2604.09557, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  2. [2]

    Large Language Diffusion Models

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  3. [3]

    Dream 7B: Diffusion Large Language Models

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation.arXiv preprint arXiv:2510.06303, 2025

    Shuang Cheng, Yihan Bian, Dawei Liu, Linfeng Zhang, Qian Yao, Zhongbo Tian, Wenhai Wang, Qipeng Guo, Kai Chen, Biqing Qi, et al. Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation.arXiv preprint arXiv:2510.06303, 2025

    work page arXiv 2025

  5. [5]

    Scaling up Masked Diffusion Models on Text

    Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text.arXiv preprint arXiv:2410.18514, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [6]

    Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture

    Shuchen Xue, Tianyu Xie, Tianyang Hu, Zijin Feng, Jiacheng Sun, Kenji Kawaguchi, Zhenguo Li, and Zhi-Ming Ma. Any-order gpt as masked diffusion model: Decoupling formulation and architecture. arXiv preprint arXiv:2506.19935, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential

    Mohammad Samragh, Arnav Kundu, David Harri- son, Kumari Nishu, Devang Naik, Minsik Cho, and Mehrdad Farajtabar. Your llm knows the future: Un- covering its multi-token prediction potential.arXiv preprint arXiv:2507.11851, 2025. 16 Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  8. [8]

    EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time test.arXiv preprint arXiv:2503.01840, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  9. [9]

    Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models

    Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models.arXiv preprint arXiv:2503.09573, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  10. [10]

    Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed

    YongganFu, LexingtonWhalen, ZhifanYe, XinDong, Shizhe Diao, Jingyu Liu, Chengyue Wu, Hao Zhang, Enze Xie, Song Han, et al. Efficient-dlm: From au- toregressive to diffusion language models, and beyond in speed.arXiv preprint arXiv:2512.14067, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  11. [11]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open- source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [13]

    Fast-dllm v2: Efficient block-diffusion llm.arXiv preprint arXiv:2509.26328, 2025

    Chengyue Wu, Hao Zhang, Shuchen Xue, Shizhe Diao, Yonggan Fu, Zhijian Liu, Pavlo Molchanov, Ping Luo, Song Han, and Enze Xie. Fast-dllm v2: Efficient block-diffusion llm.arXiv preprint arXiv:2509.26328, 2025

    work page arXiv 2025

  14. [14]

    Set Block Decoding is a Language Model Inference Accelerator

    Itai Gat, Heli Ben-Hamu, Marton Havasi, Daniel Haz- iza, Jeremy Reizenstein, Gabriel Synnaeve, David Lopez-Paz, Brian Karrer, and Yaron Lipman. Set block decoding is a language model inference acceler- ator.arXiv preprint arXiv:2509.04185, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  15. [15]

    Deepseek-v3 technical report, 2024

    DeepSeek-AI. Deepseek-v3 technical report, 2024

    work page 2024

  16. [16]

    Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding

    Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding.arXiv preprint arXiv:2505.22618, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  17. [17]

    Fast inference from transformers via speculative de- coding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative de- coding. InInternational Conference on Machine Learning, pages 19274–19286. PMLR, 2023

    work page 2023

  18. [18]

    Lora: Low-rank adaptation of large language models.Iclr, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.Iclr, 1(2):3, 2022

    work page 2022

  19. [19]

    LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding

    Aleksei Samarin et al. Lk losses: Direct acceptance rate optimization for speculative decoding.arXiv preprint arXiv:2602.23881, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  20. [20]

    Tidar: Think in diffusion, talk in autoregression.arXiv preprint arXiv:2511.08923, 2025

    Jingyu Liu, Xin Dong, Zhifan Ye, Rishabh Mehta, Yonggan Fu, Vartika Singh, Jan Kautz, Ce Zhang, and Pavlo Molchanov. Tidar: Think in diffusion, talk in autoregression.arXiv preprint arXiv:2511.08923, 2025

    work page arXiv 2025

  21. [21]

    Ministral 3

    Alexander H Liu, Kartik Khandelwal, Sandeep Sub- ramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, et al. Ministral 3.arXiv preprint arXiv:2601.08584, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  22. [22]

    NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model

    Aarti Basant, Abhijit Khairnar, Abhijit Paithankar, Abhinav Khattar, Adithya Renduchintala, Aditya Malte, Akhiad Bercovich, Akshay Hazare, Ale- jandra Rico, Aleksander Ficek, et al. Nvidia nemotron nano 2: An accurate and efficient hybrid mamba-transformer reasoning model.arXiv preprint arXiv:2508.14444, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  23. [23]

    MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yux- iang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies.arXiv preprint arXiv:2404.06395, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  24. [24]

    Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

    Aakshita Chandiramani, Aaron Blakeman, Abdul- lahi Olaoye, Abhibha Gupta, Abhilash Somasamu- dramath, Abhinav Khattar, Adeola Adesoba, Adi Renduchintala, Adil Asif, Aditya Agrawal, et al. Nemotron 3 super: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reason- ing.arXiv preprint arXiv:2604.12374, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  25. [25]

    Finevi- sion: Open data is all you need, 2025

    Luis Wiedmann, Orr Zohar, Amir Mahla, Xiaohan Wang, Rui Li, Thibaud Frere, Leandro von Werra, Aritra Roy Gosthipaty, and Andrés Marafioti. Finevi- sion: Open data is all you need, 2025

    work page 2025

  26. [26]

    Nemo-skills: A toolkit for improving skills of large language models

    NVIDIA Corporation. Nemo-skills: A toolkit for improving skills of large language models. https: //github.com/NVIDIA-NeMo/Skills, 2024. GitHub repository

    work page 2024

  27. [27]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min- joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. InEuropean Con- ference on Computer Vision (ECCV), pages 235–251, 2016

    work page 2016

  28. [28]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. InFindings of the Association for Computational Linguistics: ACL 2022, pages 2263– 2279, 2022

    work page 2022

  29. [29]

    Minesh Mathew, Dimosthenis Karatzas, and C.V. Jawahar. Docvqa: A dataset for vqa on document images. InIEEE Winter Conference on Applications of Computer Vision (W ACV), pages 2200–2209, 2021. 17 Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

    work page 2021

  30. [30]

    Mmmu: A mas- sive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A mas- sive multi-discipline multimodal understanding and reasoning benchmark for e...

    work page 2024

  31. [31]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun- yuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. InInternational Confer- ence on Learning Representations (ICLR), 2024

    work page 2024

  32. [32]

    Realworldqa, 2024

    xAI. Realworldqa, 2024

    work page 2024

  33. [33]

    Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. InProceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (ACL), 2025

    work page 2025

  34. [34]

    MMaDA: Multimodal Large Diffusion Language Models

    Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models.arXiv preprint arXiv:2505.15809, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  35. [35]

    LaViDa: A Large Diffusion Language Model for Multimodal Understanding

    Shufan Li, Konstantinos Kallidromitis, Hritik Bansal, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Ja- son Kuen, Zhe Lin, Kai-Wei Chang, and Aditya Grover. Lavida: A large diffusion language model for multimodal understanding.arXiv preprint arXiv:2505.16839, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning

    Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, and Chongxuan Li. Llada-v: Large language diffusion models with visual instruction tuning.arXiv preprint arXiv:2505.16933, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  37. [37]

    Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding

    Runpeng Yu, Xinyin Ma, and Xinchao Wang. Dim- ple: Discrete diffusion multimodal large language model with parallel decoding.arXiv preprint arXiv:2505.16990, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  38. [38]

    VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models

    Haodong Duan, Xinyu Fang, Junming Yang, Xiangyu Zhao, Yuxuan Qiao, Mo Li, Amit Agarwal, Zhe Chen, Lin Chen, Yuan Liu, Yubo Ma, Hailong Sun, Yifan Zhang, Shiyin Lu, Tack Hwa Wong, Weiyun Wang, Peiheng Zhou, Xiaozhe Li, Chaoyou Fu, Junbo Cui, Jixuan Chen, Enxin Song, Song Mao, Shengyuan Ding, Tianhao Liang, Zicheng Zhang, Xiaoyi Dong, Yuhang Zang, Pan Zhang...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  39. [39]

    Diffusion-lm improves controllable text generation.Advances in neural information processing systems, 35:4328–4343, 2022

    Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-lm improves controllable text generation.Advances in neural information processing systems, 35:4328–4343, 2022

    work page 2022

  40. [40]

    DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models

    ShansanGong, MukaiLi, JiangtaoFeng, ZhiyongWu, and LingPeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models.arXiv preprint arXiv:2210.08933, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  41. [41]

    SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control

    Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular con- trol.arXiv preprint arXiv:2210.17432, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  42. [42]

    Structured denoising diffusion models in discrete state-spaces

    Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:17981–17993, 2021

    work page 2021

  43. [43]

    DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models

    Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuan- jing Huang, and Xipeng Qiu. Diffusionbert: Improv- ing generative masked language models with diffusion models.arXiv preprint arXiv:2211.15029, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  44. [44]

    Simple and effec- tive masked diffusion language models.Advances in Neural Information Processing Systems, 37:130136– 130184, 2024

    Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effec- tive masked diffusion language models.Advances in Neural Information Processing Systems, 37:130136– 130184, 2024

    work page 2024

  45. [45]

    Simplified and generalized masked diffusion for discrete data.Advances in neural information processing systems, 37:103131–103167, 2024

    Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data.Advances in neural information processing systems, 37:103131–103167, 2024

    work page 2024

  46. [46]

    Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ra- tios of the data distribution.arXiv preprint arXiv:2310.16834, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  47. [47]

    Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data

    Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the con- ditional distributions of clean data.arXiv preprint arXiv:2406.03736, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  48. [48]

    Esoteric Language Models: A Family of Any-Order Diffusion LLMs

    Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, and Arash Vahdat. Esoteric language models.arXiv preprint arXiv:2506.01928, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  49. [49]

    The diffusion duality.arXiv preprint arXiv:2506.10892, 2025

    Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin Chiu, and Volodymyr Kuleshov. The diffusion duality.arXiv preprint arXiv:2506.10892, 2025

    work page arXiv 2025

  50. [50]

    Tiwei Bie, Maosong Cao, Kun Chen, Lun Du, Min- gliang Gong, Zhuochen Gong, Yanmei Gu, Jiaqi Hu, 18 Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding Zenan Huang, Zhenzhong Lan, et al. Llada2. 0: Scal- ing up diffusion language models to 100b.arXiv preprint arXiv:2512.15745, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  51. [51]

    Tiwei Bie, Maosong Cao, Xiang Cao, Bingsen Chen, Fuyuan Chen, Kun Chen, Lun Du, Daozhuo Feng, Haibo Feng, Mingliang Gong, et al. Llada2. 1: Speed- ing up text diffusion via token editing.arXiv preprint arXiv:2602.08676, 2026

    work page arXiv 2026

  52. [52]

    Gemini diffusion, 2025

    Google DeepMind. Gemini diffusion, 2025. Model page: state-of-the-art, experimental text diffusion model

    work page 2025

  53. [53]

    Mercury: Ultra-Fast Language Models Based on Diffusion

    Samar Khanna, Siddhant Kharbanda, Shufan Li, HarshitVarma, EricWang, SawyerBirnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, Stefano Ermon, et al. Mercury: Ultra-fast language models based on diffusion.arXiv preprint arXiv:2506.17298, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  54. [54]

    Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference

    Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, et al. Seed diffusion: A large-scale diffusion language model with high-speed inference.arXiv preprint arXiv:2508.02193, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  55. [55]

    DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation

    Shansan Gong, Ruixiang Zhang, Huangjie Zheng, JiataoGu, NavdeepJaitly, LingpengKong, andYizhe Zhang. Diffucoder: Understanding and improving masked diffusion models for code generation.arXiv preprint arXiv:2506.20639, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  56. [56]

    Dream-Coder 7B: An Open Diffusion Language Model for Code

    Zhihui Xie, Jiacheng Ye, Lin Zheng, Jiahui Gao, Jingwei Dong, Zirui Wu, Xueliang Zhao, Shansan Gong, Xin Jiang, Zhenguo Li, et al. Dream-coder 7b: An open diffusion language model for code.arXiv preprint arXiv:2509.01142, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  57. [57]

    d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning

    Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. d1: Scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint arXiv:2504.12216, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  58. [58]

    LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models

    Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, ChunweiWu, JunHu, JunZhou, JianfeiChen, Yankai Lin, Ji-Rong Wen, et al. Llada 1.5: Variance- reduced preference optimization for large language diffusion models.arXiv preprint arXiv:2505.19223, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  59. [59]

    Diffusionbeats autoregressive in data-constrained settings.arXiv preprint arXiv:2507.15857, 2025

    MihirPrabhudesai, MengningWu, AmirZadeh, Kate- rinaFragkiadaki, andDeepakPathak. Diffusionbeats autoregressive in data-constrained settings.arXiv preprint arXiv:2507.15857, 2025

    work page arXiv 2025

  60. [60]

    Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective

    Siyue Zhang, Yilun Zhao, Liyuan Geng, Arman Co- han, Anh Tuan Luu, and Chen Zhao. Diffusion vs. autoregressive language models: A text embedding perspective.arXiv preprint arXiv:2505.15045, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  61. [61]

    dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching

    Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. dllm-cache: Accelerating diffu- sion large language models with adaptive caching. arXiv preprint arXiv:2506.06295, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  62. [62]

    dKV-Cache: The Cache for Diffusion Language Models

    Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models.arXiv preprint arXiv:2505.15781, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  63. [63]

    Accelerating diffusion llms via adaptive paral- lel decoding.arXiv preprint arXiv:2506.00413, 2025

    Daniel Israel, Guy Van den Broeck, and Aditya Grover. Accelerating diffusion llms via adaptive paral- lel decoding.arXiv preprint arXiv:2506.00413, 2025

    work page arXiv 2025

  64. [64]

    Accelerating diffusion large language models with slowfast: The three golden principles.arXiv preprint arXiv:2506.10848, 2025

    Qingyan Wei, Yaojie Zhang, Zhiyuan Liu, Dongrui Liu, and Linfeng Zhang. Accelerating diffusion large language models with slowfast: The three golden principles.arXiv preprint arXiv:2506.10848, 2025

    work page arXiv 2025

  65. [65]

    Scaling diffusion language models via adapta- tion from autoregressive models

    Shansan Gong, Shivam Agarwal, Yizhe Zhang, Ji- acheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling diffusion language models via adapta- tion from autoregressive models. InThe Thirteenth International Conference on Learning Representa- tions, 2025

    work page 2025

  66. [66]

    Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing

    Xu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, and Zhijie Deng. Diffusion llms can do faster- than-ar inference via discrete diffusion forcing.arXiv preprint arXiv:2508.09192, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  67. [67]

    d3llm: Ultra-fast diffusion llm us- ing pseudo-trajectory distillation.arXiv preprint arXiv:2601.07568, 2026

    Yu-Yang Qian, Junda Su, Lanxiang Hu, Peiyuan Zhang, Zhijie Deng, Peng Zhao, and Hao Zhang. d3llm: Ultra-fast diffusion llm us- ing pseudo-trajectory distillation.arXiv preprint arXiv:2601.07568, 2026

    work page arXiv 2026

  68. [68]

    Flextron: Many-in-One Flexible Large Language Model

    Ruisi Cai, Saurav Muralidharan, Greg Heinrich, Hongxu Yin, Zhangyang Wang, Jan Kautz, and Pavlo Molchanov. Flextron: Many-in-one flexible large lan- guage model.arXiv preprint arXiv:2406.10260, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  69. [69]

    Amoeballm: Constructing any-shape large language models for efficient and instant deployment.Advances in Neu- ral Information Processing Systems, 37:78299–78319, 2024

    Yonggan Fu, Zhongzhi Yu, Junwei Li, Jiayi Qian, Yongan Zhang, Xiangchi Yuan, Dachuan Shi, Ro- man Yakunin, and Yingyan Celine Lin. Amoeballm: Constructing any-shape large language models for efficient and instant deployment.Advances in Neu- ral Information Processing Systems, 37:78299–78319, 2024

    work page 2024

  70. [70]

    Nemotron-Post-Training-Dataset-v2, August 2025

    Dhruv Nathawani, Shuoyang Ding, Vitaly Lavrukhin, Igor Gitman, Somshubra Majumdar, Evelina Bakh- turina, Boris Ginsburg, and Jane Polak Scowcroft. Nemotron-Post-Training-Dataset-v2, August 2025. 19 Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding A. Diffusion Sampler Details This appendix...

    work page 2025