Pith. sign in

REVIEW 3 major objections 5 minor 66 references

Cached diffusion LLMs inherit the AR serving design space once a deficit token-budget scheduler replaces chunked prefill.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 20:57 UTC pith:MH7QDVNU

load-bearing objection Solid systems paper: deficit-budget scheduling for indivisible dLLM prefills plus a clean interference-vs-partitioning map; the 9–20% regime numbers are real under the chosen 5P3 point but not yet shown to be robust to re-partitioning. the 3 major comments →

arxiv 2607.04206 v1 pith:MH7QDVNU submitted 2026-07-05 cs.DC cs.LG

Sangam: Efficiently Serving Diffusion LLMs with the AR Stack

Nitin Kedia , Saurabh Agarwal , Myungjin Lee , Aditya Akella This is my paper
classification cs.DC cs.LG
keywords diffusion language modelsLLM servingKV cachecontinuous batchingdisaggregated servingprefill-decode interferencetoken budget schedulinghybrid serving
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Diffusion language models generate text by iteratively unmasking many positions at once under bidirectional attention. Approximate key-value caching turns each request into a cycle of full-context prefills and block-sized decodes, so the machinery of autoregressive serving becomes relevant—but not usable as-is. Prefills cannot be split, decodes saturate the GPU at small batch sizes, and every in-flight request repeatedly re-enters prefill. Sangam shows that a deficit token-budget scheduler recovers amortized stall-free colocated serving: seat ongoing decodes first, admit whole prefills only when leftover budget has accumulated, and carry unused budget forward. Pairing that scheduler with hybrid overflow of prefills onto protected decode workers makes the same two factors that structure AR serving—prefill-decode interference and prefill/decode partitioning—also structure dLLM serving. Colocated execution wins on decode-heavy work; hybrid wins on prefill-heavy work.

Core claim

Approximate KV caching induces a repeated prefill/decode structure in diffusion LLM inference whose serving design space is governed by the same two axes as autoregressive serving: prefill-decode interference and prefill/decode resource partitioning. Because bidirectional attention forbids chunked prefill, temporal deferral via a deficit token budget achieves amortized stall-free colocated batching, and hybrid overflow of prefills onto those budget-protected workers relieves static partitioning; colocated cuts mean latency 9–20% on decode-heavy LLaDA-8B ShareGPT while hybrid cuts mean latency 8–20% on prefill-heavy Dream-7B arXiv.

What carries the argument

Deficit token-budget scheduler: each iteration first seats all in-flight block-sized decodes, then admits whole indivisible prefills only when residual budget plus carried deficit can fit them, carrying unused budget forward so large prefills are admitted without per-iteration stall-free guarantees.

Load-bearing premise

The measured interference-versus-partitioning trade-off is representative when approximate blockwise caching, fixed sampling defaults, a few static worker splits, and a single eight-GPU node are used for two models and two filtered traces.

What would settle it

If, on the same hardware and traces, a pure prefill-prioritizing colocated scheduler or a better static disaggregated split matched or beat Sangam’s mean and p99 end-to-end latency across the reported QPS range on both ShareGPT and arXiv, the claim that deficit budgets and hybrid overflow are required to navigate the design space would fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Operators can choose colocated versus hybrid dLLM serving by whether the workload is decode-heavy or prefill-heavy, reusing the same rule of thumb already used for AR models.
  • Static disaggregated dLLM deployments can be upgraded by replacing decode workers with tight-budget colocated workers that absorb overflow prefills under the deficit scheduler.
  • Continuous batching at iteration granularity is mandatory for cached dLLMs because block boundaries arrive at different times for different requests.
  • When chunked prefill is unavailable, the iteration token budget becomes the primary knob trading prefill queueing delay against decode interference.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same deficit construction should transfer to any model class whose prefills are indivisible yet still alternates prefill and decode phases.
  • As commercial dLLMs grow, hybrid overflow may become the default for mixed multi-tenant traces where static partitions cannot track load.
  • Block-causal dLLM variants that re-prefill less often would still use the same scheduler, only with a higher effective budget because refresh events are rarer.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents Sangam, a serving system for cached diffusion LLMs (dLLMs) that reuse approximate KV caches (e.g., Fast-dLLM blockwise caching). It argues that approximate caching induces a repeated prefill/decode structure, so AR serving mechanisms are relevant but not directly applicable: dLLM decodes are block-sized, prefills recur at data-dependent boundaries, and bidirectional attention precludes chunked prefill. Sangam contributes (1) a deficit token-budget scheduler that admits in-flight decodes first, admits whole indivisible prefills only when residual plus carried deficit budget fits, and carries unused budget forward to achieve amortized stall-free colocated batching; and (2) a hybrid architecture that pairs a dedicated prefill pool with deficit-protected colocated decode-role workers and overflows prefills when the prefill pool is overloaded. On LLaDA-8B and Dream-7B with ShareGPT and arXiv traces on an 8-H100 node, the paper reports that colocated serving cuts mean latency 9–20% over hybrid on decode-heavy LLaDA-8B ShareGPT, while hybrid cuts mean latency 8–20% over colocated on prefill-heavy Dream-7B arXiv, and frames the design space as governed by prefill-decode interference and prefill/decode partitioning.

Significance. If the results hold, this is a timely systems contribution for an emerging model class that commercial vendors already advertise as 4–10× faster per request. The paper correctly identifies three concrete mismatches with the AR stack (big block decodes, recurring prefills, no chunked prefill) and supplies a practical scheduler that restores amortized stall-free colocated serving without spatial chunking, plus a simple hybrid overflow path that protects decode workers. The open-source release and the explicit two-axis framing (interference vs. partitioning) are useful for follow-on work. The evaluation is empirical and reproducible in principle; strengths include clear motivation measurements (Figs. 3–5), mean/p99 load curves, and queueing/decode breakdowns that make the regime story legible. The main limitation is that the quantitative regime ranking is demonstrated under a narrow set of operating points rather than a systematic robustness study.

major comments (3)
  1. §6.3 and Figs. 9–10: The central design-space claim (colocated wins on decode-heavy LLaDA-8B ShareGPT by 9–20% mean latency over hybrid; hybrid wins on prefill-heavy Dream-7B arXiv by 8–20% over colocated) is measured almost exclusively under one static split (5P3D/5P3C) and one hybrid operating point (τ=1024, θ=8k). §3.3 already shows that neighboring splits (4P4D, 6P2D) invert which side queues. Without a modest split and (τ,θ) robustness sweep showing that the ranking survives re-partitioning or budget retuning, the quantitative support for the two-axis claim remains tied to a hand-chosen strong configuration rather than established as a stable regime property.
  2. §6.1–§6.5: All results are single-node (8×H100, NVLink). Hybrid and disaggregated rely on KV transfer and overflow routing; the paper notes transfer is negligible intra-node (p99 192–384 ms) but does not evaluate multi-node bandwidth, layerwise streaming, or transfer-decode overlap. The hybrid overflow path and the claim that hybrid is a simple augmentation of static disaggregation therefore need at least a multi-node discussion or experiment before the architecture claim can be taken as general for cluster serving.
  3. §6.2 and §6.4: The Fast-dLLM baseline is batch-size-1 (even the in-system version), so the large throughput gap mainly isolates continuous batching, not the deficit scheduler. The paper acknowledges this, but the scheduler’s contribution is then only relative to in-system colocated/disaggregated/hybrid variants. A stronger load-bearing comparison would include a continuous-batching baseline without deficit carry (e.g., pure decode-first with no carry, or a fixed non-deficit token budget) so that amortized stall-free behavior is isolated from batching itself.
minor comments (5)
  1. Abstract and §1: The 9–20% / 8–20% ranges should state the QPS points or load region they summarize; as written they read as global constants.
  2. §4.1 Algorithm 1: Clarify the idle-rule interaction with deficit carry when a single prefill exceeds τ (liveness is stated, but the amortized bound language is informal).
  3. §6.1: Filtering to ≤4096 tokens and appending fixed MASK lengths (1024/512) is reasonable but should note how sensitive results are to those caps, since sequence length affects every prefill/re-prefill.
  4. Figure 6 and Algorithm 1: Notation for deficit S_t vs. budget R_t is slightly inconsistent between the figure caption and the algorithm; unify.
  5. Related work §7: A short explicit comparison table (batching, scheduling policy, disaggregation support) versus dInfer and dLLM-Serve would help readers place Sangam.

Circularity Check

0 steps flagged

No circularity: Sangam's claims are empirical latency measurements and an algorithmic scheduling policy, not a derivation that reduces to fitted inputs or self-referential definitions.

full rationale

This is a systems paper. Its strongest claims are (1) that a deficit token-budget scheduler achieves amortized stall-free colocated serving for indivisible dLLM prefills, and (2) that colocated vs hybrid effectiveness is governed by prefill-decode interference and prefill/decode partitioning, with measured 9–20% / 8–20% mean-latency wins on specific (model, trace) pairs. Neither claim is a mathematical derivation that reduces to its inputs by construction. The deficit construction (Algorithm 1, §4.1) is an explicit admission policy with a stated amortized bound over busy periods; it is not fitted to the latency numbers later reported. The comparative ranking in §6.3 is measured end-to-end latency under stated workloads, splits (5P3D/5P3C), and budgets (τ=1024, θ=8k), not a prediction forced by a fitted constant or a uniqueness theorem. Self-citations to prior AR serving work (Orca, Sarathi-Serve, Splitwise, DistServe, etc.) are ordinary background for continuous batching, chunked prefill, and disaggregation; they are not load-bearing uniqueness results that force Sangam's design. No step renames a known empirical pattern as a first-principles prediction. The reader's and skeptic's concerns about generalization of the chosen split/budget are correctness/robustness issues, not circularity. Score 0 is the honest finding.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central claims rest on standard systems assumptions plus a small set of operator-chosen knobs and the domain premise that approximate blockwise caching induces a usable cyclic prefill/decode structure. No new physical entities are postulated; free parameters are the usual serving hyperparameters.

free parameters (4)
  • iteration token budget τ = 1024 (main); also 512, 2048, 4096, 16384
    Hand-chosen per-iteration token budget (swept 512–16384; default 1024 in main comparisons) that controls the interference–throughput trade-off.
  • overflow threshold θ = 8192
    Token threshold at which the hybrid scheduler declares the prefill pool overloaded and overflows work.
  • prefill:decode / prefill:colocated worker split = 5P3D / 5P3C primary
    Static pool sizes (e.g., 5P3D, 5P3C, 3P5C) chosen as strong configurations for the 8-GPU node.
  • block size and confidence threshold = block size 32, threshold 0.9
    Fast-dLLM defaults that determine how often re-prefills occur and how many tokens each decode contributes.
axioms (4)
  • domain assumption Approximate KV caching (Fast-dLLM / dKV-Cache style) keeps activations stable enough across short windows that a cyclic prefill/decode execution is valid.
    Stated in §2.2 and used throughout; quality of approximation is taken from prior work rather than re-proved.
  • domain assumption Bidirectional attention makes prefills indivisible; chunked prefill is structurally unavailable.
    Core motivation in §3.2; underpins why deficit (temporal) rather than spatial chunking is required.
  • domain assumption Continuous batching at iteration granularity and paged KV management from AR serving remain applicable once the above mismatches are handled.
    Background in §2.3 and architecture §4.3.
  • standard math Deficit Round-Robin style carry-over yields an amortized per-iteration prefill bound of τ over busy periods.
    Algorithm 1 and the amortized bound stated in §4.1; standard scheduling accounting.
invented entities (2)
  • deficit token-budget scheduler independent evidence
    purpose: Admit whole indivisible prefills only when accumulated budget allows, carrying unused budget forward to achieve amortized stall-free colocated serving.
    Primary algorithmic contribution of the paper; independent_evidence is the open-source implementation and latency experiments, not an external physical prediction.
  • hybrid prefill-overflow configuration (dedicated prefill pool + deficit-protected colocated decode-role workers) independent evidence
    purpose: Relieve static prefill under-provisioning while bounding interference on decode workers.
    Architectural contribution in §4.2; evaluated against pure colocated and static disaggregated baselines.

pith-pipeline@v1.1.0-grok45 · 29069 in / 3152 out tokens · 31803 ms · 2026-07-11T20:57:00.594269+00:00 · methodology

0 comments
read the original abstract

Diffusion language models (dLLMs) generate text by iteratively denoising a masked response and can commit multiple output positions per model invocation. Their bidirectional attention prevents exact autoregressive-style KV caching, since committing one position shifts the KV activations of all others. Approximate caching techniques such as Fast-dLLM and dKV-Cache refresh KV activations repeatedly and reuse them across intervening decodes, inducing a repeated prefill/decode structure. This makes AR serving mechanisms relevant to dLLMs, but not directly applicable. dLLM decodes are block-sized rather than token-sized, prefills recur, and bidirectional attention precludes the chunked prefill mechanism used for stall-free colocated serving. We present Sangam, a serving system for cached dLLM inference. Sangam introduces a deficit token-budget scheduler that admits in-flight decodes first, admits whole indivisible prefills only when the accumulated token budget allows, and carries unused budget forward. This achieves amortized stall-free scheduling. Disaggregated serving avoids prefill-decode interference but suffers from prefill/decode resource partitioning problem. Sangam adopts a hybrid serving strategy, overflowing prefills onto decode workers to relieve prefill under-provisioning, and uses the same deficit-budget scheduler to protect those workers' decodes from the overflow. We show that like AR serving, dLLM serving design space is governed by prefill-decode interference and prefill/decode partitioning. Colocated serving is most effective on decode-heavy workloads, cutting mean latency by 9-20% over hybrid execution on LLaDA-8B ShareGPT; while hybrid execution is most effective on prefill-heavy workloads, cutting mean latency by 8-20% over colocated execution on Dream-7B arXiv. Sangam is available at https://github.com/UT-InfraAI/sangam.

Figures

Figures reproduced from arXiv: 2607.04206 by Aditya Akella, Myungjin Lee, Nitin Kedia, Saurabh Agarwal.

Figure 1
Figure 1. Figure 1: Diffusion LLM inference for the prompt "Where did the cat sit?" and response "On the warm mat". A dLLM appends a fixed-length block of mask tokens (shown as [M]) to the prompt and unmasks a subset of positions in parallel and out of order each iteration. Generation stops once eos is committed (iter 3) and every position before it is unmasked (iter 4); positions after eos are ignored. for dLLMs without rely… view at source ↗
Figure 2
Figure 2. Figure 2: Execution state machines for three LLM serving regimes. The autoregressive baseline (a) prefills once and then decodes one token at a time. A naive diffusion LLM (b) re-runs prefill at every denoising step. Adding a KV cache to the diffusion model (c) reintroduces a decode state but still requires an occasional return to prefill when the cache must be refreshed. operations. Mixing them forces the shared it… view at source ↗
Figure 3
Figure 3. Figure 3: plots decode iteration time against batch size for two dLLMs (LLaDA-8B, Dream-7B) and their AR counterparts1 (Llama-3-8B [14], Qwen2.5-7B [37], measured on SGLang [58]). For both dLLMs, iteration time grows several-fold across the measured range, while for the AR models it grows only mod￾estly. Decode batching is therefore significantly less impact￾ful for high-throughput low-latency dLLM serving compared … view at source ↗
Figure 4
Figure 4. Figure 4: Iterations (forward passes) versus output length, one point per request. Because the number of positions un￾masked per iteration is data-dependent, requests that emit the same number of tokens still take widely different num￾bers of iterations to finish, visible as the vertical spread at any fixed output length. this, dLLM inference engines cannot singularly rely on creating large batches to drive throughp… view at source ↗
Figure 6
Figure 6. Figure 6: Deficit token-budget scheduling over three itera￾tions. Each iteration first fills the per-iteration budget 𝜏 with in-flight decodes 𝐷𝑡 , then admits waiting prefills greedily in request arrival order while they fit in the remaining budget 𝑅𝑡 = 𝜏 −𝐷𝑡 +𝑆𝑡−1. Any unspent budget 𝑆𝑡 is carried forward, raising the effective ceiling in the next iteration so that a large prefill (e.g. 𝑝3) can be admitted without… view at source ↗
Figure 7
Figure 7. Figure 7: Sangam architectures. (a) Colocated: identical workers each run prefill and decode locally under the deficit￾budget scheduler (Algorithm 1). (b) Hybrid: dedicated prefill workers transfer KV to (primarily) decode-role colocated workers, with prefill overflow to colocated workers when all prefill workers exceed a load threshold 𝜃. The overflow trigger is token-based. The hybrid scheduler tracks for each wor… view at source ↗
Figure 8
Figure 8. Figure 8: Mean and p99 end-to-end latency under varying QPS for in-system Fast-dLLM and Sangam (Colocated with 𝜏 = 1024) running LLaDA-8B on a single H100 80GB GPU. and confidence-threshold sampling, but run it inside Sangam so it benefits from the optimizations such as CUDA Graph which reduce batch execution time by up to 2×. On LLaDA-8B with the ShareGPT trace, Fast-dLLM’s mean latency stays flat up to QPS 0.3 and… view at source ↗
Figure 9
Figure 9. Figure 9: End-to-end latency vs. QPS for colocated, static-split disaggregated (5P3D), and hybrid (5P3C). Top row mean, bottom row p99; columns are (model, trace) pairs. Colocated and hybrid degrade gracefully while the static disaggregated typically saturates first. Colocated-1024 Disaggregated-5P3D Hybrid-5P3C 0 5 10 15 Prefill Queueing Delay (s) 0.00 0.25 0.50 0.75 1.00 CDF QPS=6.5 (a) LLaDA-8B ShareGPT. 0 10 20 … view at source ↗
Figure 10
Figure 10. Figure 10: Per-request CDFs of prefill queueing delay (left) and request decode time (right), at moderate request load (LLaDA-8B ShareGPT 6.5 QPS, Dream-7B arXiv 8.5 QPS). p99 values are highlighted. is even more prefill-bound and 6P2D runs the decode pool into its KV Cache capacity limit. Hybrid 5P3C (§ 4.2) reuses the same split as disaggregated, but its three decode-role workers are colocated workers. We run hybr… view at source ↗
Figure 11
Figure 11. Figure 11: Deficit token-budget sweep 𝜏 ∈ {512, 1024, 2048, 16384} for colocated serving. The top row reports a p99 end-to-end latency load curve (vs. QPS) for each (model, trace) pair across all four budgets. The bottom row decomposes a single mean and tail (p99) request, taken at one fixed QPS in that pair’s stable regime, into prefill queueing delay, prefill time, decode time, and other. Columns are (model, trace… view at source ↗
Figure 12
Figure 12. Figure 12: Sensitivity of hybrid scheduling to the deficit token budget 𝜏 on LLaDA-8B with the ShareGPT trace, at the 5P3C (top row) and 3P5C (bottom row) ratios; each row reports mean (left) and p99 (right) end-to-end latency vs. QPS, with a low (𝜏=1024) and a high (𝜏=4096) budget. The reduction in p99 latencies by using low budgets is not free. Processing overflow prefills slowly via a low budget means the colocat… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 16 linked inside Pith

  1. [1]

    Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S Gulavani, Ramachandran Ramjee, and Alexey Tu- manov. 2024. Vidur: A large-scale simulation framework for llm inference.Proceedings of Machine Learning and Systems6 (2024), 351– 366

    2024

  2. [2]

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ram- jee. 2024. Taming {Throughput-Latency} tradeoff in {LLM} inference with {Sarathi-Serve}. In18th USENIX symposium on operating systems design and implementation (OSDI 24). 117–134

    2024

  3. [3]

    Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. Gqa: Training general- ized multi-query transformer models from multi-head checkpoints. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 4895–4901

    2023

  4. [4]

    anon8231489123. 2023. ShareGPT Vicuna Unfiltered. https://huggingface.co/datasets/anon8231489123/ShareGPT_ Vicuna_unfiltered. Apache-2.0 license. Approximately 53k cleaned English ShareGPT conversations derived from 100k, split into 2048-token chunks for fine-tuning Vicuna-style models

    2023

  5. [5]

    Marianne Arriola, Aaron Gokaslan, Justin Chiu, Zhihan Yang, Zhix- uan Qi, Jiaqi Han, Subham Sahoo, and Volodymyr Kuleshov. 2025. Block diffusion: Interpolating between autoregressive and diffusion language models. InInternational Conference on Learning Representa- tions, Vol. 2025. 50726–50753

    2025

  6. [6]

    Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. 2021. Structured denoising diffusion models in discrete state-spaces.Advances in neural information processing systems34 (2021), 17981–17993

    2021

  7. [7]

    Parikshit Bansal and Sujay Sanghavi. 2025. Enabling approximate joint sampling in diffusion lms.arXiv preprint arXiv:2509.22738(2025)

    arXiv 2025

  8. [8]

    Tiwei Bie, Maosong Cao, Kun Chen, Lun Du, Mingliang Gong, Zhuochen Gong, Yanmei Gu, Jiaqi Hu, Zenan Huang, Zhenzhong Lan, et al. 2025. Llada2. 0: Scaling up diffusion language models to 100b.arXiv preprint arXiv:2512.15745(2025)

    Pith/arXiv arXiv 2025

  9. [9]

    Shuang Cheng, Yihan Bian, Dawei Liu, Yuhua Jiang, Yihao Liu, Lin- feng Zhang, Qian Yao, Zhongbo Tian, Wenhai Wang, Qipeng Guo, et al. 2026. Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation. InFindings of the Association for Computational Linguistics: ACL 2026. 22058–22075

    2026

  10. [10]

    Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse- aware attention model for abstractive summarization of long docu- ments. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 2 (S...

    2018

  11. [11]

    Jiakun Fan, Yanglin Zhang, Xiangchen Li, and Dimitrios S Nikolopou- los. 2025. Taming the Memory Footprint Crisis: System Design for Production Diffusion LLM Serving.arXiv preprint arXiv:2512.17077 (2025)

    arXiv 2025

  12. [12]

    2026.Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unify- ing Autoregressive, Diffusion, and Self-Speculation Decoding

    Yonggan Fu, Lexington Whalen, Abhinav Garg, Chengyue Wu, Mak- sim Khadkevich, Nicolai Oswald, Enze Xie, Daniel Egert, Sharath Tu- ruvekere Sreenivas, Shizhe Diao, Chenhan Yu, Ye Yu, Weijia Chen, Sajad Norouzi, Jingyu Liu, Shiyi Lan, Ligeng Zhu, Jin Wang, Jindong Jiang, Morteza Mardani, Mehran Maghoumi, Song Han, Ante Jukić, Nima Tajbakhsh, Jan Kautz, and ...

    2026

  13. [13]

    Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer

  14. [14]

    InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP)

    Mask-predict: Parallel decoding of conditional masked language models. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 6112–6121

    2019

  15. [15]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)

    Pith/arXiv arXiv 2024

  16. [16]

    Ishaan Gulrajani and Tatsunori B Hashimoto. 2023. Likelihood-based diffusion language models.Advances in Neural Information Processing Systems36 (2023), 16693–16715

    2023

  17. [17]

    Michael Hersche, Samuel Moor-Smith, Thomas Hofmann, and Abbas Rahimi. 2026. Soft-Masked Diffusion Language Models. InThe Four- teenth International Conference on Learning Representations.https: //openreview.net/forum?id=Gba02UMvrG

    2026

  18. [18]

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, et al

  19. [19]

    Inference without interference: Disaggregate llm inference for mixed downstream workloads.arXiv preprint arXiv:2401.11181(2024)

    Pith/arXiv arXiv 2024

  20. [20]

    Daniel Israel, Guy Van den Broeck, and Aditya Grover. 2025. Accelerating Diffusion LLMs via Adaptive Parallel Decoding. InAdvances in Neural Information Processing Systems, D. Bel- grave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38. Curran Associates, Inc., 52870– 52888.https://proceedings.neurips.cc/paper_files/pape...

    2025

  21. [21]

    Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, Stefano Ermon, et al. 2025. Mercury: Ultra-fast language models based on diffusion.arXiv e-prints(2025), arXiv–2506

    2025

  22. [22]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  23. [23]

    InProceedings of the 29th symposium on operating systems principles

    Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th symposium on operating systems principles. 611–626

  24. [24]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast in- ference from transformers via speculative decoding. InInternational Conference on Machine Learning. PMLR, 19274–19286

    2023

  25. [25]

    Tianyi Li, Mingda Chen, Bowei Guo, and Zhiqiang Shen. 2025. A survey on diffusion language models.arXiv preprint arXiv:2508.10875 (2025)

    Pith/arXiv arXiv 2025

  26. [26]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077(2024)

    Pith/arXiv arXiv 2024

  27. [27]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al . 2024. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437(2024)

    Pith/arXiv arXiv 2024

  28. [28]

    Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyan Wei, Shaobo Wang, Yichen Zhu, and Linfeng Zhang. 2026. dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching. arXiv:2506.06295 [cs.LG]https://arxiv.org/abs/ 2506.06295

    Pith/arXiv arXiv 2026

  29. [29]

    Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. 2025. dkv-cache: The cache for diffusion language models.Advances in Neural Information Processing Systems38 (2025), 149009–149033

    2025

  30. [30]

    Yuxin Ma, Lun Du, Lanning Wei, Kun Chen, Qian Xu, Kangyu Wang, Guofeng Feng, Guoshan Lu, Lin Liu, Xiaojing Qi, et al. 2025. dinfer: An efficient inference framework for diffusion language models.arXiv preprint arXiv:2510.08666(2025)

    arXiv 2025

  31. [31]

    Tiyasa Mitra, Ritika Borkar, Nidhi Bhatia, Ramon Matas, Shivam Raj, Dheevatsa Mudigere, Ritchie Zhao, Maximilian Golub, Arpan Dutta, Sailaja Madduri, et al. 2025. Beyond the buzz: A pragmatic take on inference disaggregation.arXiv preprint arXiv:2506.05508(2025). 14 Sangam : Efficiently Serving Diffusion LLMs with the AR Stack

    Pith/arXiv arXiv 2025

  32. [32]

    Shen Nie, Qiyang Min, Shaoxuan Xu, Zihao Huang, Yuxuan Song, Yong Shan, Yankai Lin, Wayne Xin Zhao, Chongxuan Li, and Ji-Rong Wen. 2026. Improved Large Language Diffusion Models.arXiv preprint arXiv:2606.25331(2026)

    Pith/arXiv arXiv 2026

  33. [33]

    Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. 2025. Scaling up masked diffusion models on text. InInternational Conference on Learning Representations, Vol. 2025. 82974–82997

    2025

  34. [34]

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. 2026. Large language diffusion models.Advances in Neural Information Processing Systems38 (2026), 50608–50646

    2026

  35. [35]

    2026.Dynamo: A Datacenter Scale Distributed Inference Serv- ing Framework

    NVIDIA. 2026.Dynamo: A Datacenter Scale Distributed Inference Serv- ing Framework. Version 1.1.1.https://github.com/ai-dynamo/dynamo

    2026

  36. [36]

    Brendan O’Donoghue and Sebastian Flennerhag. 2026. Dif- fusionGemma: 4x Faster Text Generation. Google Blog. https://blog.google/innovation-and-ai/technology/developers- tools/diffusion-gemma-faster-text-generation/

    2026

  37. [37]

    Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhen- guo Li, and Chongxuan Li. 2025. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. InInterna- tional Conference on Learning Representations, Vol. 2025. 64972–65009

    2025

  38. [38]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative llm inference using phase splitting. In2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 118–132

    2024

  39. [39]

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading more storage for less computation—a {KVCache-centric} architecture for serving {LLM} chatbot. In23rd USENIX conference on file and storage technologies (FAST 25). 155–170

    2025

  40. [40]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

    Pith/arXiv arXiv 2025

  41. [41]

    Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. 2026. An- chored diffusion language model.Advances in Neural Information Processing Systems38 (2026), 89607–89661

    2026

  42. [42]

    Subham S Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov

  43. [43]

    Simple and effective masked diffusion language models.Ad- vances in Neural Information Processing Systems37 (2024), 130136– 130184

    2024

  44. [44]

    Xing, John Thickstun, and Arash Vahdat

    Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, Johnna Liu, Deep- ansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric P. Xing, John Thickstun, and Arash Vahdat. 2026. Esoteric Language Models. https://openreview.net/forum?id=XepOJx5ng4

    2026

  45. [45]

    Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. 2024. Simplified and generalized masked diffusion for discrete data.Advances in neural information processing systems37 (2024), 103131–103167

    2024

  46. [46]

    Madhavapeddi Shreedhar and George Varghese. 1995. Efficient fair queueing using deficit round robin. InProceedings of the conference on Applications, technologies, architectures, and protocols for computer communication. 231–242

    1995

  47. [47]

    Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, et al

  48. [48]

    Seed diffusion: A large-scale diffusion language model with high-speed inference.arXiv preprint arXiv:2508.02193(2025)

    Pith/arXiv arXiv 2025

  49. [49]

    Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. 2025. Dynamollm: Designing llm inference clusters for per- formance and energy efficiency. In2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 1348–1362

    2025

  50. [50]

    Foteini Strati, Sara Mcallister, Amar Phanishayee, Jakub Tarnawski, and Ana Klimovic. 2024. D \’ej\avu: Kv-cache streaming for fast, fault-tolerant generative llm serving.arXiv preprint arXiv:2403.01876 (2024)

    Pith/arXiv arXiv 2024

  51. [51]

    Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: Dynamic scheduling for large language model serving. In18th USENIX symposium on operating systems design and implementation (OSDI 24). 173–191

    2024

  52. [52]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information processing systems30 (2017)

    2017

  53. [53]

    Dimitri von Rütte, Janis Fluri, Omead Pooladzandi, Bernhard Schölkopf, Thomas Hofmann, and Antonio Orvieto. 2025. Scal- ing behavior of discrete diffusion language models.arXiv preprint arXiv:2512.10858(2025)

    arXiv 2025

  54. [54]

    Qingyan Wei, Yaojie Zhang, Zhiyuan Liu, Puyu Zeng, Yuxuan Wang, Biqing Qi, Dongrui Liu, and Linfeng Zhang. 2026. Accelerating Dif- fusion Large Language Models with SlowFast Sampling: The Three Golden Principles. arXiv:2506.10848 [cs.CL]https://arxiv.org/abs/ 2506.10848

    arXiv 2026

  55. [55]

    Chengyue Wu, Hao Zhang, Shuchen Xue, Shizhe Diao, Yonggan Fu, Zhijian Liu, Pavlo Molchanov, Ping Luo, Song Han, and Enze Xie

  56. [56]

    Fast-dllm v2: Efficient block-diffusion llm.arXiv preprint arXiv:2509.26328(2025)

    arXiv 2025

  57. [57]

    Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. 2025. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding.arXiv preprint arXiv:2505.22618(2025)

    Pith/arXiv arXiv 2025

  58. [58]

    Zhihui Xie, Jiacheng Ye, Lin Zheng, Jiahui Gao, Jingwei Dong, Zirui Wu, Xueliang Zhao, Shansan Gong, Xin Jiang, Zhenguo Li, et al. 2025. Dream-coder 7b: An open diffusion language model for code.arXiv preprint arXiv:2509.01142(2025)

    Pith/arXiv arXiv 2025

  59. [59]

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. 2025. Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487(2025)

    Pith/arXiv arXiv 2025

  60. [60]

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al. 2025. Flashinfer: Efficient and customizable attention engine for llm inference serving.Proceedings of Machine Learning and Systems7 (2025)

    2025

  61. [61]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for {Transformer-Based} generative models. In16th USENIX symposium on operating systems design and implementation (OSDI 22). 521–538

    2022

  62. [62]

    Yifan Yu, Yuqing Jian, Junxiong Wang, Zhongzhu Zhou, Donglin Zhuang, Xinyu Fang, Sri Yanamandra, Xiaoxia Wu, Qingyang Wu, Shuaiwen Leon Song, et al . 2026. Introspective diffusion language models.arXiv preprint arXiv:2604.11035(2026)

    Pith/arXiv arXiv 2026

  63. [63]

    Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover

  64. [64]

    d1: Scaling reasoning in diffusion large language models via reinforcement learning.Advances in Neural Information Processing Systems38 (2026), 56729–56762

    2026

  65. [65]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody H Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured lan- guage model programs.Advances in neural information processing systems37 (2024), 62557–62583. 15 Kedia et al

    2024

  66. [66]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 193–210. 16

    2024