Pith. sign in

REVIEW 2 major objections 2 minor 74 cited by

Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models

T0 review · 2 major / 2 minor · reviewed 2026-05-15 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Block diffusion language models interpolate between autoregressive and diffusion approaches to support arbitrary-length generation.

desk verdict Block diffusion gives a workable hybrid of AR and diffusion LMs via blocks, KV caching, and a training recipe, but the arbitrary-length claim rests on unverified scaling. read the letter →

arxiv 2503.09573 v3 pith:BTT26H53 submitted 2025-03-12 cs.LG cs.AI

Marianne Arriola , Aaron Gokaslan , Justin T. Chiu , Zhihan Yang , Zhixuan Qi , Jiaqi Han , Subham Sekhar Sahoo , Volodymyr Kuleshov This is my paper · ORCID
classification cs.LGcs.AI
keywords blockdiffusionlanguagemodelsautoregressivetextgenerationparallelsamplingKVcachingflexiblelength
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces block diffusion language models that combine elements of discrete denoising diffusion with autoregressive processing by operating on blocks of tokens. This structure allows parallel sampling inside each block while advancing autoregressively across blocks, removing the fixed-length restriction of prior diffusion models and enabling KV caching for faster inference. A supporting training recipe uses gradient variance estimators and data-driven noise schedules to stabilize optimization. If these models scale, they could close the performance gap between diffusion and autoregressive language models while retaining controllability and parallel generation advantages.

What carries the argument

The block diffusion process, which applies diffusion within fixed-size token blocks while chaining blocks autoregressively.

What would settle it

An evaluation on a standard language modeling benchmark where block diffusion models fail to exceed prior diffusion baselines in likelihood or produce incoherent text when generating sequences longer than the training block size.

Watch

Extended reading notes

Core claim

Block diffusion processes sequences in blocks, applying diffusion denoising within each block and autoregressive prediction across blocks. This interpolation supports flexible-length generation, improves inference speed via KV caching and parallel token sampling, and reaches new state-of-the-art results among diffusion language models on standard benchmarks.

Load-bearing premise

The training algorithm, variance estimators, and data-driven noise schedules will produce stable models that generalize to new data without hidden instabilities or overfitting.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces block diffusion language models that interpolate between autoregressive and discrete denoising diffusion approaches for language modeling. It proposes an efficient training algorithm, estimators for gradient variance, and data-driven noise schedules, claiming new state-of-the-art results among diffusion models on language benchmarks together with support for flexible-length and arbitrary-length generation via KV caching and parallel token sampling.

Significance. If the central claims hold, the work would meaningfully narrow the performance gap between diffusion and autoregressive language models while adding controllability and parallel-generation advantages; the open release of code, weights, and a blog post further increases the potential impact.

major comments (2)
  1. [Experiments] Experiments section: the headline claim that block diffusion enables generation of arbitrary-length sequences rests on the unverified assumption that block-wise KV caching and the data-driven noise schedule remain stable and variance-minimizing for lengths substantially exceeding the training block size (e.g., 4–8× longer contexts). No scaling curves, perplexity-vs-length measurements, or out-of-distribution length ablations are reported, which is load-bearing for the central flexibility claim.
  2. [§3 and Experiments] §3 (training recipe) and Experiments: the paper asserts that the combination of the efficient training algorithm, gradient-variance estimators, and data-driven noise schedules produces stable models, yet provides neither quantitative ablations isolating each component’s contribution nor error bars on the reported benchmark numbers. Without these, the SOTA claim among diffusion models cannot be fully assessed.
minor comments (2)
  1. Figure captions and axis labels should be expanded to make the reported metrics (e.g., perplexity, generation speed) immediately interpretable without reference to the main text.
  2. [Abstract] The abstract states that code and weights are provided; the manuscript should include a permanent DOI or archive link in addition to the project-page URL.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback. We address each major comment below and will revise the manuscript accordingly to strengthen the empirical support for our claims.

read point-by-point responses

  1. Referee: Experiments section: the headline claim that block diffusion enables generation of arbitrary-length sequences rests on the unverified assumption that block-wise KV caching and the data-driven noise schedule remain stable and variance-minimizing for lengths substantially exceeding the training block size (e.g., 4–8× longer contexts). No scaling curves, perplexity-vs-length measurements, or out-of-distribution length ablations are reported, which is load-bearing for the central flexibility claim.

    Authors: We acknowledge that the current experiments focus on lengths comparable to the training block size and do not include explicit scaling curves or out-of-distribution ablations for substantially longer contexts. The design of block diffusion, which interpolates between autoregressive and diffusion models, provides a basis for expecting that KV caching and the data-driven noise schedule will generalize, but we agree this requires direct empirical verification. In the revision we will add perplexity-versus-length curves and length-ablation experiments testing up to 4× the training block size. revision: yes

  2. Referee: §3 (training recipe) and Experiments: the paper asserts that the combination of the efficient training algorithm, gradient-variance estimators, and data-driven noise schedules produces stable models, yet provides neither quantitative ablations isolating each component’s contribution nor error bars on the reported benchmark numbers. Without these, the SOTA claim among diffusion models cannot be fully assessed.

    Authors: We agree that quantitative ablations isolating the contribution of each element of the training recipe and error bars on benchmark results would make the stability and SOTA claims more robust. In the revised manuscript we will add ablations that separately measure the effect of the efficient training algorithm, the gradient-variance estimators, and the data-driven noise schedules. We will also report standard deviations computed over multiple independent runs for all main benchmark numbers. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; block diffusion introduces independent block structure, training algorithm, and noise schedules on top of existing diffusion and autoregressive frameworks.

full rationale

The derivation chain in the abstract and described method is self-contained. Block diffusion is presented as an interpolation via explicit new elements (block-wise structure, KV caching for flexible lengths, efficient training algorithm, gradient-variance estimators, and data-driven noise schedules) that are not shown to reduce by construction to prior fitted quantities or self-citations. Performance claims and arbitrary-length generation are positioned as empirical outcomes of these additions rather than tautological renamings or load-bearing self-references. No equations or steps equate outputs to inputs via definition or fitting alone.

Assumptions & free parameters 1 free parameters · 1 assumptions · 1 invented entities

The central claim rests on the effectiveness of the block interpolation and the custom training recipe; limited details available from abstract only.

free parameters (1)
  • data-driven noise schedule parameters
    Chosen to minimize gradient variance; likely fitted or optimized on data as described.
assumptions (1)
  • domain assumption Diffusion and autoregressive models can be effectively interpolated via a block structure for language sequences.
    Core premise enabling the hybrid approach.
invented entities (1)
  • block diffusion language model
    purpose: To combine parallel generation and controllability of diffusion with flexible length and efficiency of autoregressive models.
    New class introduced by the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models." pith.science (2026). https://pith.science/paper/BTT26H53

@misc{pith2026250309573,
  author       = {Pith},
  title        = {Pith review of: Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BTT26H53}},
  note         = {Machine review of arXiv:2503.09573}
}
read the original abstract

Diffusion language models offer unique benefits over autoregressive models due to their potential for parallelized generation and controllability, yet they lag in likelihood modeling and are limited to fixed-length generation. In this work, we introduce a class of block diffusion language models that interpolate between discrete denoising diffusion and autoregressive models. Block diffusion overcomes key limitations of both approaches by supporting flexible-length generation and improving inference efficiency with KV caching and parallel token sampling. We propose a recipe for building effective block diffusion models that includes an efficient training algorithm, estimators of gradient variance, and data-driven noise schedules to minimize the variance. Block diffusion sets a new state-of-the-art performance among diffusion models on language modeling benchmarks and enables generation of arbitrary-length sequences. We provide the code, along with the model weights and blog post on the project page: https://m-arriola.com/bd3lms

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 74 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 74 Pith citations

  1. Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages

    cs.LG 2026-03 unverdicted novelty 8.0 of 10

    Derives an exact unbiased policy gradient for RL post-training of diffusion LLMs via entropy-guided step selection and one-step denoising rewards, achieving state-of-the-art results on coding and logical reasoning benchmarks.

  2. NPU Design for Diffusion Language Model Inference

    cs.AR 2026-01 unverdicted novelty 8.0 of 10

    Introduces the first NPU accelerator for diffusion language models with dLLM-specific ISA, hardware execution model, BAOS KV quantization, and 7nm RTL synthesis.

  3. Scaling Inherently Interpretable Language Models

    cs.CL 2026-08 conditional novelty 7.0 of 10

    Training a language model with a built-in concept bottleneck preserves compute-optimal scaling and yields interpretability metrics that improve with scale, demonstrated on an 8B causal diffusion model.

  4. Context-weighted Discrete Flow Matching

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Reweighting discrete-flow-matching updates by local context—via a context-weighted sampler or a scaled cross-entropy loss—improves text and molecular generation, cutting generative perplexity on OpenWebText by up to 63%.

  5. FlowBlock: Wavefront-Parallel Decoding for Self-Correcting Diffusion Language Models

    cs.AI 2026-07 conditional novelty 7.0 of 10

    FlowBlock overlaps adjacent blocks of self-correcting diffusion LLMs, achieving up to 2.95x and 4.01x higher tokens/sec over serial baselines with up to 77.1% lower latency and matched or better accuracy.

  6. Trees from Marginals: Autoregressive drafting with factorized priors

    cs.LG 2026-07 accept novelty 7.0 of 10

    Weaver restores conditional dependencies on top-K factorized marginals to build high-acceptance draft trees, plus a fused GDN tree-verify kernel, yielding 4.37× AR speedup and 24.7% over DFlash.

  7. Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    TwoTower decouples autoregressive context from diffusion denoising using a frozen AR tower and bidirectional diffusion tower, retaining 98.7% baseline quality with 2.42X throughput on a 30B model trained on 2.1T tokens.

  8. Learning from the Self-future: On-policy Self-distillation for dLLMs

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    d-OPSD reframes on-policy self-distillation for dLLMs via suffix conditioning from self-generated answers and step-level supervision, outperforming RLVR and SFT on reasoning benchmarks with ~10% of the optimization steps.

  9. Unified Energy for Invariant and Independent Decoding in Diffusion Language Models

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    The paper introduces Uni-E, a unified energy for DLMs that accounts for model capacity, dependency and invariance, can be computed exactly, and corrects distribution shifts from dependency and invariance.

  10. Continuous Language Diffusion as a Decoder-Interface Problem

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    Continuous language diffusion works by entering high-margin decoder basins where frozen T5 embeddings recover 93-96% of native decisions and linear readouts reach 97.9% agreement, implying models should be evaluated a...

  11. Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics

    cs.CL 2026-06 conditional novelty 7.0 of 10

    Zero-parameter naive samplers achieve state-of-the-art generative perplexity while producing incoherent text, proving the metric is unsound; distributional divergences like MAUVE and energy distance correctly rank the...

  12. MaskForge: Structure-Aware Adaptive Attacks for Jailbreaking Diffusion Large Language Models

    cs.CR 2026-06 unverdicted novelty 7.0 of 10

    MaskForge reaches 79.3% average attack success rate on five dLLMs by adaptively searching and accumulating structural attack patterns with a UCB bandit, improving 17.6% over baselines and transferring to 88.2% on AdvBench.

  13. Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    BASTION is a budget-aware speculative decoding framework with adaptive tree-structured block diffusion drafting that reports up to 6.61x speedup and 39% improvement over block-diffusion baselines.

  14. Learned Relay Representations for Forward-Thinking Discrete Diffusion Models

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Learned Relay Representations enable masked diffusion models to propagate useful latent information across denoising steps, scaling to Fast-dLLM v2 to outperform supervised finetuning on coding tasks while cutting inf...

  15. Dynamic Chunking for Diffusion Language Models

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    DCDM replaces positional blocks with learnable semantic chunks via differentiable Chunking Attention, yielding consistent gains over block and unstructured diffusion baselines up to 1.5B parameters.

  16. Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL

    cs.AI 2026-05 conditional novelty 7.0 of 10

    Masked diffusion language models, not larger autoregressive LLMs, are the better building block for text-based world models in agentic RL, improving rollout fidelity, diversity, and downstream task success.

  17. DMax: Aggressive Parallel Decoding for dLLMs

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    DMax uses On-Policy Uniform Training and Soft Parallel Decoding to enable aggressive parallelism in dLLMs, raising TPF on GSM8K from 2.04 to 5.47 and on MBPP from 2.71 to 5.86 while preserving accuracy.

  18. Discrete Stochastic Localization for Non-autoregressive Generation

    cs.LG 2026-02 unverdicted novelty 7.0 of 10

    Discrete Stochastic Localization lets a single trained network support an entire family of per-token SNR paths for discrete sequence generation, with masked diffusion as a special case, and improves MAUVE scores when ...

  19. DICE: Diffusion Large Language Models Excel at Generating CUDA Kernels

    cs.LG 2026-02 conditional novelty 7.0 of 10

    DICE, a family of diffusion LLMs trained with a bi-phase infilling-to-generation RL curriculum, achieves state-of-the-art CUDA kernel generation at data and token budgets far below prior specialized autoregressive models.

  20. Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models

    cs.LG 2026-02 unverdicted novelty 7.0 of 10

    Early and late denoising steps in masked diffusion LMs are robust to smaller-model replacement, enabling 17% FLOPs reduction with modest generative quality loss.

  21. PartDiffuser: Part-wise 3D Mesh Generation via Discrete Diffusion

    cs.CV 2025-11 unverdicted novelty 7.0 of 10

    PartDiffuser is a semi-autoregressive discrete diffusion framework that generates high-fidelity 3D meshes from point clouds by combining inter-part autoregression with intra-part parallel diffusion using a part-aware ...

  22. DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech

    eess.AS 2026-07 conditional novelty 6.5 of 10

    A LoRA-plus-convolution adaptation converts an AR TTS backbone into a confidence-ordered discrete diffusion model that improves WER and speed on limited data.

  23. S2D2: Fast Decoding for Diffusion LLMs via Training-Free Self-Speculation

    cs.CL 2026-03 conditional novelty 6.5 of 10

    Training-free self-speculation reuses a block-diffusion model’s block-size-1 mode as a local AR verifier, improving accuracy–speed tradeoffs over confidence-threshold decoding.

  24. Reducing Pretraining-Generation Mismatch in Diffusion Language Models

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A training objective that keeps prompts clean and applies diffusion only to the continuation improves diffusion language models on continuation benchmarks without changing inference.

  25. Thinking with Anchors: Grounded and Efficient Document Reasoning

    cs.CV 2026-08 conditional novelty 6.0 of 10

    ADOPD 2026 extends ADOPD 2024 with semantic tags, captions, and grounded reasoning traces, and its DocCount benchmark shows top VLMs reach just 72.85% exact-match accuracy.

  26. WAM-Diff2: Hierarchical AR-to-Diffusion Distillation for Highly Efficient Autonomous Driving VLA

    cs.RO 2026-08 conditional novelty 6.0 of 10

    A three-stage distillation converts an autoregressive driving VLA into a block-causal masked diffusion model, preserving planning accuracy while decoding 2.8x faster (15.1x with optimized kernels).

  27. Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMs

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Evolutionary Decoding, a selection-and-mutation test-time strategy, improves LLaDA 2.0 mathematical reasoning on AIME and AMC benchmarks.

  28. Hierarchical Domain Generalization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Over infinite domains, hierarchy-uniform domain generalization is impossible for every nontrivial hypothesis class; a length-generalization bound is a property of the length hierarchy, not a hierarchy-free guarantee.

  29. Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A diffusion multimodal LLM's early-layer MLP sparsity at denoising step 0 pinpoints the semantic boundary, enabling one-shot suffix truncation that raises throughput up to ~31x.

  30. Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

    cs.CL 2026-07 accept novelty 6.0 of 10

    Joint AR–diffusion training yields one tri-mode LM that switches AR, diffusion, and self-speculation, beating open AR/diffusion models on accuracy and tokens-per-forward.

  31. Multi-Block Diffusion Language Models

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    MBD-LMs raise average tokens per forward pass from 3.47 to 6.19 (and to 9.34 with DMax) via multi-block teacher forcing and optimized parallel decoding while holding or slightly improving accuracy on math and code tasks.

  32. Bifocal Diffusion Language Models: Asymmetric Bidirectional Context for Parallel Generation

    cs.IR 2026-06 unverdicted novelty 6.0 of 10

    R2LM combines causal attention with a reverse Mamba SSM sidecar to supply right-side context in dLLMs, claiming 2.4x-12.9x throughput gains over bidirectional dLLMs and 1.9x-2.9x over AR baselines while matching or ex...

  33. Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Models

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Causal-rCM unifies teacher-forcing and self-forcing distillation for autoregressive video diffusion, delivering a 2-step model with VBench-T2V score 84.63 and enabling interactive world models on Cosmos 3 using only s...

  34. Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    PAPO improves reasoning performance in diffusion LLMs by converting sparse terminal rewards into dense step-wise credit and replaying real high-uncertainty trajectories, reporting gains up to 42.2% on Countdown.

  35. Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    On-policy distillation from a frozen autoregressive teacher to a bidirectional student eliminates train-inference mismatch and enables data-efficient ARLM-to-DLM conversion.

  36. SemBlock: Semantic Boundary Dynamic Blocks for Diffusion LLMs

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    SemBlock adds semantic-boundary prediction to enable dynamic block decoding in diffusion LLMs and reports gains over fixed-block and AdaBlock baselines on GSM8K, IFEval, MATH, and HumanEval.

  37. AMix-2: Establishing Protein as a Native Modality in Large Language Models

    q-bio.BM 2026-05 unverdicted novelty 6.0 of 10

    AMix-2 unifies protein sequences and text in one LLM via shared tokens and block-wise diffusion modeling, introduces the ProteinArena benchmark, and reports competitive performance against task-specific protein models...

  38. GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    GDSD reduces RL for dLLMs to likelihood-free self-distillation via a normalization-free logit-matching objective, outperforming ELBO methods with more stable training on LLaDA-8B and Dream-7B.

  39. Optimus: Elastic Decoding for Efficient Diffusion LLM Serving

    cs.DC 2026-05 unverdicted novelty 6.0 of 10

    Optimus enables elastic decoding granularity adaptation in diffusion LLMs via chunked decoding and load-based scheduling to raise throughput under dynamic conditions.

  40. Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    Fast-dDrive is a block-diffusion VLA that reports SOTA ADE on WOD-E2E, 0.32 m L2 on nuScenes, and 12x throughput over AR baselines via section scaffolds and test-time rollout averaging.

  41. Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Orthrus unifies autoregressive LLMs and diffusion models via shared KV cache and consensus to enable up to 7.8x parallel token generation speedup with O(1) memory overhead and lossless results.

  42. Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs

    cs.CL 2026-05 conditional novelty 6.0 of 10

    Diffusion-LLM inference can be accelerated with one measured signal — representational drift — that decides which cached positions to refresh and which tokens to commit early, reaching 3.67 tokens per forward pass.

  43. Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamics

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    Query position is a first-order variable in dLLM ICL whose variance matches semantic quality impact; mitigated via Average Confidence metric and training-free Auto-ICL routing.

  44. LightningRL: Breaking the Accuracy-Parallelism Trade-off of Block-wise dLLMs via Reinforcement Learning

    cs.LG 2026-03 conditional novelty 6.0 of 10

    A GRPO-based post-training method for block-wise diffusion LLMs improves average tokens-per-forward from ~3 to 7.3 while preserving accuracy.

  45. MetaState: Persistent Working Memory Enhances Reasoning in Discrete Diffusion Language Models

    cs.CL 2026-03 conditional novelty 6.0 of 10

    Adding a 0.6%-parameter recurrent working memory to frozen discrete diffusion LLMs yields an average 4.5-percentage-point gain on GSM8K, MATH-500, HumanEval, and MBPP.

  46. Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Under an exact oracle denoiser derived from a ground-truth Markov chain, SEDD, MDLM, LLaDA, and ReMDM samplers show large transition-level error at few diffusion steps; GenPPL/MAUVE can improve or stay flat while tran...

  47. DODO: Discrete OCR Diffusion Models

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Block-based discrete diffusion can transcribe documents in parallel, roughly matching autoregressive OCR accuracy while cutting inference time by up to about 3x in a lower-accuracy fast variant.

  48. MAGE: All-[MASK] Block Already Knows Where to Look in Block Diffusion LLM

    cs.LG 2026-02 conditional novelty 6.0 of 10

    MAGE uses the initial all-masked block's attention to choose one sparse KV index set for the whole block, matching exact-attention accuracy at moderate budgets while cutting long-context latency roughly 1.5-6.3x.

  49. Training-Trajectory-Aware Token Selection

    cs.CL 2026-01 unverdicted novelty 6.0 of 10

    Training-Trajectory-Aware Token Selection (T3S) reconstructs the token-level training objective to overcome a performance bottleneck in continual distillation of reasoning capabilities from large to small language models.

  50. Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed

    cs.CL 2025-12 unverdicted novelty 6.0 of 10

    Efficient-DLM converts AR models to dLMs via block-wise causal attention and position-dependent masking, yielding higher accuracy and 2.7-4.5x throughput than Dream 7B and Qwen3 4B.

  51. Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Sparse-LaViDa speeds up masked diffusion LLMs by truncating masked tokens, using register tokens and a step-causal attention mask, reaching ~2-2.8x speedup over LaViDa-O with comparable generation quality.

  52. DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone

    cs.LG 2025-11 conditional novelty 6.0 of 10

    A masked diffusion language model built on a bidirectional Mamba backbone matches Transformer-based denoisers on quality while decoding with near-linear time scaling.

  53. CANDI: Hybrid Discrete-Continuous Diffusion Models

    cs.LG 2025-10 conditional novelty 6.0 of 10

    CANDI combines masked and Gaussian corruption in one noising process, letting discrete diffusion models use continuous gradients for joint updates and guidance.

  54. Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces

    cs.LG 2025-09 unverdicted novelty 6.0 of 10

    A method trains discrete diffusion policies for combinatorial RL by matching to a PMD-regularized target distribution, reporting SOTA performance and sample efficiency on DNA generation, macro-action, and multi-agent ...

  55. Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphs

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Spiffy speeds up diffusion LLM inference up to about 3x (and up to 7.9x with parallel decoding) by verifying multiple candidate unmasked states in one batched model call, while preserving greedy output.

  56. Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models

    cs.CL 2025-09 conditional novelty 6.0 of 10

    TraceRL aligns the RL objective with the DLM's actual step-by-step decoding, producing TraDo-4B/8B models that beat autoregressive baselines on math reasoning.

  57. Set Block Decoding is a Language Model Inference Accelerator

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Fine-tuning an autoregressive LLM with an extra masked-token loss lets it decode several non-consecutive future tokens per forward pass, giving 3-5x fewer passes at similar accuracy.

  58. Diffusion Language Models Know the Answer Before Decoding

    cs.CL 2025-08 conditional novelty 6.0 of 10

    DLMs show early answer convergence allowing Prophet to cut decoding steps by up to 3.4x on LLaDA-8B and Dream-7B while keeping output quality.

  59. Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference

    cs.CL 2025-08 unverdicted novelty 6.0 of 10

    Seed Diffusion Preview is a discrete diffusion language model that reaches 2146 tokens per second inference on H20 GPUs with competitive code benchmark performance, establishing a new speed-quality Pareto frontier.

  60. Generative Distribution Distillation

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Knowledge distillation is reformulated as conditional diffusion over teacher feature tokens, with class-center contraction replacing the classification loss, yielding state-of-the-art ImageNet distillation numbers.

See all 74 Pith citations

Pith tools

Reviewed May 15, 2026 · model on record in the stance chip above.