Pith. sign in

REVIEW 23 cited by

Scaling up Masked Diffusion Models on Text

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.18514 v3 pith:EPJWV7UZ submitted 2024-10-24 cs.AI cs.CLcs.LG

Shen Nie , Fengqi Zhu , Chao Du , Tianyu Pang , Qian Liu , Guangtao Zeng , Min Lin , Chongxuan Li This is my paper · ORCID
classification cs.AIcs.CLcs.LG
keywords mdmsarmsdatalanguagemodelsperformancescalingtext
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Masked diffusion models (MDMs) have shown promise in language modeling, yet their scalability and effectiveness in core language tasks, such as text generation and language understanding, remain underexplored. This paper establishes the first scaling law for MDMs, demonstrating a scaling rate comparable to autoregressive models (ARMs) and a relatively small compute gap. Motivated by their scalability, we train a family of MDMs with up to 1.1 billion (B) parameters to systematically evaluate their performance against ARMs of comparable or larger sizes. Fully leveraging the probabilistic formulation of MDMs, we propose a simple yet effective unsupervised classifier-free guidance that effectively exploits large-scale unpaired data, boosting performance for conditional inference. In language understanding, the 1.1B MDM outperforms the 1.1B TinyLlama model trained on the same data across four of eight zero-shot benchmarks. Notably, it achieves competitive math reasoning ability with the 7B Llama-2 model on the GSM8K dataset. In text generation, MDMs with 16 times more pre-training time offer a flexible trade-off against ARMs with the accelerated sampling technique KV-Cache: MDMs match ARMs in performance while being 1.4 times faster during sampling. Moreover, MDMs address challenging tasks for ARMs by effectively handling bidirectional reasoning and adapting to temporal shifts in data. Notably, a 1.1B MDM breaks the reverse curse encountered by much larger ARMs with significantly more data and computation, such as 13B Llama-2 and 175B GPT-3. Our code is available at https://github.com/ML-GSAI/SMDM.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 23 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Scaling Inherently Interpretable Language Models

    cs.CL 2026-08 conditional novelty 7.0 of 10

    Training a language model with a built-in concept bottleneck preserves compute-optimal scaling and yields interpretability metrics that improve with scale, demonstrated on an 8B causal diffusion model.

  2. Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL

    cs.AI 2026-05 conditional novelty 7.0 of 10

    Masked diffusion language models, not larger autoregressive LLMs, are the better building block for text-based world models in agentic RL, improving rollout fidelity, diversity, and downstream task success.

  3. d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning

    cs.CL 2025-04 conditional novelty 7.0 of 10

    A diffusion language model can be trained to reason via supervised fine-tuning plus a new policy-gradient RL method, diffu-GRPO, improving benchmark accuracy over the base model.

  4. Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMs

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Evolutionary Decoding, a selection-and-mutation test-time strategy, improves LLaDA 2.0 mathematical reasoning on AIME and AMC benchmarks.

  5. LaCache: Exact Caching and Precision-Adaptive Inference for Diffusion Large Language Models

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A training-free caching and per-group FP8 quantization framework reports ~1.3× standalone and up to 40.2× combined speedups for diffusion LLM inference while keeping benchmark accuracy roughly stable.

  6. Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

    cs.CL 2026-07 accept novelty 6.0 of 10

    Joint AR–diffusion training yields one tri-mode LM that switches AR, diffusion, and self-speculation, beating open AR/diffusion models on accuracy and tokens-per-forward.

  7. From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons

    cs.CL 2026-04 conditional novelty 6.0 of 10

    FLUID adapts AR LLMs into strictly causal diffusion models with entropy-driven elastic horizons, matching strong AR performance on reasoning and code while using only billions of adaptation tokens.

  8. Gumbel Distillation for Parallel Text Generation

    cs.CL 2026-03 conditional novelty 6.0 of 10

    Conditioning parallel decoders on Gumbel noise sampled from an autoregressive teacher's Gumbel-Max process improves generation quality on LM1B and OpenWebText.

  9. CANDI: Hybrid Discrete-Continuous Diffusion Models

    cs.LG 2025-10 conditional novelty 6.0 of 10

    CANDI combines masked and Gaussian corruption in one noising process, letting discrete diffusion models use continuous gradients for joint updates and guidance.

  10. Inpainting-Guided Policy Optimization for Diffusion Large Language Models

    cs.LG 2025-09 reject novelty 6.0 of 10

    IGPO injects partial ground-truth reasoning hints into masked diffusion LLM generation during RL sampling, restoring non-zero advantages and improving math benchmarks, but as written the RL training set overlaps the e...

  11. Any-Order Flexible Length Masked Diffusion

    cs.LG 2025-08 conditional novelty 6.0 of 10

    FlexMDM is a discrete diffusion model that provably supports any-order generation over variable-length sequences by learning an insertion expectation alongside the unmasking posterior, validated by length-fidelity, ma...

  12. Amadeus: Autoregressive Model with Bidirectional Attribute Modelling for Symbolic Music

    cs.SD 2025-08 conditional novelty 6.0 of 10

    Amadeus generates symbolic music by autoregressively predicting note-level latents and decoding their attributes in parallel with a masked discrete diffusion model, yielding faster and more controllable generation tha...

  13. ViLaD: A Large Vision Language Diffusion Framework for End-to-End Autonomous Driving

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    ViLaD applies masked diffusion, instead of autoregressive decoding, to generate whole driving decision sequences in parallel within a vision-language model, reporting faster and more accurate planning on nuScenes.

  14. Jailbreaking Large Language Diffusion Models: Revealing Hidden Safety Flaws in Diffusion-Based Text Generation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    PAD, a jailbreak that injects sequence connectors into the parallel denoising positions of diffusion language models, achieves up to 97% attack success on LLaDA and MMaDA variants.

  15. DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 7B masked-diffusion code model plus complementary-mask GRPO (coupled-GRPO) improves benchmark scores and shifts decoding away from strict left-to-right order.

  16. Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Adaptive Classifier-Free Guidance (A-CFG) re-masks low-confidence tokens in the unconditional input at each generation step, improving reasoning and planning accuracy for masked diffusion language models.

  17. FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A 1.5B unified multimodal model trained with discrete flow matching and metric-induced probability paths matches autoregressive baselines of similar size on generation and understanding benchmarks.

  18. Theoretical Benefit and Limitation of Diffusion Language Model

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Masked diffusion language models have a metric-dependent efficiency tradeoff: near-optimal perplexity in constant steps, but sequence-level correctness needs linearly many steps in the worst case.

  19. Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Masked diffusion models trained order-agnostically can solve puzzles better than autoregressive models when inference unmasking order is chosen adaptively by confidence.

  20. Enabling Autoregressive Models to Fill In Masked Tokens

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Combining a frozen MLM and AR model with a trained linear layer enables autoregressive models to perform masked token infilling with KV-cached inference.

  21. UniCMs: A Unified Consistency Model For Efficient Multimodal Generation and Understanding

    cs.CV 2025-02 conditional novelty 6.0 of 10

    UniCMs applies consistency distillation to a unified multimodal transformer, treating image mask-diffusion steps and text parallel-decoding steps as one shared denoising trajectory, enabling 2 to 8 step generation and...

  22. A Survey on Latent Reasoning

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A survey that organizes latent reasoning methods into vertical recurrence, horizontal recurrence, and infinite-depth diffusion, arguing that silent reasoning can beat explicit chain-of-thought.

  23. [MASK] is All You Need

    cs.CV 2024-12 conditional novelty 4.0 of 10

    Discrete Interpolants frames image generation, segmentation, and video generation as unmasking discrete [MASK] tokens, connecting masked generative models and discrete diffusion models.

Pith tools