Pith. sign in

REVIEW 29 cited by

Simplified and Generalized Masked Diffusion for Discrete Data

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.04329 v4 pith:HBOV2F7K submitted 2024-06-06 cs.LG stat.ML

Jiaxin Shi , Kehang Han , Zhe Wang , Arnaud Doucet , Michalis K. Titsias This is my paper
classification cs.LGstat.ML
keywords modelsdiffusionmaskeddiscretemodelingautoregressivedataframework
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Masked (or absorbing) diffusion is actively explored as an alternative to autoregressive models for generative modeling of discrete data. However, existing work in this area has been hindered by unnecessarily complex model formulations and unclear relationships between different perspectives, leading to suboptimal parameterization, training objectives, and ad hoc adjustments to counteract these issues. In this work, we aim to provide a simple and general framework that unlocks the full potential of masked diffusion models. We show that the continuous-time variational objective of masked diffusion models is a simple weighted integral of cross-entropy losses. Our framework also enables training generalized masked diffusion models with state-dependent masking schedules. When evaluated by perplexity, our models trained on OpenWebText surpass prior diffusion language models at GPT-2 scale and demonstrate superior performance on 4 out of 5 zero-shot language modeling tasks. Furthermore, our models vastly outperform previous discrete diffusion models on pixel-level image modeling, achieving 2.75 (CIFAR-10) and 3.40 (ImageNet 64x64) bits per dimension that are better than autoregressive models of similar sizes. Our code is available at https://github.com/google-deepmind/md4.

Discussion (0). Sign in to comment.

Forward citations

Cited by 29 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Large Language Diffusion Models

    cs.CL 2025-02 unverdicted novelty 8.0 of 10

    LLaDA is a scalable diffusion-based language model that matches autoregressive LLMs like LLaMA3 8B on tasks and surpasses GPT-4o on reversal poem completion.

  2. Autoregressive Boltzmann Generators

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    ArBG replaces flow-based methods with autoregressive models for Boltzmann sampling, showing gains on peptide benchmarks and a 132M-parameter model Robin cutting zero-shot energy error by over 60% on 8-residue systems.

  3. Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics

    cs.CL 2026-06 accept novelty 7.0 of 10

    Naive samplers beat published diffusion and flow models on gen-PPL with incoherent output, proving the metric unsound and motivating distributional evaluation suites.

  4. Adaptive Order Policies for Masked Diffusion

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    A policy network learns to choose unmasking order in masked diffusion by reweighting the loss, outperforming random and heuristic baselines on ordering-sensitive tasks.

  5. Learned Relay Representations for Forward-Thinking Discrete Diffusion Models

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Learned Relay Representations enable masked diffusion models to propagate useful latent information across denoising steps, scaling to Fast-dLLM v2 to outperform supervised finetuning on coding tasks while cutting inf...

  6. Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster

    stat.ML 2026-05 unverdicted novelty 7.0 of 10

    FLDD learns non-Markovian marginal and posterior distributions for the forward process so a factorized reverse process can match the target better and produce higher-quality samples in fewer steps.

  7. Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    TABOM is a trajectory-aligned Boltzmann modeling framework that turns self-distilled inference paths into a pairwise ranking loss to close the training-inference gap in diffusion language models and expand their effec...

  8. Block-R1: Rethinking the Role of Block Size in Multi-domain Reinforcement Learning for Diffusion Large Language Models

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Introduces Block-R1 benchmark, Block-R1-41K dataset, and a conflict score to handle domain-specific optimal block sizes in RL post-training of diffusion LLMs.

  9. Block-R1: Rethinking the Role of Block Size in Multi-domain Reinforcement Learning for Diffusion Large Language Models

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Block-R1 formulates domain block size conflicts in multi-domain RL for dLLMs, releases a 41K-sample dataset with per-sample best block sizes and a conflict score, and provides a benchmark plus simple cross-domain trai...

  10. Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models

    cs.LG 2026-02 unverdicted novelty 7.0 of 10

    Early and late denoising steps in masked diffusion LMs are robust to smaller-model replacement, enabling 17% FLOPs reduction with modest generative quality loss.

  11. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding

    cs.CL 2025-05 conditional novelty 7.0 of 10

    Fast-dLLM adds reusable KV cache blocks and selective parallel decoding to diffusion LLMs, closing most of the speed gap with autoregressive models without retraining.

  12. Diffusion and Flow Matching Models for Tabular Data: A Survey

    cs.LG 2025-02 unverdicted novelty 7.0 of 10

    First dedicated survey organizing diffusion and flow matching models for tabular data synthesis, imputation, anomaly detection, and related tasks, covering literature from 2015 to 2026 and highlighting open problems.

  13. Bifocal Diffusion Language Models: Asymmetric Bidirectional Context for Parallel Generation

    cs.IR 2026-06 unverdicted novelty 6.0 of 10

    R2LM combines causal attention with a reverse Mamba SSM sidecar to supply right-side context in dLLMs, claiming 2.4x-12.9x throughput gains over bidirectional dLLMs and 1.9x-2.9x over AR baselines while matching or ex...

  14. Adversarial Diffusion Across Modalities: A Fusion Survey of Attacks, Defenses, and Evaluation for Text, Vision, and Vision-Language Models

    cs.CR 2026-06 unverdicted novelty 6.0 of 10

    A narrative survey that catalogs fifty papers on diffusion-based adversarial techniques across text, vision, and vision-language models, proposes a six-class taxonomy of diffusion roles plus a unified five-dimension e...

  15. Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Models

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Causal-rCM unifies teacher-forcing and self-forcing distillation for autoregressive video diffusion, delivering a 2-step model with VBench-T2V score 84.63 and enabling interactive world models on Cosmos 3 using only s...

  16. Improved Large Language Diffusion Models

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    iLLaDA is an 8B masked diffusion LM trained from scratch with bidirectional attention, reporting gains of 14-21 points on BBH, ARC, MATH and HumanEval over prior diffusion models while remaining competitive with Qwen2.5-7B.

  17. Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics

    cs.CL 2026-06 conditional novelty 6.0 of 10

    Zero-parameter naive samplers achieve state-of-the-art generative perplexity while producing incoherent text, proving the metric is unsound; distributional divergences like MAUVE and energy distance correctly rank the...

  18. Fixed-Point Masked Generative Modeling

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    FP-MGMs with consistency loss and three-state reuse (CoFRe) reduce parameters by up to 38.8% and improve low-budget perplexity and FID versus standard masked generative models on text and images.

  19. GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    GDSD reduces RL for dLLMs to likelihood-free self-distillation via a normalization-free logit-matching objective, outperforming ELBO methods with more stable training on LLaDA-8B and Dream-7B.

  20. Learned Relay Representations for Forward-Thinking Discrete Diffusion Models

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Learned Relay Representations add a differentiable per-token channel to masked diffusion models so they can propagate latent information across iterative denoising steps, yielding better coding performance and up to 3...

  21. Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    TABOM models inference unmasking preferences as a Boltzmann distribution over predictive entropies and derives a ranking loss to align DLM training with observed trajectories, yielding gains in new domains and reduced...

  22. VQ-SAD: Vector Quantized Structure Aware Diffusion For Molecule Generation

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    VQ-SAD combines a pretrained VQ-VAE with diffusion models by using its codebooks as discrete tokenizers for atoms and bonds, yielding slight improvements over prior diffusion methods on QM9 and ZINC250k.

  23. Generative Frontiers: Why Evaluation Matters for Diffusion Language Models

    cs.LG 2026-04 conditional novelty 6.0 of 10

    Generative perplexity and entropy are shown to be the two additive components of KL divergence to a reference distribution, motivating generative frontiers as a principled evaluation method for diffusion language models.

  24. CANDI: Hybrid Discrete-Continuous Diffusion Models

    cs.LG 2025-10 conditional novelty 6.0 of 10

    CANDI combines masked and Gaussian corruption in one noising process, letting discrete diffusion models use continuous gradients for joint updates and guidance.

  25. Diffusion Language Models Know the Answer Before Decoding

    cs.CL 2025-08 conditional novelty 6.0 of 10

    DLMs show early answer convergence allowing Prophet to cut decoding steps by up to 3.4x on LLaDA-8B and Dream-7B while keeping output quality.

  26. Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference

    cs.CL 2025-08 unverdicted novelty 6.0 of 10

    Seed Diffusion Preview is a discrete diffusion language model that reaches 2146 tokens per second inference on H20 GPUs with competitive code benchmark performance, establishing a new speed-quality Pareto frontier.

  27. LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    LLaDA-V is a diffusion-based multimodal large language model that reaches competitive or state-of-the-art results on visual instruction tasks while using a non-autoregressive architecture.

  28. Scaling Diffusion Language Models via Adaptation from Autoregressive Models

    cs.CL 2024-10 conditional novelty 6.0 of 10

    Adapting autoregressive models via continual pre-training yields diffusion language models from 127M to 7B parameters that outperform prior diffusion models and compete with their autoregressive counterparts on langua...

  29. Masked Diffusion Language Models with Frequency-Informed Training

    cs.CL 2025-09 conditional novelty 4.0 of 10

    Masked diffusion language models trained on 100M words match a hybrid GPT-BERT baseline on BabyLM tests, with a rare-word-focused masking variant.

Pith tools