Pith. sign in

REVIEW 17 cited by

Energy-Based Diffusion Language Models for Text Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.21357 v4 pith:ODNYPODD submitted 2024-10-28 cs.CL cs.LG

Minkai Xu , Tomas Geffner , Karsten Kreis , Weili Nie , Yilun Xu , Jure Leskovec , Stefano Ermon , Arash Vahdat This is my paper
classification cs.CLcs.LG
keywords modelsdiffusiongenerationautoregressivelanguagemodelenergy-basedsampling
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Despite remarkable progress in autoregressive language models, alternative generative paradigms beyond left-to-right generation are still being actively explored. Discrete diffusion models, with the capacity for parallel generation, have recently emerged as a promising alternative. Unfortunately, these models still underperform the autoregressive counterparts, with the performance gap increasing when reducing the number of sampling steps. Our analysis reveals that this degradation is a consequence of an imperfect approximation used by diffusion models. In this work, we propose Energy-based Diffusion Language Model (EDLM), an energy-based model operating at the full sequence level for each diffusion step, introduced to improve the underlying approximation used by diffusion models. More specifically, we introduce an EBM in a residual form, and show that its parameters can be obtained by leveraging a pretrained autoregressive model or by finetuning a bidirectional transformer via noise contrastive estimation. We also propose an efficient generation algorithm via parallel important sampling. Comprehensive experiments on language modeling benchmarks show that our model can consistently outperform state-of-the-art diffusion models by a significant margin, and approaches autoregressive models' perplexity. We further show that, without any generation performance drop, our framework offers a 1.3$\times$ sampling speedup over existing diffusion models. Reproduced code is available at https://github.com/MinkaiXu/Energy-Diffusion-LLM.

Discussion (0). Sign in to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding

    cs.LG 2026-07 unverdicted novelty 7.0 of 10

    Set diffusion factorizes likelihood over arbitrary token sets and uses a set-causal diffusion architecture to support KV caching and any-order decoding, yielding improved speed-quality tradeoffs versus prior diffusion LMs.

  2. Unified Energy for Invariant and Independent Decoding in Diffusion Language Models

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    The paper introduces Uni-E, a unified energy for DLMs that accounts for model capacity, dependency and invariance, can be computed exactly, and corrects distribution shifts from dependency and invariance.

  3. AsyncPatch Diffusion: spatially-flexible image generation

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    AsyncPatch Diffusion introduces asynchronous per-region noise levels in diffusion models, proves a valid ELBO, and uses a controlled sampler to support spatially adaptive generation and native inpainting.

  4. Discrete Stochastic Localization for Non-autoregressive Generation

    cs.LG 2026-02 unverdicted novelty 7.0 of 10

    Discrete Stochastic Localization lets a single trained network support an entire family of per-token SNR paths for discrete sequence generation, with masked diffusion as a special case, and improves MAUVE scores when ...

  5. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding

    cs.CL 2025-05 conditional novelty 7.0 of 10

    Fast-dLLM adds reusable KV cache blocks and selective parallel decoding to diffusion LLMs, closing most of the speed gap with autoregressive models without retraining.

  6. Continuous Latent Diffusion Language Model

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    Cola DLM proposes a hierarchical latent diffusion model that learns a text-to-latent mapping, fits a global semantic prior in continuous space with a block-causal DiT, and performs conditional decoding, establishing l...

  7. Gumbel Distillation for Parallel Text Generation

    cs.CL 2026-03 conditional novelty 6.0 of 10

    Conditioning parallel decoders on Gumbel noise sampled from an autoregressive teacher's Gumbel-Max process improves generation quality on LM1B and OpenWebText.

  8. Diffusion Models Memorize in Training -- and Generalize in Inference

    cs.LG 2026-03 unverdicted novelty 6.0 of 10

    Diffusion models overfit denoising loss at intermediate noise but generalize in inference as model error smooths the flow field and sampling paths avoid memorized noisy training data.

  9. IDLM: Inverse-distilled Diffusion Language Models

    cs.LG 2026-02 reject novelty 6.0 of 10

    IDLM distills pretrained discrete diffusion language models into few-step generators, cutting inference steps by 4–64× with roughly matched GenPPL and entropy.

  10. Fine-Tuning Masked Diffusion for Provable Self-Correction

    cs.LG 2025-10 conditional novelty 6.0 of 10

    PRISM fine-tunes any masked diffusion model with a binary-cross-entropy loss so its new head provably estimates per-token quality p(x_i=y_i|y⊕m_i) and can remask low-quality tokens at inference.

  11. Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows

    cs.LG 2025-07 conditional novelty 6.0 of 10

    TarFlowLM models language in a continuous latent space with transformer-based autoregressive normalizing flows, using mixture-CDF and Rosenblatt couplings, and reports competitive NELBO on TEXT8 and OpenWebText.

  12. Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformers

    cs.LG 2026-07 conditional novelty 5.5 of 10

    SNLP reduces encrypted Transformer nonlinear depth from L sequential stages to (L−N)+K, cutting symbolic bootstraps ~2.65× with lower error amplification than sequential inference.

  13. Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformers

    cs.LG 2026-07 conditional novelty 5.0 of 10

    SNLP reduces symbolic FHE bootstraps from 53 to 20 on a 0.5B model with +1.2% PPL degradation and lower polynomial-error amplification than sequential inference.

  14. RichMap: A Reachability Map Balancing Precision, Efficiency, and Flexibility for Rich Robot Manipulation Tasks

    cs.RO 2026-04 unverdicted novelty 5.0 of 10

    RichMap refines classic grid reachability maps with S^2/SO(3) capacity bounds and asynchronous construction to deliver over 98% accuracy, low false positives, fast queries, and up to 26% gains in cross-embodiment diff...

  15. On the Separability of Information in Diffusion Models

    cs.LG 2025-09 conditional novelty 5.0 of 10

    Diffusion models devote most of their information budget to class-agnostic texture, and the small class-relevant slice is what classifier-free guidance amplifies.

  16. Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling

    cs.LG 2025-05 unverdicted novelty 5.0 of 10

    VADD augments masked diffusion models with an auxiliary recognition model and variational inference to implicitly model inter-dimensional correlations, yielding higher-quality samples than standard MDMs at low denoisi...

  17. A Survey on Diffusion Language Models

    cs.CL 2025-08 unverdicted novelty 3.0 of 10

    A comprehensive survey of diffusion language models covering taxonomy, training and inference techniques, and comparisons with autoregressive models.

Pith tools