DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling

Ante Juki\'c; Arash Vahdat; Jean-Marie Lemercier; Karsten Kreis; Morteza Mardani; Tomas Geffner

arxiv: 2605.23605 · v1 · pith:7QR5RDTOnew · submitted 2026-05-22 · 💻 cs.LG · cs.AI· cs.CL

DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling

Jean-Marie Lemercier , Tomas Geffner , Karsten Kreis , Morteza Mardani , Arash Vahdat , Ante Juki\'c This is my paper

Pith reviewed 2026-05-25 04:37 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CL

keywords diffusion language modelslatent diffusionconsistency distillationmasked diffusionlanguage modelingauto-encodertext generationsampling acceleration

0 comments

The pith

A continuous latent space learned from masked diffusion language models captures token correlations and speeds up sampling.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper seeks to fix the inability of standard diffusion language models to model dependencies between output tokens, which forces a difficult choice between generation quality and computational speed. It does this by fine-tuning an auto-encoder on an existing masked diffusion model to produce a continuous latent representation, training a latent diffusion model on that representation, and then distilling the latent model into a fast consistency model. The resulting system is shown to exceed the baseline masked diffusion model in quality even before distillation, while cutting inference time, and distillation reduces the latent generation cost to almost nothing relative to the discrete decoding steps. A reader would care because diffusion-based language models have been limited by their slow sampling and weak coherence; removing that limitation would make them more competitive with autoregressive alternatives for practical text generation.

Core claim

DiLaDiff augments masked diffusion language models by learning a continuous latent space with semantic capabilities through an auto-encoder fine-tuned from the base model, training a latent diffusion model to capture the prior over the encoder outputs, and applying consistency distillation to obtain a few-step generator. Even without the distillation step the latent-guided model already surpasses the masked diffusion baseline while accelerating inference, and the distilled version makes latent sampling negligible in time compared with the subsequent discrete token decoding.

What carries the argument

The auto-encoder that produces a continuous semantic latent space from a masked diffusion language model, enabling a separate latent diffusion process to model token correlations before discrete decoding.

If this is right

The latent-guided diffusion model outperforms the masked diffusion baseline on quality metrics.
Inference accelerates substantially even before consistency distillation is applied.
After distillation the time to generate the latent becomes negligible relative to discrete token decoding.
The approach resolves part of the quality-throughput trade-off that diffusion language models face.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same latent-augmentation pattern could be tested on non-language diffusion tasks such as image or audio generation to see whether token or patch correlations improve similarly.
If the auto-encoder can be trained once and reused, the method might lower the cost of adapting diffusion models to new domains without retraining the entire discrete model.
The negligible latent cost after distillation suggests diffusion language models could eventually match the step count of autoregressive sampling while retaining parallel generation advantages.

Load-bearing premise

A continuous latent space with semantic capabilities can be learned by an auto-encoder fine-tuned from an existing masked diffusion language model and that this space enables the latent diffusion model to capture correlations between decoded tokens.

What would settle it

A direct comparison experiment in which the latent-guided model shows no improvement in perplexity or sample quality over the masked diffusion baseline, or in which inference time is not reduced.

Figures

Figures reproduced from arXiv: 2605.23605 by Ante Juki\'c, Arash Vahdat, Jean-Marie Lemercier, Karsten Kreis, Morteza Mardani, Tomas Geffner.

**Figure 2.** Figure 2: Speed-Quality Pareto frontier, for batch size [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 4.** Figure 4: Semantic similarity decoded sentences from same or different latents. 4.1 Learning text representations We evaluate the reconstruction performance of our auto-encoders on the held-out validation set of OpenWebText. The performance is evaluated in terms of the token recovery rate and the corresponding lower bound on perplexity (see Appendix G.1). As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

**Figure 5.** Figure 5: Temperature scaling and nucleus sampling. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

**Figure 6.** Figure 6: Diffusion schedules following the shape in [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗

**Figure 7.** Figure 7: Performance of MDLM and auto-encoder as a function of the masking ratio. Left: Token [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗

**Figure 8.** Figure 8: Cross- and self-attention contributions in decoder. Left: Fraction of the cross-attention over [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗

**Figure 9.** Figure 9: Confidence-based vs random token selection. [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗

**Figure 10.** Figure 10: Decoder robustness to latent noise on OpenWebText: MDLM and LaDiff with different compression factors. d GenPPL (↓) Entropy (↑) MAUVE (↑) Data - 14.8 5.44 1.00 LaDiff 2 92.0 5.52 0.78 LaDiff 5 71.4 5.45 0.80 LaDiff 10 62.3 5.40 0.82 LaDiff 15 75.4 5.45 0.81 [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗

**Figure 11.** Figure 11: Empirical decoding error rate and corresponding sigmoid-tanh fit ωfit(t) [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗

**Figure 13.** Figure 13: Ablations of self-distillation method for DiLaDiff. [PITH_FULL_IMAGE:figures/full_fig_p024_13.png] view at source ↗

**Figure 14.** Figure 14: γ-sampling with DiLaDiff. Ncont = 5, Ndisc = 32. Self-conditioning We show here our experiments regarding self-conditioning design in DiLaDiff. When using an extra MLP head to estimate the self-conditioning input during training, we can either use that prediction at each step, and therefore obtain self-conditioning for free, or compute an extra forward pass to compute zη(zt, 0) ∶= Φ (uη(zt, t, t), t), cf.… view at source ↗

**Figure 15.** Figure 15: DiLaDiff with/without extra-MLP head for predicting self-conditioning. [PITH_FULL_IMAGE:figures/full_fig_p025_15.png] view at source ↗

**Figure 16.** Figure 16: Speed-quality Pareto frontier for batch size [PITH_FULL_IMAGE:figures/full_fig_p026_16.png] view at source ↗

read the original abstract

Diffusion language models intrinsically fail to capture correlations between decoded tokens, which leads to a harsh trade-off between sampling quality and throughput. To solve this issue, we propose DiLaDiff, a variant of masked diffusion language models with three components: (1) a continuous latent space with semantic capabilities, learned by an auto-encoder fine-tuned from an existing masked diffusion language model; (2) a latent diffusion model learning the prior over the encoder distribution; (3) a consistency model distilling the learned prior into a few-step latent generative model. We show that, even without distillation, our latent-guided diffusion model outperforms the masked diffusion baseline while significantly accelerating inference. Consistency distillation further lowers the computational overhead of continuous diffusion, such that the latent is generated in negligible time compared to discrete decoding.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

DiLaDiff sketches a latent-augmented diffusion LM to fix token correlations but the abstract supplies zero metrics or details, so the claims stay uncheckable.

read the letter

The main takeaway is that this paper outlines a three-part architecture—latent auto-encoder fine-tuned from a masked diffusion LM, a latent diffusion prior, and consistency distillation—to address the known weakness of diffusion language models in capturing token correlations. The combination itself appears new for this setting and directly targets the quality-throughput trade-off mentioned in the abstract. That is a reasonable direction given existing work on non-autoregressive generation. The description of the components is clear at a high level and shows the authors have thought through how to move the diffusion process into a continuous space while keeping the decoder discrete. The claim that the latent-guided version already beats the baseline and speeds up sampling, with distillation making the latent step almost free, follows logically from the setup if the latent space works as hoped. The soft spot is large and central: the abstract states these performance gains without any numbers, datasets, baselines, or even a single metric. There is also no description of the auto-encoder fine-tuning objective, its architecture, or any check that the resulting latent space actually encodes semantic or correlation information (for example via reconstruction quality or dependency statistics). The stress-test concern lands cleanly here—the outperformance cannot be attributed to the proposed mechanism without evidence that the latent space improves correlation capture. The paper is aimed at researchers already working on diffusion or masked language models. Someone in that narrow group might extract an architectural idea worth trying, but the absence of results limits its usefulness right now. It does not yet deserve peer review; the core claims need experimental grounding first.

Referee Report

3 major / 1 minor

Summary. The manuscript proposes DiLaDiff, a masked diffusion language model augmented with (1) a continuous latent space learned by an auto-encoder fine-tuned from an existing masked diffusion LM, (2) a latent diffusion model over the encoder distribution, and (3) consistency distillation of the prior into a few-step generator. The central claim is that the latent-guided model (even without distillation) outperforms the masked diffusion baseline while accelerating inference, with distillation further reducing overhead so that latent generation becomes negligible relative to discrete decoding.

Significance. If the empirical claims hold and the latent space demonstrably improves token correlation capture, the work would address a recognized limitation of diffusion LMs (poor modeling of inter-token dependencies) and offer a practical route to higher-quality, faster sampling. The combination of latent augmentation with consistency distillation is a plausible direction, but its value depends on validation that the auto-encoder actually produces semantically useful latents rather than merely adding parameters.

major comments (3)

[Abstract] Abstract: performance claims ('outperforms the masked diffusion baseline while significantly accelerating inference') are stated without any metrics, baselines, datasets, or experimental protocol. This absence makes it impossible to evaluate whether the reported gains are attributable to the latent mechanism or to other factors.
[Abstract] Abstract (and implied §3–4): the key assumption that 'a continuous latent space with semantic capabilities' learned by fine-tuning an auto-encoder from a masked diffusion LM enables the latent diffusion model to capture correlations between decoded tokens is presented without any supporting detail on the fine-tuning objective, architecture, or quantitative evidence (e.g., reconstruction fidelity, mutual information between tokens, or dependency metrics on reconstructions). Without this, the central mechanistic claim cannot be assessed.
[Abstract] Abstract: the statement that 'the latent is generated in negligible time compared to discrete decoding' after consistency distillation is asserted without reporting wall-clock times, number of function evaluations, or comparison against the baseline sampling schedule, leaving the efficiency claim ungrounded.

minor comments (1)

[Abstract] Notation for the encoder/decoder and latent diffusion components should be introduced with explicit equations rather than prose descriptions alone.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive comments. We address each major comment point-by-point below and agree that targeted revisions to the abstract will improve clarity while the supporting details and evidence already appear in the body of the manuscript.

read point-by-point responses

Referee: [Abstract] Abstract: performance claims ('outperforms the masked diffusion baseline while significantly accelerating inference') are stated without any metrics, baselines, datasets, or experimental protocol. This absence makes it impossible to evaluate whether the reported gains are attributable to the latent mechanism or to other factors.

Authors: We agree that the abstract would be strengthened by including concrete metrics. The experimental protocol, datasets, baselines, and quantitative results are fully reported in Section 4. We will revise the abstract to reference key performance numbers and the experimental setup. revision: yes
Referee: [Abstract] Abstract (and implied §3–4): the key assumption that 'a continuous latent space with semantic capabilities' learned by fine-tuning an auto-encoder from a masked diffusion LM enables the latent diffusion model to capture correlations between decoded tokens is presented without any supporting detail on the fine-tuning objective, architecture, or quantitative evidence (e.g., reconstruction fidelity, mutual information between tokens, or dependency metrics on reconstructions). Without this, the central mechanistic claim cannot be assessed.

Authors: The fine-tuning objective, auto-encoder architecture, and quantitative evidence (reconstruction fidelity and dependency metrics) are presented in Section 3. We will revise the abstract to briefly reference these elements so the mechanistic claim is more explicitly grounded. revision: yes
Referee: [Abstract] Abstract: the statement that 'the latent is generated in negligible time compared to discrete decoding' after consistency distillation is asserted without reporting wall-clock times, number of function evaluations, or comparison against the baseline sampling schedule, leaving the efficiency claim ungrounded.

Authors: Section 4 reports the number of function evaluations, wall-clock times, and direct comparisons to the baseline sampling schedule. We will revise the abstract to cite these specific efficiency results. revision: yes

Circularity Check

0 steps flagged

Empirical construction with no circular derivation steps

full rationale

The paper describes an empirical method with three components—an auto-encoder for a continuous latent space, a latent diffusion model, and consistency distillation—applied to masked diffusion language models. The provided abstract and description contain no equations, fitted parameters, or self-citations that reduce the claimed outperformance or inference acceleration to a definition or self-referential quantity by construction. The central claims rest on experimental comparisons rather than mathematical identities or load-bearing self-referential arguments, making the derivation self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract supplies no explicit free parameters, axioms, or invented entities; evaluation is limited by absence of full text.

pith-pipeline@v0.9.0 · 5683 in / 981 out tokens · 20993 ms · 2026-05-25T04:37:30.451114+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages · 7 internal anchors

[1]

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

URLhttps://arxiv.org/abs/2604.11748. Cheng, C., Li, J., Peng, J., and Liu, G. Categorical flow matching on statistical manifolds,

work page internal anchor Pith review Pith/arXiv arXiv
[2]

Dhariwal, P

URL https://arxiv.org/abs/2405.16441. Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis.NeurIPS,

work page arXiv
[3]

Continuous diffusion for categorical data

URLhttps://arxiv.org/abs/2211.15089. Gat, I., Remez, T., Shaul, N., Kreuk, F., Chen, R. T. Q., Synnaeve, G., Adi, Y ., and Lipman, Y . Discrete flow matching.NeurIPS,

work page internal anchor Pith review Pith/arXiv arXiv
[4]

URL https://arxiv.org/abs/2602.16813. Li, X. L. et al. Diffusion-lm improves controllable text generation.NeurIPS,

work page internal anchor Pith review Pith/arXiv arXiv
[5]

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I

URL https://arxiv.org/abs/2510.22510. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners,

work page arXiv
[6]

Latent-Augmented Discrete Diffusion Models

URL https://arxiv.org/abs/2510.18114. Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K. Simplified and generalized masked diffusion for discrete data.NeurIPS,

work page internal anchor Pith review Pith/arXiv arXiv
[7]

CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language

URL https://arxiv.org/ abs/2603.20210. Watson, J. L., Juergens, D., Bennett, N. R., and et al. De novo design of protein structure and function with RFdiffusion.Nature, 620:1089–1100,

work page internal anchor Pith review Pith/arXiv arXiv
[8]

Xie, T., Xue, S., Feng, Z., Hu, T., Sun, J., Li, Z., and Zhang, C

doi: 10.1038/s41586-023-06415-8. Xie, T., Xue, S., Feng, Z., Hu, T., Sun, J., Li, Z., and Zhang, C. Variational autoencoding discrete diffusion with enhanced dimensional correlations modeling.ICLR,

work page doi:10.1038/s41586-023-06415-8
[9]

Dream 7B: Diffusion Large Language Models

Ye, J., Xie, Z., Zheng, L., Gao, J., Wu, Z., Jiang, X., Li, Z., and Kong, L. Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487,

work page internal anchor Pith review Pith/arXiv arXiv
[10]

Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner

URLhttps://arxiv.org/abs/2510.03206. Zhou, L., Parger, M., Haque, A., and Song, J. Terminal velocity matching.ICLR,

work page internal anchor Pith review Pith/arXiv arXiv
[11]

URL https://arxiv.org/abs/ 2510.22926. 12 A Proofs We prove here our main result in (10): Theorem A.1(LaDiff reverse posterior).Given a latent vector z capturing correlations in x0, and the conditional denoiser qθ(x0∣xt,z)∶=∏ ℓ δ(xℓ 0 =x ℓ θ(xt,z)) , the reverse posterior distribution of LaDiff is, fors<t: qθ s∣t(xs∣xt)∶=∫ ( ∏ ℓ qs∣t(xℓ s∣xℓ t,x ℓ 0 =x ℓ ...

work page arXiv 2026
[12]

and discrete flow matching (Gat et al., 2024). An essential step for bridging the gap to AR models was reached with mask-based diffusion practical enhancements (Sahoo et al., 2024; Zheng et al., 2025; Shi et al., 2024; Ou et al., 2025), and with uniform-state diffusion (Sahoo et al., 2025; Zhu et al., 2026). Large mask-based diffusion language models LLaD...

work page 2024
[13]

have become staples for larger-scale open research. Continuous diffusionEfforts for embedding text diffusion in a continuous space operate on representations include probability distributions (Cheng et al., 2025; Jo & Hwang, 2025), one-hot encodings (Lee et al.,

work page 2025
[14]

, token embeddings (Li et al., 2022; Dieleman et al., 2022; Gulrajani & Hashimoto, 2023; Chen et al.,

work page 2022
[15]

Closely related to our work is Chen et al

and contextual representations (Meshchaninov et al., 2025; Zhou et al., 2025). Closely related to our work is Chen et al. (2026) which defines self-distilled flow maps in a continuous token-wise embedding space. The main differences are: (1) we consider a hybrid continuous-discrete model while they design a purely continuous model, therefore concentrating...

work page 2025
[16]

and adaptive parallel decoding (Israel et al., 2025). Hybrid diffusionHybrid models can be coarsely divided into (1) cross-modality diffusion, where a parallel or joint diffusion process over discrete and continuous modalities of text is conducted, and (2) cascaded-modality diffusion, where a continuous latent is first generated then used as guide for dis...

work page 2025
[17]

15 C Training details C.1 Auto-encoding Following Meshchaninov et al

which we regard as concurrent to ours, although evolving in a different experimental framework (they focus on Python coding with 8B-scale models). 15 C Training details C.1 Auto-encoding Following Meshchaninov et al. (2025), the BERT encodings and latents are standardized coordinate- wise using statistics aggregated along training. This serves the triple-...

work page 2025
[18]

Following Meshchaninov et al

We use variance-preserving noise schedules as the latents are standardized coordinate-wise using aggregated statistics from the auto-encoder’s training. Following Meshchaninov et al. (2025), we 16 Algorithm 1Auto-encoder training Require: Frozen BERT, encoder Eϕ, decoder xθ, masking kernel qt(⋅∣x), regularization hyperpa- rametersp ˜x mask, σ˜x reg, pz ma...

work page 2025
[19]

Algorithm 2LaDiff training Require:encoderE ϕ, latent denoiserz ψ, schedulesα t, σt, latent statistics(µ z, σz) 1:z←E ϕ(x)▷encode text 2:z←(z−µz)/σz ▷normalize latent 3:samplet∼U(0,1) 4:sampleϵ∼N(0,I) 5:z t←αt z+σ t ϵ▷forward diffuse latent 6:ifrand()< 1 2 then 7: ˜z←zψ(zt, t,∅)▷self-conditioning prediction 8:else 9: ˜z←∅ ▷no self-conditioning 10:end if 1...

work page 2023
[20]

We sample t and r in the same logit-normal distribution LogitNormal(−1,1), where samples are taken from a Gaussian N(−1,1)and mapped to [0,1] using the logistic function

In practice, the second term in utgt is efficiently computed using the true Jacobian vector product(∂ zuη ∂tuη ∂ruη) T (v⊺1 0). We sample t and r in the same logit-normal distribution LogitNormal(−1,1), where samples are taken from a Gaussian N(−1,1)and mapped to [0,1] using the logistic function. At random with 25% chance, r is taken equal to t, thereby ...

work page 2048
[21]

the", "

Similar to the experiment on temperature-based logits rescaling in Section 4.1, LaDiff yields reasonable results when applying top-k confidence-based token selection, although the resulting sample entropy decreases severely from≈5.40 down to≈5.00, and MAUVE score suffers 22 significantly from this loss of diversity. In comparison, MDLM cannot exploit its ...

work page 2025
[22]

We further show here that they exhibit similar robustness profiles, i.e

As already demonstrated in Table 1, larger latent spaces have better token recovery rates. We further show here that they exhibit similar robustness profiles, i.e. that the decoder is very robust to noise levels below a standard deviation of 0.8, above which the token recovery rate drops steeply. As the transition region is located toward σ≈0.8, we skew t...

work page 2026
[23]

Ncont = 5, Ndisc =64

101 102 Ncont 0.60 0.65 0.70 0.75 0.80MAUVE MeanFlow ; LogitNormal(-1,1) MeanFlow-ExtraMLP ; LogitNormal(-1,1) T eacher 101 102 Ncont 80 90 100 110 120 130 140 150GenPPL MeanFlow ; LogitNormal(-1,1) MeanFlow-ExtraMLP ; LogitNormal(-1,1) T eacher 101 102 Ncont 5.475 5.500 5.525 5.550 5.575 5.600 5.625 5.650Entropy MeanFlow ; LogitNormal(-1,1) MeanFlow-Extr...

work page 2025
[24]

It’s just a small box, for sure, but it’s just a crime. They can’t keep them in themselves. They went there, it was horrible, but it was hard to think they could move it

use the setup in Sahoo et al. (2024). For transparency, the main differences between the respective setups are listed in Table 7, and mostly result from the different DiT implementation and different tokenizer (influencing the number of embedding and logit head parameters). Ours Sahoo et al. (2024) Architecture Absolute positional embedding QK norm RoPe (...

work page 2024
[25]

I’m very happy with the results, so I’m really looking forward to back racing after the race next year,

“I was introduced from the start of the season and had a very bad weather. When I started training, I kicked off at Turn 6 and then dropped off a little but didn’t know what else to do. I started training a little bit and was really very excited to race the next race. Now in the TRT is an amazing opportunity to continue in my four years in the RRA.” “Ferr...

work page 2007
[26]

move down,

There’s 150 investors plus a top tier investor per step vs. algorithm. Our algorithm carries a fixed list price. Next, we choose confidence in trading. Since moving down, some investors will move lower. Weeks 2, 3, 5 “move down,” stocks are more than 50, but by Weeks 30, investors move up. So, while we’ve picked the SPF over the last 4 years, we don’t wan...

work page 2014

[1] [1]

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

URLhttps://arxiv.org/abs/2604.11748. Cheng, C., Li, J., Peng, J., and Liu, G. Categorical flow matching on statistical manifolds,

work page internal anchor Pith review Pith/arXiv arXiv

[2] [2]

Dhariwal, P

URL https://arxiv.org/abs/2405.16441. Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis.NeurIPS,

work page arXiv

[3] [3]

Continuous diffusion for categorical data

URLhttps://arxiv.org/abs/2211.15089. Gat, I., Remez, T., Shaul, N., Kreuk, F., Chen, R. T. Q., Synnaeve, G., Adi, Y ., and Lipman, Y . Discrete flow matching.NeurIPS,

work page internal anchor Pith review Pith/arXiv arXiv

[4] [4]

URL https://arxiv.org/abs/2602.16813. Li, X. L. et al. Diffusion-lm improves controllable text generation.NeurIPS,

work page internal anchor Pith review Pith/arXiv arXiv

[5] [5]

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I

URL https://arxiv.org/abs/2510.22510. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners,

work page arXiv

[6] [6]

Latent-Augmented Discrete Diffusion Models

URL https://arxiv.org/abs/2510.18114. Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K. Simplified and generalized masked diffusion for discrete data.NeurIPS,

work page internal anchor Pith review Pith/arXiv arXiv

[7] [7]

CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language

URL https://arxiv.org/ abs/2603.20210. Watson, J. L., Juergens, D., Bennett, N. R., and et al. De novo design of protein structure and function with RFdiffusion.Nature, 620:1089–1100,

work page internal anchor Pith review Pith/arXiv arXiv

[8] [8]

Xie, T., Xue, S., Feng, Z., Hu, T., Sun, J., Li, Z., and Zhang, C

doi: 10.1038/s41586-023-06415-8. Xie, T., Xue, S., Feng, Z., Hu, T., Sun, J., Li, Z., and Zhang, C. Variational autoencoding discrete diffusion with enhanced dimensional correlations modeling.ICLR,

work page doi:10.1038/s41586-023-06415-8

[9] [9]

Dream 7B: Diffusion Large Language Models

Ye, J., Xie, Z., Zheng, L., Gao, J., Wu, Z., Jiang, X., Li, Z., and Kong, L. Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487,

work page internal anchor Pith review Pith/arXiv arXiv

[10] [10]

Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner

URLhttps://arxiv.org/abs/2510.03206. Zhou, L., Parger, M., Haque, A., and Song, J. Terminal velocity matching.ICLR,

work page internal anchor Pith review Pith/arXiv arXiv

[11] [11]

URL https://arxiv.org/abs/ 2510.22926. 12 A Proofs We prove here our main result in (10): Theorem A.1(LaDiff reverse posterior).Given a latent vector z capturing correlations in x0, and the conditional denoiser qθ(x0∣xt,z)∶=∏ ℓ δ(xℓ 0 =x ℓ θ(xt,z)) , the reverse posterior distribution of LaDiff is, fors<t: qθ s∣t(xs∣xt)∶=∫ ( ∏ ℓ qs∣t(xℓ s∣xℓ t,x ℓ 0 =x ℓ ...

work page arXiv 2026

[12] [12]

and discrete flow matching (Gat et al., 2024). An essential step for bridging the gap to AR models was reached with mask-based diffusion practical enhancements (Sahoo et al., 2024; Zheng et al., 2025; Shi et al., 2024; Ou et al., 2025), and with uniform-state diffusion (Sahoo et al., 2025; Zhu et al., 2026). Large mask-based diffusion language models LLaD...

work page 2024

[13] [13]

have become staples for larger-scale open research. Continuous diffusionEfforts for embedding text diffusion in a continuous space operate on representations include probability distributions (Cheng et al., 2025; Jo & Hwang, 2025), one-hot encodings (Lee et al.,

work page 2025

[14] [14]

, token embeddings (Li et al., 2022; Dieleman et al., 2022; Gulrajani & Hashimoto, 2023; Chen et al.,

work page 2022

[15] [15]

Closely related to our work is Chen et al

and contextual representations (Meshchaninov et al., 2025; Zhou et al., 2025). Closely related to our work is Chen et al. (2026) which defines self-distilled flow maps in a continuous token-wise embedding space. The main differences are: (1) we consider a hybrid continuous-discrete model while they design a purely continuous model, therefore concentrating...

work page 2025

[16] [16]

and adaptive parallel decoding (Israel et al., 2025). Hybrid diffusionHybrid models can be coarsely divided into (1) cross-modality diffusion, where a parallel or joint diffusion process over discrete and continuous modalities of text is conducted, and (2) cascaded-modality diffusion, where a continuous latent is first generated then used as guide for dis...

work page 2025

[17] [17]

15 C Training details C.1 Auto-encoding Following Meshchaninov et al

which we regard as concurrent to ours, although evolving in a different experimental framework (they focus on Python coding with 8B-scale models). 15 C Training details C.1 Auto-encoding Following Meshchaninov et al. (2025), the BERT encodings and latents are standardized coordinate- wise using statistics aggregated along training. This serves the triple-...

work page 2025

[18] [18]

Following Meshchaninov et al

We use variance-preserving noise schedules as the latents are standardized coordinate-wise using aggregated statistics from the auto-encoder’s training. Following Meshchaninov et al. (2025), we 16 Algorithm 1Auto-encoder training Require: Frozen BERT, encoder Eϕ, decoder xθ, masking kernel qt(⋅∣x), regularization hyperpa- rametersp ˜x mask, σ˜x reg, pz ma...

work page 2025

[19] [19]

Algorithm 2LaDiff training Require:encoderE ϕ, latent denoiserz ψ, schedulesα t, σt, latent statistics(µ z, σz) 1:z←E ϕ(x)▷encode text 2:z←(z−µz)/σz ▷normalize latent 3:samplet∼U(0,1) 4:sampleϵ∼N(0,I) 5:z t←αt z+σ t ϵ▷forward diffuse latent 6:ifrand()< 1 2 then 7: ˜z←zψ(zt, t,∅)▷self-conditioning prediction 8:else 9: ˜z←∅ ▷no self-conditioning 10:end if 1...

work page 2023

[20] [20]

We sample t and r in the same logit-normal distribution LogitNormal(−1,1), where samples are taken from a Gaussian N(−1,1)and mapped to [0,1] using the logistic function

In practice, the second term in utgt is efficiently computed using the true Jacobian vector product(∂ zuη ∂tuη ∂ruη) T (v⊺1 0). We sample t and r in the same logit-normal distribution LogitNormal(−1,1), where samples are taken from a Gaussian N(−1,1)and mapped to [0,1] using the logistic function. At random with 25% chance, r is taken equal to t, thereby ...

work page 2048

[21] [21]

the", "

Similar to the experiment on temperature-based logits rescaling in Section 4.1, LaDiff yields reasonable results when applying top-k confidence-based token selection, although the resulting sample entropy decreases severely from≈5.40 down to≈5.00, and MAUVE score suffers 22 significantly from this loss of diversity. In comparison, MDLM cannot exploit its ...

work page 2025

[22] [22]

We further show here that they exhibit similar robustness profiles, i.e

As already demonstrated in Table 1, larger latent spaces have better token recovery rates. We further show here that they exhibit similar robustness profiles, i.e. that the decoder is very robust to noise levels below a standard deviation of 0.8, above which the token recovery rate drops steeply. As the transition region is located toward σ≈0.8, we skew t...

work page 2026

[23] [23]

Ncont = 5, Ndisc =64

101 102 Ncont 0.60 0.65 0.70 0.75 0.80MAUVE MeanFlow ; LogitNormal(-1,1) MeanFlow-ExtraMLP ; LogitNormal(-1,1) T eacher 101 102 Ncont 80 90 100 110 120 130 140 150GenPPL MeanFlow ; LogitNormal(-1,1) MeanFlow-ExtraMLP ; LogitNormal(-1,1) T eacher 101 102 Ncont 5.475 5.500 5.525 5.550 5.575 5.600 5.625 5.650Entropy MeanFlow ; LogitNormal(-1,1) MeanFlow-Extr...

work page 2025

[24] [24]

It’s just a small box, for sure, but it’s just a crime. They can’t keep them in themselves. They went there, it was horrible, but it was hard to think they could move it

use the setup in Sahoo et al. (2024). For transparency, the main differences between the respective setups are listed in Table 7, and mostly result from the different DiT implementation and different tokenizer (influencing the number of embedding and logit head parameters). Ours Sahoo et al. (2024) Architecture Absolute positional embedding QK norm RoPe (...

work page 2024

[25] [25]

I’m very happy with the results, so I’m really looking forward to back racing after the race next year,

“I was introduced from the start of the season and had a very bad weather. When I started training, I kicked off at Turn 6 and then dropped off a little but didn’t know what else to do. I started training a little bit and was really very excited to race the next race. Now in the TRT is an amazing opportunity to continue in my four years in the RRA.” “Ferr...

work page 2007

[26] [26]

move down,

There’s 150 investors plus a top tier investor per step vs. algorithm. Our algorithm carries a fixed list price. Next, we choose confidence in trading. Since moving down, some investors will move lower. Weeks 2, 3, 5 “move down,” stocks are more than 50, but by Weeks 30, investors move up. So, while we’ve picked the SPF over the last 4 years, we don’t wan...

work page 2014