Pith. sign in

REVIEW 3 major objections 7 minor 89 references

Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single diffusion model can generate text, images, and tables jointly in their native spaces, trained as a sum of per-modality score-matching losses.

desk verdict A genuinely new formal framework for multimodal diffusion with decoupled time variables, worth refereeing despite overstated 'native space' claims and a real but fixable gap in Theorem 2's boundary conditions. read the letter →

arxiv 2506.07903 v2 pith:LCCYRP2Z submitted 2025-06-09 cs.LG cs.AIcs.CV

Kevin Rojas , Yuchen Zhu , Sichen Zhu , Felix X.-F. Ye , Molei Tao This is my paper
classification cs.LGcs.AIcs.CV MSC 60J6060J2760H10
keywords multimodaldiffusiondecouplednoiseschedulesarbitrarystatespacesdenoisingMarkovmodelsscorematchingcontinuous-discretetext-to-imagegenerationtabulardatasynthesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that multimodal diffusion models can be built directly on the product of each modality's native state space — continuous vectors, discrete tokens, Riemannian manifolds — without tokenizers, autoencoders, or other preprocessing that forces data into one shared format. The key move is giving each modality its own independent noise time: image noise and text noise evolve on separate clocks. Under this design, the joint score-matching objective splits into a sum of unimodal conditional score-matching losses (Prop. 1), so training a multimodal model is no harder than adding per-modality losses together. A correspondingly decoupled backward process (Prop. 2) then yields the joint distribution as well as any conditional distribution, so one trained score network can sample image-text pairs, text from images, or images from text. If this is right, multimodal generation no longer hinges on the accuracy of encoders and decoders, and the framework extends to arbitrary combinations of state-space types.

What carries the argument

The machinery is the unified generator description of diffusion from the denoising-Markov-model perspective. Each modality's forward process is summarized by an infinitesimal generator $L_{X^i}$ on its own state space, and the joint forward process (5) is the product of these independent processes, one time index per modality. On top of this, the paper introduces the generalized explicit score matching objective $I_{\mathrm{GESM}}$ and its denoising/implicit equivalents $I_{\mathrm{GDSM}}$ and $I_{\mathrm{GISM}}$, where the parameterized $\beta_\theta$ plays the role of an unnormalized density whose score is learned. The load-bearing identity, derived in Prop. 1, is that the multimodal conditional score $\nabla \log p(x_t, y_s \mid x_0, y_0)$ equals the unimodal conditional score $\nabla \log p(x_t \mid x_0)$ when the two noises are independent given the initial data, so the joint denoising loss collapses into a sum of unimodal losses. The decoupled-time backward process (12) then turns the learned joint score into samples at marginals $p(x,y,T-t,T-s)$.

What would settle it

Run the method on a toy product distribution with analytically known joint score — e.g., a two-dimensional OU process paired with a two-state CTMC, with heavy-tailed initial data so boundary terms do not obviously vanish — and compute both IGDSM and IGESM for a fixed one-hidden-layer score network; if the two losses differ by a theta-dependent term, Theorem 2's equivalence fails and the learned scores are not guaranteed to be the joint score.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that independent per-modality noise does not complicate multimodal score learning; it simplifies it. For a forward process (5) where each modality $X^i$ evolves under its own Markov generator with its own time variable $t_i$, the authors define a generalized score-matching objective $I_{\mathrm{GESM}}$ and prove (Thm. 1) that it is minimized when the parameterized distribution $\beta_\theta$ is proportional to the joint forward marginal $p(x,t)$. They then show (Thm. 2) that this intractable objective is equivalent, up to constants, to a denoising variant $I_{\mathrm{GDSM}}$ that only requires the conditional forward marginals of each modality. Specializing to a continuous-discrete product $(\mathbb{R}^d \times \text{finite set})$, Prop. 1 establishes that $I_{\mathrm{GDSM}}$ decomposes into the standard unimodal score-matching loss for the continuous part plus the score-entropy loss for the discrete part, because by Bayes' rule and conditional independence the joint conditional score equals the unimodal conditional score. Prop. 2 provides a backward process running on decoupled time variables whose marginals are $p(x,y,T-t,T-s)$, which is what makes both unconditional joint sampling and conditional sampling (including conditioning on partially noised data) available from one model. For masked discrete modalities, Prop. 3 factorizes the discrete score into a clean-data prediction term, yielding a cross-entropy loss and letting the discrete score ignore its own time variable.

Load-bearing premise

The argument collapses if the integration-by-parts identities linking the denoising loss to score matching fail on the unbounded Euclidean modalities, or if the per-modality noises are not conditionally independent given the initial data.

Editorial extensions

If this is right

  • Training a multimodal diffusion model reduces to summing the standard unimodal denoising objectives for each modality, with a single score network that takes all noise times as inputs.
  • One trained model can sample the joint distribution unconditionally, or any modality conditioned on the others, including conditioning on partially noised data via the backward process (12).
  • Masked discrete modalities admit a score factorization that makes the discrete part a clean-token prediction (cross-entropy) loss and removes dependence on that modality's own noise time.
  • The decoupled-time backward process also yields a noisy guidance mechanism where the guiding model is a partially-noised condition rather than an unconditional score, with empirically better FID than the fully-noised baseline.
  • Because the framework is stated in terms of generators, the same recipe applies to products like continuous-discrete-Riemannian, demonstrated in the paper on SO(3) with discrete labels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same conditional-independence argument should carry over to flow-matching or stochastic-interpolant objectives on product spaces, suggesting a broader recipe for multimodal generative models beyond diffusion.
  • If per-modality generators commute in the relevant sense, existing pretrained unimodal diffusion models could be fine-tuned into a shared multimodal score network rather than trained from scratch — a route the paper names as future work.
  • The boundary-term assumption behind Theorem 2 is checkable: for heavy-tailed continuous data the equivalence between denoising and explicit score matching may break, so the framework's practical range could be tested by comparing learned joint scores against analytic ones on non-compact examples.
  • Decoupled times also let the sampler choose how fast each modality is generated; asymmetric time reparameterizations could partially order generation (e.g., text first, image second) within a single model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes a framework for multimodal diffusion models on product state spaces, where each modality follows its own Markov forward process with an independent time variable. The authors extend denoising Markov models to this setting, deriving a generalized explicit score-matching objective (GESM), a denoising objective (GDSM), and an implicit objective (GISM), and they prove equivalence among them (Theorems 1--2). For continuous-discrete data they show that the joint denoising objective decomposes into a sum of unimodal conditional score losses (Prop. 1), and they construct a decoupled-time backward process whose marginals equal the forward marginals at reversed times (Prop. 2). They also propose a 'noisy guidance' mechanism and validate the framework on text-image generation and mixed-type tabular data synthesis, with an additional Riemannian-discrete toy experiment. The central algorithmic recipe is to train one score network with per-modality losses and then sample unconditionally, conditionally, or jointly by choosing the decoupled time variables appropriately.

Significance. If the theoretical claims are made rigorous, the framework is a valuable unification: it reduces multimodal training to a sum of unimodal denoising losses, enables any-to-any conditional and unconditional generation from one model, and extends denoising Markov models to multiple time variables. The decoupled-time backward process (Prop. 2) and the loss decomposition (Prop. 1) are elegant and, as the reader's report notes, the derivation is self-contained rather than circular. The tabular experiments are repeated 20 times and the parameter efficiency (64K parameters) is a striking positive result. However, the theoretical route as written has a load-bearing gap in Theorem 2, and the image experiments rely on an 83M VAE, which tempers the 'native state space' claim. The central idea is defensible and likely fixable, but the manuscript needs substantive revision before the claims are fully supported.

major comments (3)
  1. [Sec. 3.2 / Appendix A.2] The proof of Theorem 2 applies the adjoint operators L_X^* by integration by parts on the unbounded Euclidean factor and silently drops boundary terms. The paper only says 'under weak technical assumptions' (Sec. 3.1) without stating them. For heavy-tailed data such as Cauchy marginals under an Ornstein-Uhlenbeck forward process, the boundary term need not vanish, so the claimed equivalence between IGESM and IGDSM/IGISM is not established for general state spaces as stated. Since Section 3.2 uses Theorem 2 to justify replacing the intractable explicit objective with the trainable denoising objective, this is load-bearing. I recommend either stating explicit regularity/decay assumptions (e.g., compactly supported or sufficiently fast-decaying densities and test functions) or, more simply, proving directly that the minimizer of IGDSM in Prop. 1 equals the joint score via the conditional score identity E_{x0,y0|xt,ys}[nabla log p_t(x_t|x0)] = nabla log p_{t,s}(x_t,y_s), which does not require boundary conditions.
  2. [Sec. 4.1 / Table 1] The abstract and introduction claim 'native generation' and 'without relying on pre-trained models or powerful extra encoders', but the image modality in the text-image experiment is generated through an 83M VAE, as explicitly listed in Table 1 under 'Extra Encoders'. This means the image modality is not modeled in native pixel space, and the comparison to methods that also use VAEs is less favorable than the text suggests. The authors should either remove image generation from the claim of native state-space generation, or retrain/evaluate without the VAE for at least a small-scale demonstration, or clearly state that the framework is native per modality but the image modality in this particular experiment is latent.
  3. [Sec. 4.1 / Table 1 and Fig. 4] The FID-30K results in Table 1 are reported without error bars, multiple seeds, or confidence intervals, and the FID-10K noisy-guidance curves in Fig. 4 also lack error bars. Given that the FID differences between methods are small (e.g., 16.16 vs. 11.10) and the noisy-guidance noise level sigma is selected on the basis of FID on the same evaluation set, the empirical claim of 'competitive performance' needs variance estimates and, ideally, a held-out model selection procedure. Without these, it is difficult to assess whether the reported improvements are significant.
minor comments (7)
  1. [Sec. 3.2 (Theorem 2)] Theorem 2 states that IGESM, IGDSM, and IGISM are 'equivalent up to constants'; the proof shows equivalence up to additive, theta-independent constants. Please state this explicitly to avoid ambiguity with multiplicative constants.
  2. [Sec. 3.2 (Eq. 5)] The vector of times t=(t_1,...,t_n) is overloaded with the scalar time variable used elsewhere; please use a different symbol for the vector, e.g., tau, in the joint process definition.
  3. [Sec. 3.3 (Prop. 1)] In the displayed statement of Prop. 1, the discrete loss is written with Q_s(y_s,y) for y != y_s; the proof uses a variable swap to obtain this form. This is correct, but the intermediate step 'exchange variable y_s and y' is difficult to follow; a short clarification would help.
  4. [Sec. B.1 (Qmask definition)] The definition of Qmask with 'the last row corresponds to M' is ambiguous. Please write the matrix explicitly with entries for all states and the mask state, and specify the sign convention for row sums.
  5. [Sec. C.4 / Table 1] Table 1 reports 481M parameters for the model, while Sec. C.4 states the total model contains 578M parameters. Please reconcile these numbers and clarify which parameter count is used in the comparisons.
  6. [Fig. 4] The legend labels sigma = 3, 5, 8, but the caption and text describe the x-axis as 'Noise Level' without units. Please state what these values represent (e.g., sigma in the forward noising schedule) and how they correspond to the condition noise level used in Table 4.
  7. [Sec. 6 (Limitations)] The limitations paragraph mentions only the lack of pretrained-initialization experiments. It should also acknowledge the reliance on an image VAE in the text-image experiments and the technical assumptions required for Theorem 2.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the joint score-matching decomposition and the decoupled-time backward-process marginals are derived from the stated generator formalism and the explicitly assumed conditional independence, not from fitted parameters or a self-citation chain.

full rationale

The central derivation chain is self-contained. The forward process in (5) is defined with conditionally independent per-modality Markov processes, and the proof of Prop. 1 in Appendix A.3 expands the IGDSM objective into the continuous generator term (1/2)g^2 ||s^X_theta - grad log p_t(x_t|x_0)||^2 and the discrete score-entropy term; the replacement of joint conditional scores by unimodal conditional scores uses the explicitly stated conditional-product structure p(x_t,y_s|x_0,y_0) = p(x_t|x_0)p(y_s|y_0). This is a consequence of the stated construction, not a restatement of the conclusion. Theorem 2 is an adjoint/integration-by-parts identity proved in Appendix A.2; Theorem 1 follows from Jensen's inequality; Proposition 2 verifies the reverse Fokker-Planck equations. No fitted parameter is renamed as a prediction: the only tuned values are inference hyperparameters (e.g., noise level 0.77, guidance interval [0.3,0.8] in Table 4), and the benchmark comparisons are external. Self-citations (Zhu et al. 2025a,b; Rojas et al. 2025; Ren et al. 2025a) appear only as related-work examples or as a suggested future sampler improvement, and none carries the load of the theoretical claims. The notable weakness is technical rather than circular: Section 3.1 defers to "weak technical assumptions" without stating them, and Appendix A.2 drops boundary terms on the unbounded Euclidean factor, so the GESM/GDSM equivalence may fail as stated for heavy-tailed data. That is a correctness or rigor gap, not a reduction of the result to its own inputs.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central theoretical framework does not introduce free parameters; it rests on standard generator and Fokker-Planck machinery and on the design choice of conditionally independent forward processes per modality. The fitted quantities listed are inference and training hyperparameters that affect reported benchmarks, not constants used in the derivation. No new physical or mathematical entity is postulated beyond the usual mask token of masked diffusion.

free parameters (4)
  • Noisy guidance condition noise level sigma = 0.77 (T2I default)
    Selected by scanning FID-10K on MS-COCO in Fig. 4; it directly affects the reported FID-30K and is tuned on the evaluation benchmark.
  • Guidance scale and guidance interval = omega=5.0 and [0.3,0.8] for T2I and joint generation
    Default inference hyperparameters in Table 4; chosen to improve FID or CLIP, with only the interval and sigma ablated in Fig. 4.
  • Noise schedule endpoints beta_start, beta_end, delta = text-image: 0.00085, 0.0120, 1e-5; tabular: 0.1, 20, 1e-5
    Standard VP and masked schedule constants chosen by hand; they affect generation quality but are not predicted by the theory.
  • Weighting between continuous and discrete losses = not specified numerically
    Sec. D.3 states the weighting parameter is chosen to balance the numerical and discrete loss, without giving a value or a selection protocol; it influences reported tabular metrics.
assumptions (5)
  • domain assumption Each unimodal forward process is a Feller process on its state space with a unique easy-to-sample stationary distribution, and terminal time T is large enough that the joint law at (T,T) is essentially the product of stationary distributions.
    Invoked in Sec. 3.2 around Eq. (5) and in Sec. B.1; the generative sampler starts from independent noise, which is only valid if the terminal marginals factor in this way.
  • domain assumption Per-modality forward processes are conditionally independent given initial data, so joint conditional scores factor into unimodal conditional scores.
    This is the key step in the proof of Prop. 1 that turns the joint training loss into a sum of unimodal losses; it holds by construction of Eq. (5) but is an assumption about how modalities are noised.
  • standard math Generator calculus and adjoint integration by parts are valid on product state spaces, including unbounded Euclidean factors, without boundary terms.
    The paper states weak technical assumptions in Sec. 3.1 and does not verify them; if boundary terms do not vanish, IGDSM and IGESM are not equivalent, so the training loss would not learn the joint score.
  • domain assumption The masked discrete forward process with an added mask token M is a valid CTMC and its transition law Eq. (17) holds; text is adequately represented by discrete tokens.
    Used in Prop. 3 to simplify the discrete loss to token-prediction cross-entropy; it requires tokenizing text, so text is not purely native.
  • standard math Existence and uniqueness of backward Markov and SDE processes (Anderson 1982; Kelly 2011).
    Invoked in Sec. 2 and in the proof of Prop. 2, which constructs the reverse-time dynamics from forward generators and score functions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces." pith.science (2026). https://pith.science/paper/LCCYRP2Z

@misc{pith2026250607903,
  author       = {Pith},
  title        = {Pith review of: Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LCCYRP2Z}},
  note         = {Machine review of arXiv:2506.07903}
}
read the original abstract

Diffusion models have demonstrated remarkable performance in generating unimodal data across various tasks, including image, video, and text generation. On the contrary, the joint generation of multimodal data through diffusion models is still in the early stages of exploration. Existing approaches heavily rely on external preprocessing protocols, such as tokenizers and variational autoencoders, to harmonize varied data representations into a unified, unimodal format. This process heavily demands the high accuracy of encoders and decoders, which can be problematic for applications with limited data. To lift this restriction, we propose a novel framework for building multimodal diffusion models on arbitrary state spaces, enabling native generation of coupled data across different modalities. By introducing an innovative decoupled noise schedule for each modality, we enable both unconditional and modality-conditioned generation within a single model simultaneously. We empirically validate our approach for text-image generation and mixed-type tabular data synthesis, demonstrating that it achieves competitive performance.

Figures

Figures reproduced from arXiv: 2506.07903 by the authors.

Figure 1
Figure 1. By injecting noise into different modalities in a decoupled fashion, we enable the unconditional and modality-conditioned generation in a single model. (a) Joint generation of image and text. (b) Image generation given text captions as conditions. (c) Text generation given images as conditions. multimodal data simultaneously through iterative denoising of a randomly sampled initial noise. A commonality among the afo… view at source ↗
Figure 2
Figure 2. Network backbone for text-image generation, motivated by MMDiT (Esser et al., 2024) and DiT (Peebles & Xie, 2023). 2. Preliminaries In this section, we review basic concepts and formulations of common diffusion models in different state spaces. 2.1. Continuous Space Diffusion Models For continuous diffusion models (Song et al., 2020; Ho et al., 2020), one considers a continuous time stochastic differen￾tial equation… view at source ↗
Figure 3
Figure 3. Visualization of samples generated by our approach. Captions are truncated for brevity. to a multimodal scenario. This enables us to perform genera￾tive modeling of data distributions consisting of mixed-type data without requiring complicated preprocessing pipelines. We begin by formally defining the forward process in terms of generators of Markov processes. Assume that we have a data distribution pdata defined on… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Performance of noisy guidance on MS-COCO FID-10K. We note that using partially noised conditions results in a better performance. A guidance interval of t ∈ [0.3, 0.8] was used. this enables us to design versatile conditional and uncondi￾tional generative sampling algo…
Figure 5
Figure 5. Figure 5: Visual representation of ground truth labeled Riemannian data [PITH_FULL_IMAGE:figures/full_fig_p027_5.png]
Figure 6
Figure 6. Figure 6: Peformance of inferring discrete label based on SO(3) data. E. Additional Experiment on Riemannian-Discrete Multimodal Diffusion Model In this section, we demonstrate another application of our proposed multimodal diffusion model framework by focusing on the combinatio…
Figure 7
Figure 7. Figure 7: Performance of joint generation of the labeled Riemannian data. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_7.png]
Figure 8
Figure 8. Figure 8: Code for generating the dataset F. Additional Numerical Results for Text-Image Generation CLIP Similarity We generate 5000 samples and evaluate the CLIP similarity between the text and image. For this evaluation, we use CLIP-ViT-large-patch14 and we limit the captions …
Figure 9
Figure 9. Figure 9: Visualization of texts generated conditioning on the images. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_9.png]
Figure 10
Figure 10. Figure 10: Visualization of images generated conditioning on the text caption. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_10.png]
Figure 11
Figure 11. Figure 11: Visualization of text-image pairs generated jointly and unconditionally. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

89 extracted references · 35 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Anderson, B. D. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12 0 (3): 0 313--326, 1982

    1982

  3. [3]

    All are worth words: A vit backbone for diffusion models

    Bao, F., Nie, S., Xue, K., Cao, Y., Li, C., Su, H., and Zhu, J. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 22669--22679, 2023 a

    2023

  4. [4]

    One transformer fits all distributions in multi-modal diffusion at scale

    Bao, F., Nie, S., Xue, K., Li, C., Pu, S., Wang, Y., Yue, G., Cao, Y., Su, H., and Zhu, J. One transformer fits all distributions in multi-modal diffusion at scale. In International Conference on Machine Learning, pp.\ 1692--1717. PMLR, 2023 b

    2023

  5. [5]

    From denoising diffusions to denoising markov models

    Benton, J., Shi, Y., De Bortoli, V., Deligiannidis, G., and Doucet, A. From denoising diffusions to denoising markov models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86 0 (2): 0 286--301, 2024

    2024

  6. [6]

    A continuous time framework for discrete denoising models

    Campbell, A., Benton, J., De Bortoli, V., Rainforth, T., Deligiannidis, G., and Doucet, A. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35: 0 28266--28279, 2022

    2022

  7. [7]

    Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design

    Campbell, A., Yim, J., Barzilay, R., Rainforth, T., and Jaakkola, T. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. In International Conference on Machine Learning, pp.\ 5453--5512. PMLR, 2024

    2024

  8. [8]

    Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 11315--11325, 2022

    2022

Show all 89 references
  1. [9]

    Z., Tran, S

    Chen, C., Ding, H., Sisman, B., Xu, Y., Xie, O., Yao, B. Z., Tran, S. D., and Zeng, B. Diffusion models for multi-modal generative modeling. 2024 a . URL https://arxiv.org/abs/2407.17571

    2024 arXiv

  2. [10]

    Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023

    arXiv

  3. [11]

    Chen, R. T. and Lipman, Y. Flow matching on general geometries. In The Twelfth International Conference on Learning Representations, 2024

    2024

  4. [12]

    S., Boffi, N

    Chen, Y., Goldstein, M., Hua, M., Albergo, M. S., Boffi, N. M., and Vanden-Eijnden, E. Probabilistic forecasting with stochastic interpolants and f \"o llmer processes. In Forty-first International Conference on Machine Learning, 2024 b

    2024

  5. [13]

    Cheng, A., Lo, A., Lee, K. L. K., Miret, S., and Aspuru-Guzik, A. Stiefel flow matching for moment-constrained structure elucidation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=84WmbzikPP

    2025

  6. [14]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., and Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, pp.\ 02783649241273668, 2023

    2023

  7. [15]

    W., and Doucet, A

    De Bortoli, V., Mathieu, E., Hutchinson, M., Thornton, J., Teh, Y. W., and Doucet, A. Riemannian score-based generative modelling. Advances in Neural Information Processing Systems, 35: 0 2406--2422, 2022

    2022

  8. [16]

    and Nichol, A

    Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34: 0 8780--8794, 2021

    2021

  9. [17]

    Taming transformers for high-resolution image synthesis

    Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 12873--12883, 2021

    2021

  10. [18]

    Scaling rectified flow transformers for high-resolution image synthesis

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., M \"u ller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024

    2024

  11. [19]

    T., Synnaeve, G., Adi, Y., and Lipman, Y

    Gat, I., Remez, T., Shaul, N., Kreuk, F., Chen, R. T., Synnaeve, G., Adi, Y., and Lipman, Y. Discrete flow matching. arXiv preprint arXiv:2407.15595, 2024

    2024 arXiv

  12. [20]

    Seed-x: Multimodal models with unified multi-granularity comprehension and generation

    Ge, Y., Zhao, S., Zhu, J., Ge, Y., Yi, K., Song, L., Li, C., Ding, X., and Shan, Y. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404.14396, 2024

    2024 arXiv

  13. [21]

    J., Oktay, D., Lin, Z., Verkuil, R., Tran, V

    Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wiggert, M., et al. Simulating 500 million years of evolution with a language model. Science, pp.\ eads0018, 2025

    2025

  14. [22]

    and Salimans, T

    Ho, J. and Salimans, T. Classifier-free diffusion guidance. In NeurIPS Workshop on Deep Generative Models and Downstream Applications, 2021. URL https://openreview.net/forum?id=qw8AKxfYbI

    2021

  15. [23]

    Denoising diffusion probabilistic models

    Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 0 6840--6851, 2020

    2020

  16. [24]

    Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. Video diffusion models. Advances in Neural Information Processing Systems, 35: 0 8633--8646, 2022

    2022

  17. [25]

    T., and Lipman, Y

    Holderrieth, P., Havasi, M., Yim, J., Shaul, N., Gat, I., Jaakkola, T., Karrer, B., Chen, R. T., and Lipman, Y. Generator matching: Generative modeling with arbitrary markov processes. arXiv preprint arXiv:2410.20587, 2024

    2024 arXiv

  18. [26]

    Simpler diffusion (sid2): 1.5 fid on imagenet512 with pixel-space diffusion

    Hoogeboom, E., Mensink, T., Heek, J., Lamerigts, K., Gao, R., and Salimans, T. Simpler diffusion (sid2): 1.5 fid on imagenet512 with pixel-space diffusion. arXiv preprint arXiv:2410.19324, 2024

    2024 arXiv

  19. [27]

    Hu, M., Zheng, C., Zheng, H., Cham, T.-J., Wang, C., Yang, Z., Tao, D., and Suganthan, P. N. Unified discrete diffusion for simultaneous vision-language generation. arXiv preprint arXiv:2211.14842, 2022

    2022 arXiv

  20. [28]

    Huang, C.-W., Aghajohari, M., Bose, J., Panangaden, P., and Courville, A. C. Riemannian diffusion models. Advances in Neural Information Processing Systems, 35: 0 2750--2761, 2022

    2022

  21. [29]

    and Dayan, P

    Hyv \"a rinen, A. and Dayan, P. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6 0 (4), 2005

    2005

  22. [30]

    B., Astolfi, P., Hall, M., Askari-Hemmat, R., Benchetrit, Y., Havasi, M., Muckley, M., Alahari, K., Romero-Soriano, A., Verbeek, J., et al

    Ifriqi, T. B., Astolfi, P., Hall, M., Askari-Hemmat, R., Benchetrit, Y., Havasi, M., Muckley, M., Alahari, K., Romero-Soriano, A., Verbeek, J., et al. On improved conditioning mechanisms and pre-training strategies for diffusion models. In NeurIPS 2024-Thirty-eighth Conference...

    2024

  23. [31]

    Pyramidal flow matching for efficient video generative modeling

    Jin, Y., Sun, Z., Li, N., Xu, K., Jiang, H., Zhuang, N., Huang, Q., Song, Y., Mu, Y., and Lin, Z. Pyramidal flow matching for efficient video generative modeling. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=...

    2025

  24. [32]

    Elucidating the design space of diffusion-based generative models

    Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35: 0 26565--26577, 2022

    2022

  25. [33]

    Guiding a diffusion model with a bad version of itself

    Karras, T., Aittala, M., Kynk \"a \"a nniemi, T., Lehtinen, J., Aila, T., and Laine, S. Guiding a diffusion model with a bad version of itself. arXiv preprint arXiv:2406.02507, 2024 a

    2024 arXiv

  26. [34]

    Analyzing and improving the training dynamics of diffusion models

    Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, S. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 24174--24184, 2024 b

    2024

  27. [35]

    Kelly, F. P. Reversibility and stochastic networks. Cambridge University Press, 2011

    2011

  28. [36]

    C., et al

    Kim, G., Martinez, A., Su, Y.-C., Jou, B., Lezama, J., Gupta, A., Yu, L., Jiang, L., Jansen, A., Walker, J. C., et al. A versatile diffusion transformer with mixture of noise levels for audiovisual generation. In The Thirty-eighth Annual Conference on Neural Information Proces...

    2024

  29. [37]

    Stasy: Score-based tabular data synthesis

    Kim, J., Lee, C., and Park, N. Stasy: Score-based tabular data synthesis. arXiv preprint arXiv:2210.04018, 2022

    2022 arXiv

  30. [38]

    Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  31. [39]

    C., Lo, W.-Y., et al

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4015--4026, 2023

    2023

  32. [40]

    Tabddpm: Modelling tabular data with diffusion models

    Kotelnikov, A., Baranchuk, D., Rubachev, I., and Babenko, A. Tabddpm: Modelling tabular data with diffusion models. In International Conference on Machine Learning, pp.\ 17564--17579. PMLR, 2023

    2023

  33. [41]

    Applying guidance in a limited interval improves sample and distribution quality in diffusion models

    Kynk \"a \"a nniemi, T., Aittala, M., Karras, T., Laine, S., Aila, T., and Lehtinen, J. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.07724, 2024

    2024 arXiv

  34. [42]

    Codi: Co-evolving contrastive diffusion models for mixed-type tabular synthesis

    Lee, C., Kim, J., and Park, N. Codi: Co-evolving contrastive diffusion models for mixed-type tabular synthesis. In International Conference on Machine Learning, pp.\ 18940--18956. PMLR, 2023

    2023

  35. [43]

    Layerdag: A layerwise autoregressive diffusion model for directed acyclic graph generation

    Li, M., Shitole, V., Chien, E., Man, C., Wang, Z., Sridharan, S., Zhang, Y., Krishna, T., and Li, P. Layerdag: A layerwise autoregressive diffusion model for directed acyclic graph generation. In The Thirteenth International Conference on Learning Representations, 2025. URL ht...

    2025

  36. [44]

    Omniflow: Any-to-any generation with multi-modal rectified flows

    Li, S., Kallidromitis, K., Gokul, A., Liao, Z., Kato, Y., Kozuka, K., and Grover, A. Omniflow: Any-to-any generation with multi-modal rectified flows. arXiv preprint arXiv:2412.01169, 2024 a

    2024 arXiv

  37. [45]

    Self-guidance: Boosting flow and diffusion generation on their own

    Li, T., Luo, W., Chen, Z., Ma, L., and Qi, G.-J. Self-guidance: Boosting flow and diffusion generation on their own. arXiv preprint arXiv:2412.05827, 2024 b

    2024

  38. [46]

    Microsoft COCO: common objects in context

    Lin, T., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Doll \' a r, P., and Zitnick, C. Microsoft COCO: common objects in context. In ECCV, 2014

    2014

  39. [47]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36, 2024

    2024

  40. [48]

    Goggle: Generative modelling for tabular data by learning relational structure

    Liu, T., Qian, Z., Berrevoets, J., and van der Schaar, M. Goggle: Generative modelling for tabular data by learning relational structure. In The Eleventh International Conference on Learning Representations, 2023

    2023

  41. [49]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Lou, A., Meng, C., and Ermon, S. Discrete diffusion modeling by estimating the ratios of the data distribution. International Conference on Machine Learning, 2024

    2024

  42. [50]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action

    Lu, J., Clark, C., Lee, S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., and Kembhavi, A. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 2...

    2024

  43. [51]

    scdiffusion: conditional generation of high-quality single-cell data using diffusion model

    Luo, E., Hao, M., Wei, L., and Zhang, X. scdiffusion: conditional generation of high-quality single-cell data using diffusion model. Bioinformatics, 40 0 (9): 0 btae518, 2024

    2024

  44. [52]

    Interpretation and generalization of score matching

    Lyu, S. Interpretation and generalization of score matching. arXiv preprint arXiv:1205.2629, 2012

    2012 arXiv

  45. [53]

    Chameleon: Mixed-modal early-fusion foundation models

    Meta, C. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024

    2024 arXiv

  46. [54]

    Scaling up masked diffusion models on text

    Nie, S., Zhu, F., Du, C., Pang, T., Liu, Q., Zeng, G., Lin, M., and Li, C. Scaling up masked diffusion models on text. In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openreview.net/forum?id=WNvvwK0tut

    2025

  47. [55]

    Large language diffusion models

    Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., and Li, C. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025 b

    2025 arXiv

  48. [56]

    Unlocking guidance for discrete state-space diffusion and flow models

    Nisonoff, H., Xiong, J., Allenspach, S., and Listgarten, J. Unlocking guidance for discrete state-space diffusion and flow models. arXiv preprint arXiv:2406.01572, 2024

    2024 arXiv

  49. [57]

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data

    Ou, J., Nie, S., Xue, K., Zhu, F., Sun, J., Li, Z., and Li, C. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=sMyXP8Tanm

    2025

  50. [58]

    and Xie, S

    Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4195--4205, 2023

    2023

  51. [59]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PMLR, 2021

    2021

  52. [60]

    Hierarchical text-conditional image generation with clip latents

    Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1 0 (2): 0 3, 2022

    2022 arXiv

  53. [61]

    M., and Ying, L

    Ren, Y., Chen, H., Rotskoff, G. M., and Ying, L. How discrete and continuous diffusion meet: Comprehensive analysis of discrete diffusion models via a stochastic integral framework. arXiv preprint arXiv:2410.03601, 2024

    2024 arXiv

  54. [62]

    M., Tao, M., and Ying, L

    Ren, Y., Chen, H., Zhu, Y., Guo, W., Chen, Y., Rotskoff, G. M., Tao, M., and Ying, L. Fast solvers for discrete diffusion models: Theory and applications of high-order algorithms. arXiv preprint arXiv:2502.00234, 2025 a

    2025

  55. [63]

    M., and Ying, L

    Ren, Y., Rotskoff, G. M., and Ying, L. A unified approach to analysis and design of denoising markov models. arXiv preprint arXiv:2504.01938, 2025 b

    2025 arXiv

  56. [64]

    Variational schrödinger momentum diffusion

    Rojas, K., Tian, Y., Tao, M., Nevmyvaka, Y., and Deng, W. Variational schrödinger momentum diffusion. arXiv preprint arXiv:2501.16675, 2025

    2025 arXiv

  57. [65]

    High-resolution image synthesis with latent diffusion models

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10684--10695, 2022

    2022

  58. [66]

    J., Jin, Q., and Guo, B

    Ruan, L., Ma, Y., Yang, H., He, H., Liu, B., Fu, J., Yuan, N. J., Jin, Q., and Guo, B. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 10219--10...

    2023

  59. [67]

    L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al

    Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35...

    2022

  60. [68]

    Simple and effective masked diffusion language models

    Sahoo, S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J., Rush, A., and Kuleshov, V. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37: 0 130136--130184, 2024

    2024

  61. [69]

    S., Phung, H., Wang, G., Boshar, S., Dalla-torre, H., de Almeida, B

    Schiff, Y., Sahoo, S. S., Phung, H., Wang, G., Boshar, S., Dalla-torre, H., de Almeida, B. P., Rush, A., Pierrot, T., and Kuleshov, V. Simple guidance mechanisms for discrete diffusion models. arXiv preprint arXiv:2412.10193, 2024

    2024 arXiv

  62. [70]

    Simplified and generalized masked diffusion for discrete data

    Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37: 0 103131--103167, 2024

    2024

  63. [71]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp.\ 2256--2265. PMLR, 2015

    2015

  64. [72]

    P., Kumar, A., Ermon, S., and Poole, B

    Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020

    2011 arXiv

  65. [73]

    K., Chen, R

    Sriram, A., Miller, B. K., Chen, R. T., and Wood, B. M. Flowllm: Flow matching for material generation with large language models as base distributions. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  66. [74]

    Unified multimodal discrete diffusion

    Swerdlow, A., Prabhudesai, M., Gandhi, S., Pathak, D., and Fragkiadaki, K. Unified multimodal discrete diffusion. arXiv preprint arXiv:2503.20853, 2025

    2025 arXiv

  67. [75]

    S., and Kolesnikov, A

    Tschannen, M., Pinto, A. S., and Kolesnikov, A. Jetformer: An autoregressive generative model of raw images and text. arXiv preprint arXiv:2411.19722, 2024

    2024 arXiv

  68. [76]

    Neural discrete representation learning

    Van Den Oord, A., Vinyals, O., et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017

    2017

  69. [77]

    A connection between score matching and denoising autoencoders

    Vincent, P. A connection between score matching and denoising autoencoders. Neural computation, 23 0 (7): 0 1661--1674, 2011

    2011

  70. [78]

    Emu3: Next-token prediction is all you need

    Wang, X., Zhang, X., Luo, Z., Sun, Q., Cui, Y., Wang, J., Zhang, F., Wang, Y., Li, Z., Yu, Q., et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024 a

    2024 arXiv

  71. [79]

    Diffusion language models are versatile protein learners

    Wang, X., Zheng, Z., Ye, F., Xue, D., Huang, S., and Gu, Q. Diffusion language models are versatile protein learners. In International Conference on Machine Learning, pp.\ 52309--52333. PMLR, 2024 b

    2024

  72. [80]

    L., Juergens, D., Bennett, N

    Watson, J. L., Juergens, D., Bennett, N. R., Trippe, B. L., Yim, J., Eisenach, H. E., Ahern, W., Borst, A. J., Ragotte, R. J., Milles, L. F., et al. De novo design of protein structure and function with rfdiffusion. Nature, 620 0 (7976): 0 1089--1100, 2023

    2023

  73. [81]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation

    Wu, C., Chen, X., Wu, Z., Ma, Y., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C., et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410.13848, 2024

    2024 arXiv

  74. [82]

    J., Wang, W., Lin, K

    Xie, J., Mao, W., Bai, Z., Zhang, D. J., Wang, W., Lin, K. Q., Gu, Y., Chen, Z., Yang, Z., and Shou, M. Z. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  75. [83]

    Versatile diffusion: Text, images and variations all in one diffusion model

    Xu, X., Wang, Z., Zhang, G., Wang, K., and Shi, H. Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 7754--7765, 2023

    2023

  76. [84]

    Discrete-state continuous-time diffusion for graph generation

    Xu, Z., Qiu, R., Chen, Y., Chen, H., Fan, X., Pan, M., Zeng, Z., Das, M., and Tong, H. Discrete-state continuous-time diffusion for graph generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  77. [85]

    L., De Bortoli, V., Mathieu, E., Doucet, A., Barzilay, R., and Jaakkola, T

    Yim, J., Trippe, B. L., De Bortoli, V., Mathieu, E., Doucet, A., Barzilay, R., and Jaakkola, T. Se (3) diffusion model with application to protein backbone generation. In International Conference on Machine Learning, pp.\ 40001--40039. PMLR, 2023

    2023

  78. [86]

    Mixed-type tabular data synthesis with score-based diffusion in latent space

    Zhang, H., Zhang, J., Srinivasan, B., Shen, Z., Qin, X., Faloutsos, C., Rangwala, H., and Karypis, G. Mixed-type tabular data synthesis with score-based diffusion in latent space. arXiv preprint arXiv:2310.09656, 2023

    2023 arXiv

  79. [87]

    Transfusion: Predict the next token and diffuse images with one multi-modal model

    Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024

    2024 arXiv

  80. [88]

    Diffusion generative modeling for spatially resolved gene expression inference from histology images

    Zhu, S., Zhu, Y., Tao, M., and Qiu, P. Diffusion generative modeling for spatially resolved gene expression inference from histology images. In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openreview.net/forum?id=FtjLUHyZAO

    2025

  81. [89]

    Trivialized momentum facilitates diffusion generative modeling on lie groups

    Zhu, Y., Chen, T., Kong, L., Theodorou, E., and Tao, M. Trivialized momentum facilitates diffusion generative modeling on lie groups. In The Thirteenth International Conference on Learning Representations, 2025 b . URL https://openreview.net/forum?id=DTatjJTDl1

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.