Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that language modeling can run entirely in a continuous latent space: each token becomes a learnable Gaussian blob, a transformer-based autoregressive flow models the blobs' joint distribution, and the result matches the…

desk verdict A clean, mostly correct flow construction with one genuinely new Rosenblatt coupling, but the headline likelihood claim rests on an NELBO upper bound that does not beat the discrete baseline. read the letter →

arxiv 2507.00425 v1 pith:ZK42YHBL submitted 2025-07-01 cs.LG cs.CL

Ruixiang Zhang , Shuangfei Zhai , Jiatao Gu , Yizhe Zhang , Huangjie Zheng , Tianrong Chen , Miguel Angel Bautista , Josh Susskind
show 1 more author
Navdeep Jaitly
This is my paper
classification cs.LGcs.CL
keywords continuouslatentlanguagemodelingautoregressivenormalizingflowsmixtureofGaussiansMixture-CDFflowRosenblatttransformationvariationalautoencoderblock-wisetextgenerationlikelihood-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that language modeling need not choose discrete tokens one at a time: each token is encoded as a learnable isotropic Gaussian blob in a continuous latent space, and a stack of transformer-based autoregressive normalizing flows models the joint distribution of those blobs instead of the token sequence itself. Its load-bearing theoretical step is an exact equivalence: any mixture of Gaussians — precisely the distribution the encoder's overlapping blobs form — can be converted into an exact, invertible normalizing-flow layer, in one dimension by composing the mixture CDF with the inverse normal CDF, and in any dimension by a sequential Rosenblatt transform whose conditional mixture weights are updated by Bayes' rule. If this equivalence is right, text modeling gains structural flexibility that discrete autoregressive models reject by design: stacked flows with alternating directions provide global bi-directional context, patches of several tokens can be generated jointly, intermediate layers can be decoded to show coarse-to-fine refinement, and the number of mixture components acts as a tunable internal vocabulary independent of the data's token count. On text8 the model reaches 1.30 bits per character and on OpenWebText 22.64 validation perplexity, placing it in the middle of the discrete-diffusion landscape in the paper's comparison table, and a formal limit argument (Proposition 4) shows the objective collapses to ordinary next-token cross-entropy when the codebook variances shrink to zero — so the continuous framework contains standard discrete autoregressive modeling as a limiting case.

What carries the argument

The load-bearing object is the probability integral transform applied to Gaussian mixtures: the map $z\mapsto\Phi^{-1}(F_{\mathrm{mix}}(z))$ through a mixture CDF, whose inverse is the mixture quantile function. For a 1D mixture this map is the whole layer, and its Jacobian identity $\log|\partial u/\partial z| = \log p_{\mathrm{mix}}(z) - \log\mathcal{N}(u;0,1)$ means fitting the mixture parameters is equivalent to fitting the flow. For a $d$-dimensional isotropic mixture there is no tractable joint CDF, so the paper invokes Rosenblatt's theorem: apply the 1D transform to $z_1$ marginally, then to $z_2$ conditional on $z_1$, and so on; Lemma 1 shows each conditional density of an isotropic MoG is itself a 1D MoG whose weights are posterior component probabilities, updated sequentially by Bayes' rule in Algorithm 1. Around this core sits the tied encoder-decoder (each token $k$ is a learnable isotropic Gaussian $\mathcal{N}(\boldsymbol{\mu}_k,\sigma_k^2 I)$, and the decoder is the uniform-prior Bayesian posterior over components) and the stacked flow prior, where each layer maximizes the mixture log-density of its input while pushing its output toward a standard Gaussian, with progressive layer-wise training freezing earlier blocks before deeper ones are added.

What would settle it

The cleanest check is the zero-variance limit of Proposition 4: take a trained TarFlowLM model on OpenWebText, freeze the codebook means, multiply every $\sigma_k$ by a decreasing factor, and recompute validation NELBO. The theorem predicts the objective approaches the discrete transformer's cross-entropy as the factor goes to zero; a large residual gap — or an advantage that persists at near-zero variance — would show the continuous-space gain lives outside the claimed limiting mechanism. A second, cheaper check is the tied decoder's top-1 reconstruction accuracy on held-out text: since every token is decoded from a Gaussian blob by Bayesian posterior, that accuracy directly measures how much of the data survives in the latent codebook that the flow must model.

Watch

Extended reading notes

Core claim

The paper's central discovery is that mixture densities and invertible flow layers are the same object. Proposition 1 states that any one-dimensional mixture of Gaussians $p_{\mathrm{mix\text{-}1}}(z)=\sum_{k=1}^{V}\pi_k\mathcal{N}(z;m_k,\sigma_k^2)$ is carried exactly to a standard normal by the map $u=\Phi^{-1}(F_{\mathrm{mix\text{-}1}}(z))$, where $F_{\mathrm{mix\text{-}1}}$ is the mixture's cumulative distribution function and $\Phi^{-1}$ is the inverse standard-normal CDF — the paper calls this layer a 1D Mixture-CDF flow. Proposition 2 lifts the same idea to $d$ dimensions: an isotropic mixture $p_{\mathrm{mix\text{-}d}}(\mathbf{z})=\sum_{k=1}^{V}\pi_k\mathcal{N}(\mathbf{z};\mathbf{m}_k,\sigma_k^2 I_d)$ is realized as a Mixture-Rosenblatt flow, which transforms coordinates one at a time, at step $i$ applying the 1D Mixture-CDF map to $z_i$ conditional on $z_{<i}$ and updating component weights by Bayes' rule; the paper proves this map is a global $C^\infty$ diffeomorphism. Because each layer's log-determinant is exactly $\log p_{\mathrm{mix}} - \log\mathcal{N}(u;0,I)$, the change-of-variables formula makes the flow an exact density model of the latent sequence. The paper closes the circle with Proposition 4: when the encoder's Gaussian codebook is tied to the prior and its variances shrink to zero, the ELBO converges to the cross-entropy of a discrete autoregressive model, so discrete next-token language modeling is the zero-variance limit of this continuous construction.

Load-bearing premise

The load-bearing premise is that a per-token isotropic Gaussian codebook with a tied Bayesian decoder — no contextual encoder, no learned covariance structure — can carry enough of each token's identity that the flow prior has something coherent to model; the paper never measures how much information that fixed codebook loses, and if it loses too much, no flow flexibility can compensate.

Editorial extensions

If this is right

  • On both benchmarks the model lands inside the discrete-diffusion range: 1.30 BPC on text8 and 22.64 perplexity on OpenWebText, lightly behind the strongest masked-diffusion baseline (MD4, 22.13) and clearly ahead of absorbing diffusion (SEDD, 24.10) on OpenWebText.
  • Block-wise generation becomes consistent rather than violating the model: patching 2, 4, or more tokens per step is a first-class operation of the latent flow, and larger patches reduce forward FLOPs for a 1024-token sequence by shifting work from sequence length to flow depth.
  • Alternating left-to-right and right-to-left flow layers give each token access to global context in both directions while inference remains a sequential autoregressive pass.
  • The internal vocabulary (the number of mixture components per layer) is a free design lever: on text8 anywhere from 2 to 27 components performs near-identically, and on OpenWebText 64 components match the full 50,257-tokenizer codebook.
  • Intermediate flow layers can be decoded, exposing step-by-step coarse-to-fine text formation; the paper reports shrinking centroid movement and rising intra-sequence cosine similarity with depth, a window into generation that discrete token spaces do not offer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported NELBO is an upper bound on the true negative log-likelihood, so its parity with discrete baselines is not a direct log-likelihood comparison; an IWAE-style multi-sample bound would settle how much of the result is modeling power versus bound structure, a check the paper does not perform.
  • Proposition 4 predicts a continuous interpolation: as the codebook variances shrink, TarFlowLM should degrade into a discrete AR transformer. Tracing that path empirically would isolate what the finite-variance soft regime contributes and would turn the paper's limit theorem into a quantitative instrument rather than a conceptual one.
  • The Mixture-Rosenblatt layer is a generic gadget for any discrete signal embedded as isotropic Gaussian latents — music, biological sequences, code — and the patch-size mechanism transfers directly; the paper's flexibility claims are therefore testable outside natural language without new theory.
  • The paper's stated sampling-speed limitation is partially addressable by its own knobs: larger patches shorten the sequential decode horizon, and the decodable intermediate layers suggest a distillation target where a student learns to jump several flow layers at once.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes TarFlowLM, a language model that operates on continuous latent representations of tokens rather than discrete tokens. A factorized Gaussian encoder maps each token to a d-dimensional latent vector, and an autoregressive prior over latent sequences is defined by stacking transformer-based normalizing flows. The paper's main theoretical contribution is to show that a 1D mixture of Gaussians can be realized exactly as a flow layer via the probability integral transform (Mixture-CDF layer), and a d-dimensional isotropic mixture of Gaussians via the Rosenblatt transformation (Mixture-Rosenblatt layer). The ELBO objective is derived, and a connection to discrete autoregressive models is given in a zero-variance limit. Experiments on TEXT8 and OpenWebText report bits-per-character and perplexity upper bounds (NELBO), along with ablations and demonstrations of block-wise generation, flexible mixture sizes, and intermediate text editing.

Significance. The mathematical core is sound: the probability integral transform and Rosenblatt transformation arguments are exact, and the change-of-variables computations are careful. The paper is among the first to apply autoregressive flows with mixture-based couplings to language modeling in a VAE framework, and the proposed layers are interesting building blocks. The framework's potential flexibility (patch-size generation, alternating directions, layer-wise training) is a genuine contribution. However, the quantitative likelihood claims rest on variational upper bounds, and the gap between NELBO and true negative log-likelihood is not quantified, which weakens the 'strong likelihood performance' statement in the abstract and conclusion.

major comments (3)
  1. [Section 4, Table 2; Eq. (9); Abstract and Conclusion] All TarFlowLM likelihood numbers are NELBO upper bounds, yet the paper compares them directly with exact likelihoods from discrete autoregressive and diffusion baselines. For OpenWebText, the reported <=22.64 upper bound on perplexity does not establish competitiveness with Transformer AR's exact 17.54, since exp(NELBO) >= true perplexity, and the paper never estimates the tightness of the bound (e.g., via importance sampling or annealed importance sampling). The abstract and conclusion say 'strong likelihood performance' without this hedging. This is load-bearing for the central viability claim and should be fixed either by providing tighter estimates of the marginal likelihood or by explicitly reframing the contribution as architectural flexibility with upper-bound likelihoods.
  2. [Section 3.1, Eq. (3.1)] The tied per-token isotropic Gaussian codebook is the only bridge between discrete text and the continuous space, and its sufficiency is not analyzed. The flow prior cannot recover information that the encoder discards, so the viability claim implicitly assumes the codebook preserves enough information. The paper provides no reconstruction metrics, no mutual-information estimates, and no ablation that varies the encoder/decoder independently of the flow prior. Qualitative samples in Table 3 suggest the codebook is reasonable, but a quantitative evaluation of the encoder/decoder is needed.
  3. [Section 4, Figures 2 and 3] The ablation results are presented without error bars or multiple seeds, and some reported differences (e.g., Mix-1 CDF at 3.21 vs. Mix-d Rosenblatt at 3.12 in Figure 2) are small. If these differences are within noise, the claims about the relative benefit of Rosenblatt vs. CDF couplings and of progressive layer-wise training are not supported. The main likelihood result in Table 2 also appears to be a single run. Adding error bars or reporting the number of seeds would increase confidence in the empirical conclusions.
minor comments (5)
  1. [Section 4, 'Perplexity evaluation'] The text states that models are trained on 'three standard benchmarks' but only TEXT8 and OpenWebText are listed.
  2. [Eq. (8)] The left-hand side should read log p_mix-d(z_t|z_<t); the logarithm is missing.
  3. [General] There are several typos: 'equivallent' in Section 3.2, 'brievity' in Section 3.2.1, and 'highlighing' in the Figure 6 caption.
  4. [Figure 5] The y-axis labels (e.g., '827x1030x1179x...') are cryptic; the multiplicative ratios should be defined in the caption or main text.
  5. [Section 4, 'Flexible vocabulary size'] The claim that using 64 mixture components on OpenWebText matches the full tokenizer size of 50257 is not supported by any figure or table; only TEXT8 is shown in Figure 4.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the mixture-CDF and Mixture-Rosenblatt claims are exact constructed equivalences, and the same-author citation for the flow architecture is not load-bearing.

full rationale

The central theoretical claims (Propositions 1 and 2) are self-contained mathematical constructions, not fitted-input predictions. The paper explicitly defines the flow map as u=Phi^{-1}(F_mix(z)) and the d-dimensional version via sequential Rosenblatt transforms, then proves via the probability integral transform and chain rule that log|det J| = log p_mix - log N(u). This is an identity by construction: the flow density is defined to equal the mixture density, so there is no parameter fitted to a target and then renamed as a prediction. Proposition 4 is an explicitly stated consistency limit (sigma_k^2 -> 0) showing the ELBO term approaches a discrete cross-entropy; it is not used to define or justify the model's claims. The citation [71] to the authors' prior work introduces the Transformer-based autoregressive flow architecture family, but the load-bearing derivations and experiments in this paper are independent of that citation; the architecture choice is not justified solely by the self-citation. Appendix D openly compares the mixture-CDF coupling to Flow++'s mixture-of-logistics coupling and proves functional equivalence, so this is an honest connection rather than a concealed renaming. The reported NELBO values in Table 2 are labeled with '<=' and are variational upper bounds on true likelihood, so the comparison with exact-NLL baselines is a quantitative-correctness risk (the ELBO gap is not quantified), but it is not circular: the numbers are held-out evaluations, not quantities forced by the objective. The Limitations section only flags sampling efficiency, which is orthogonal to circularity. Overall, the derivation chain is not circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new hypothetical entities such as particles or forces. Its free parameters are architecture choices and hyperparameters. The main hidden assumptions are the sufficiency of the simple Gaussian codebook and the tightness of the ELBO as a performance measure.

free parameters (4)
  • Latent dimension d = 16 for OpenWebText, 5 for TEXT8
    The dimensionality of each token's continuous latent vector is a key architectural free parameter; it is chosen per dataset without a principled selection rule.
  • Mixture components V = 64 (OpenWebText Mix-1 and layers 2-3 of Mix-d), 50257 (first layer Mix-d), 27 or 2 (TEXT8)
    The number of Gaussian components in the mixture coupling layers; the paper argues it can be tuned independently of the vocabulary size, and shows robustness in Figure 4.
  • Number of flow layers and transformer layers per flow = 3 flow layers; [2,2,8] transformer layers (Mix-1); 3 flow blocks (Mix-d)
    Depth of the stacked flow prior is an architecture choice, not derived from data; the ablation shows it matters for performance.
  • Patch size for block-wise generation = 2 in Figure 5 and related experiments
    When demonstrating block-wise generation, the patch size is a free parameter; the paper does not systematically tune it.
assumptions (4)
  • standard math Probability integral transform: for a continuous random variable, applying its CDF yields a uniform variable, and composing with the inverse standard normal CDF yields a standard normal.
    Used in Proposition 1 and the 1D Mixture-CDF layer (Eq. 10 and Appendix A).
  • standard math Rosenblatt transformation: sequentially applying conditional CDFs of a joint density maps the vector to i.i.d. uniform, hence to i.i.d. standard normal variables.
    Basis for Proposition 2 and Algorithm 1 in Appendix B.
  • domain assumption The per-token isotropic Gaussian codebook with tied Bayesian decoder is expressive enough to support language modeling.
    The whole framework relies on the factorized encoder q(zt|xt) = N(zt; mu_k, sigma_k^2 I) and the tied decoder (Section 3.1); if this codebook loses information needed for decoding, the flow prior cannot recover it.
  • domain assumption The reported NELBO is a faithful proxy for true likelihood when comparing models.
    The paper reports NELBO as 'strong likelihood performance' but does not assess how tight the variational bound is; this is a standard assumption for VAE-based methods but remains unverified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows." pith.science (2026). https://pith.science/paper/ZK42YHBL

@misc{pith2026250700425,
  author       = {Pith},
  title        = {Pith review of: Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZK42YHBL}},
  note         = {Machine review of arXiv:2507.00425}
}
read the original abstract

Autoregressive models have driven remarkable progress in language modeling. Their foundational reliance on discrete tokens, unidirectional context, and single-pass decoding, while central to their success, also inspires the exploration of a design space that could offer new axes of modeling flexibility. In this work, we explore an alternative paradigm, shifting language modeling from a discrete token space to a continuous latent space. We propose a novel framework TarFlowLM, that employs transformer-based autoregressive normalizing flows to model these continuous representations. This approach unlocks substantial flexibility, enabling the construction of models that can capture global bi-directional context through stacked, alternating-direction autoregressive transformations, support block-wise generation with flexible token patch sizes, and facilitate a hierarchical multi-pass generation process. We further propose new mixture-based coupling transformations designed to capture complex dependencies within the latent space shaped by discrete data, and demonstrate theoretical connections to conventional discrete autoregressive models. Extensive experiments on language modeling benchmarks demonstrate strong likelihood performance and highlight the flexible modeling capabilities inherent in our framework.

Figures

Figures reproduced from arXiv: 2507.00425 by the authors.

Figure 1
Figure 1. Our approach transforms discrete token sequences into a continuous latent space modeled [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Ablation on model variants. 0 25000 50000 75000 100000 125000 150000 175000 200000 Training Steps 3 4 5 6 7 8 9 10 11 NELBO Effect of Coupling: Affine vs. Mixture Affine Coupling Mixture Coupling [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 5
Figure 5. Generative PPL evaluation with patch size 2 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figures from the paper (3 more)
Figure 6
Figure 6. Figure 6: Demonstration of the stacked autoregressive flows for text editing in a continuous latent [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Left: Latent evolution analysis. Right: Transformer FLOPs comparison. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Comparison of training and validation loss curves across different model configurations. [PITH_FULL_IMAGE:figures/full_fig_p034_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Masked diffusion language models fail to exploit their any-order interface because of positional uncertainty; insertion-based (FlexMDM) and latent-segment (LatentMDM) variants recover distinct any-order inference beha...

Reference graph

Works this paper leans on

99 extracted references · 54 canonical work pages · cited by 1 Pith paper

  1. [1]

    Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg

    Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, edi- tors,Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Informati...

    2021

  2. [2]

    Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 ofProceedings of Machine Learning Research,...

    2019

  3. [3]

    Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio

    Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In Stefan Riezler and Yoav Goldberg, editors,Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 10–21, Berlin, Germany, 2016. Association for Computational Linguistics. doi: 10....

    doi:10.18653/v1/k16-1002 2016

  4. [4]

    Large scale GAN training for high fidelity natural image synthesis

    Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: //openreview.net/forum?id=B1xsqj09Fm

    2019

  5. [5]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz L...

    2020

  6. [6]

    A continuous time framework for discrete denoising models

    Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligianni- dis, and Arnaud Doucet. A continuous time framework for discrete denoising models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural In- formation Pro...

    2022

  7. [7]

    Gener- ative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design, 2024

    Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Gener- ative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design, 2024. URLhttps://arxiv.org/abs/2402.04997

    arXiv 2024

  8. [8]

    Block neural autoregressive flow

    Nicola De Cao, Wilker Aziz, and Ivan Titov. Block neural autoregressive flow. In Amir Globerson and Ricardo Silva, editors,Proceedings of the Thirty-Fifth Conference on Uncer- tainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, volume 115 ofProceedings of Machine Learning Research, pages 1263–1273. AUAI Press, 2019. URL http:/...

    2019

Show all 99 references
  1. [9]

    Go with the flow: Adaptive control for neural odes

    Mathieu Chalvidal, Matthew Ricci, Rufin VanRullen, and Thomas Serre. Go with the flow: Adaptive control for neural odes. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: //openreview....

    2021

  2. [10]

    Maximum-likelihood augmented discrete generative adversarial networks.ArXiv preprint, abs/1702.07983, 2017

    Tong Che, Yanran Li, Ruixiang Zhang, R Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. Maximum-likelihood augmented discrete generative adversarial networks.ArXiv preprint, abs/1702.07983, 2017. URLhttps://arxiv.org/abs/1702.07983

    2017 arXiv

  3. [11]

    Neural ordinary differential equations

    Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors,Advances in Neural Information Processing Systems 31...

    2018

  4. [12]

    Ting Chen, Ruixiang Zhang, and Geoffrey E. Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URLhttps...

    2023

  5. [13]

    Cyprien de Masson d’Autume, Shakir Mohamed, Mihaela Rosca, and Jack W. Rae. Train- ing language gans from scratch. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors,Advances in Neu- ral Information Process...

    2019

  6. [14]

    Residual energy-based models for text generation

    Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc’Aurelio Ranzato. Residual energy-based models for text generation. In8th International Conference on Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:/...

    2020

  7. [15]

    Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler

    Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler. Continuous diffusion for categor- ical data, 2022. ...

    2022 arXiv

  8. [16]

    Nice: Non-linear independent components estimation.International Conference on Learning Representations workshop Track, 2014

    Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation.International Conference on Learning Representations workshop Track, 2014

    2014

  9. [17]

    Density estimation using real NVP

    Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In5th International Conference on Learning Representations, ICLR 2017, Toulon, France, 13 April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:// openreview.net...

    2017

  10. [18]

    Augmented neural odes

    Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors,Advances in Neural Information Processing Systems 32: Annual Conference on Neural Inf...

    2019

  11. [19]

    Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching.ArXiv preprint, abs/2407.15595, 2024. URL https://arxiv.org/abs/2407.15595

    2024 arXiv

  12. [20]

    MADE: masked autoen- coder for distribution estimation

    Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. MADE: masked autoen- coder for distribution estimation. In Francis R. Bach and David M. Blei, editors,Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015...

    2015

  13. [21]

    Better & faster large language models via multi-token prediction.ArXiv preprint, abs/2404.19737, 2024

    Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Syn- naeve. Better & faster large language models via multi-token prediction.ArXiv preprint, abs/2404.19737, 2024. URLhttps://arxiv.org/abs/2404.19737

    2024 arXiv

  14. [22]

    OpenWebText Corpus

    Aaron Gokaslan and Vanya Cohen. OpenWebText Corpus. https://Skylion007.github. io/OpenWebTextCorpus/, 2019. Accessed: 2024-05-14. An open-source recreation of the WebText dataset

    2019

  15. [23]

    Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron C

    Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors,Advances...

    2014

  16. [24]

    Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. FFJORD: free-form continuous dynamics for scalable reversible generative models. In7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019....

    2019

  17. [25]

    Bayesian flow networks.ArXiv preprint, abs/2308.07037, 2023

    Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez. Bayesian flow networks.ArXiv preprint, abs/2308.07037, 2023. URL https://arxiv.org/abs/2308. 07037

    2023 arXiv

  18. [26]

    Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. Non- autoregressive neural machine translation. In6th International Conference on Learning Rep- resentations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. ...

    2018

  19. [27]

    Hashimoto

    Ishaan Gulrajani and Tatsunori B. Hashimoto. Likelihood-based diffusion language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors,Advances in Neural Information Processing Systems 36: Annual Conference on Neural Inform...

    2023

  20. [28]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV , USA, June 27-30, 2016, pages 770–778. IEEE Computer Society, 2016. doi: 10.1109/CV...

    2016 doi

  21. [29]

    Flow++: Improving flow-based generative models with variational dequantization and architecture design

    Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow-based generative models with variational dequantization and architecture design. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors,Proceedings of the 36th International Conference...

    2019

  22. [30]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors,Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural I...

    2020

  23. [31]

    Argmax flows and multinomial diffusion: Learning categorical distributions

    Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. In Marc’Aurelio Ran- zato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors,Advance...

    2021

  24. [32]

    Courville

    Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron C. Courville. Neural au- toregressive flows. In Jennifer G. Dy and Andreas Krause, editors,Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-...

    2018

  25. [33]

    A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines.Communications in Statistics-Simulation and Computation, 18(3):1059– 1076, 1989

    Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines.Communications in Statistics-Simulation and Computation, 18(3):1059– 1076, 1989

    1989

  26. [34]

    Scaling up gans for text-to-image synthesis

    Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17- 24, 2023, pages 10124...

    2023

  27. [35]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. InIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 4401–4410. Computer Vision Foundation / IEE...

    2019

  28. [36]

    Maximum likelihood training of implicit nonlinear diffusion model

    Dongjun Kim, Byeonghu Na, Se Jung Kwon, Dongsoo Lee, Wanmo Kang, and Il-Chul Moon. Maximum likelihood training of implicit nonlinear diffusion model. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors,Ad- vances in Neural Information Process...

    2022

  29. [37]

    Kingma and Prafulla Dhariwal

    Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 con- volutions. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors,Advances in Neural Information Processing Systems 31: Annual ...

    2018

  30. [38]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors,2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http://arxiv.org/abs...

    2014 arXiv

  31. [39]

    Improved variational inference with inverse autoregressive flow.Advances in neural information processing systems, 29, 2016

    Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow.Advances in neural information processing systems, 29, 2016

    2016

  32. [40]

    Hashimoto

    Xiang Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion-lm improves controllable text generation. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors,Advances in Neural In- formation Processing Systems 3...

    2022

  33. [41]

    Categorical normalizing flows via continuous transforma- tions

    Phillip Lippe and Efstratios Gavves. Categorical normalizing flows via continuous transforma- tions. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id= -GLNZeVDuik

    2021

  34. [42]

    Deepseek-v3 technical report.ArXiv preprint, abs/2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.ArXiv preprint, abs/2412.19437, 2024. URLhttps://arxiv.org/abs/2412.19437

    2024 arXiv

  35. [43]

    Theodorou

    Guan-Horng Liu, Tianrong Chen, and Evangelos A. Theodorou. Second-order neural ODE optimizer. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors,Advances in Neural Information Processing Systems 34: Annual Conference...

    2021

  36. [44]

    Think while you generate: Discrete diffusion with planned denoising

    Sulin Liu, Juno Nam, Andrew Campbell, Hannes Stärk, Yilun Xu, Tommi Jaakkola, and Rafael Gómez-Bombarelli. Think while you generate: Discrete diffusion with planned denoising. ArXiv preprint, abs/2410.06264, 2024. URLhttps://arxiv.org/abs/2410.06264

    2024 arXiv

  37. [45]

    Discrete diffusion modeling by estimat- ing the ratios of the data distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimat- ing the ratios of the data distribution. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https://openreview.n...

    2024

  38. [46]

    Maximum likelihood training for score-based diffusion odes by high order denoising score matching

    Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for score-based diffusion odes by high order denoising score matching. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors,...

    2022

  39. [47]

    text8 Corpus

    Matt Mahoney. text8 Corpus. http://mattmahoney.net/dc/textdata.html, 2011. Ac- cessed: 2024-05-14. Original site for the text8 dataset, part of the Large Text Compression Benchmark

    2011

  40. [48]

    Concrete score match- ing: Generalized score matching for discrete data

    Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score match- ing: Generalized score matching for discrete data. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors,Advances in Neural In- formation Processing Systems 35: A...

    2022

  41. [49]

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data

    Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. ArXiv preprint, abs/2406.03736, 2024. URLhttps://arxiv.org/abs/2406.03736

    2024 arXiv

  42. [50]

    Masked autoregressive flow for density estimation

    George Papamakarios, Iain Murray, and Theo Pavlakou. Masked autoregressive flow for density estimation. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processi...

    2017

  43. [51]

    Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan

    George Papamakarios, Eric T. Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference.J. Mach. Learn. Res., 22:57:1–57:64, 2021. URLhttp://jmlr.org/papers/v22/19-1028.html

    2021

  44. [52]

    Transformer neural autoregressive flows.ArXiv preprint, abs/2401.01855, 2024

    Massimiliano Patacchiola, Aliaksandra Shysheya, Katja Hofmann, and Richard E Turner. Transformer neural autoregressive flows.ArXiv preprint, abs/2401.01855, 2024. URL https: //arxiv.org/abs/2401.01855. 16

    2024 arXiv

  45. [53]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  46. [54]

    Routledge, 2018

    Lev Semenovich Pontryagin.Mathematical theory of optimal processes. Routledge, 2018

    2018

  47. [55]

    Remarks on a multivariate transformation.The annals of mathematical statistics, 23(3):470–472, 1952

    Murray Rosenblatt. Remarks on a multivariate transformation.The annals of mathematical statistics, 23(3):470–472, 1952

    1952

  48. [56]

    Chiu, Alexander M

    Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander M. Rush, and V olodymyr Kuleshov. Simple and effective masked diffusion language models.ArXiv preprint, abs/2406.07524, 2024. URL https://arxiv.org/ abs/2406.07524

    2024 arXiv

  49. [57]

    Step-unrolled denoising autoencoders for text generation

    Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aäron van den Oord. Step-unrolled denoising autoencoders for text generation. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net,

    2022

  50. [58]

    Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and generalized masked diffusion for discrete data.ArXiv preprint, abs/2406.04329, 2024. URL https://arxiv.org/abs/2406.04329

    2024 arXiv

  51. [59]

    Training and inference on any-order autore- gressive models the right way

    Andy Shih, Dorsa Sadigh, and Stefano Ermon. Training and inference on any-order autore- gressive models the right way. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Information Processing Systems 35: Annual Conference...

    2022

  52. [60]

    Weiss, Niru Maheswaranathan, and Surya Ganguli

    Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep un- supervised learning using nonequilibrium thermodynamics. In Francis R. Bach and David M. Blei, editors,Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille...

    2015

  53. [61]

    Maximum likelihood training of score-based diffusion models

    Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors,Advances in Neural Information Process- ing Sys...

    2021

  54. [62]

    Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole

    Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7...

    2021

  55. [63]

    f-vaes: Improve vaes with conditional flows.ArXiv preprint, abs/1809.05861, 2018

    Jianlin Su and Guang Wu. f-vaes: Improve vaes with conditional flows.ArXiv preprint, abs/1809.05861, 2018. URLhttps://arxiv.org/abs/1809.05861

    2018 arXiv

  56. [64]

    Score-based continuous- time discrete diffusion models

    Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous- time discrete diffusion models. InThe Eleventh International Conference on Learning Rep- resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview....

    2023

  57. [65]

    Neural discrete representation learning

    Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Garnett, editors,Advances in Neural Information Processing Sys...

    2017

  58. [66]

    Attention is all you need.(nips), 2017.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need.(nips), 2017.Advances in neural information processing systems, 30, 2017. 17

    2017

  59. [67]

    Digress: Discrete denoising diffusion for graph generation

    Clément Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, V olkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. InThe Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe...

    2023

  60. [68]

    Stabilizing generative adversarial networks: A survey.ArXiv preprint, abs/1910.00927, 2019

    Maciej Wiatrak, Stefano V Albrecht, and Andrew Nystrom. Stabilizing generative adversarial networks: A survey.ArXiv preprint, abs/1910.00927, 2019. URL https://arxiv.org/abs/ 1910.00927

    1910 arXiv

  61. [69]

    Energy-based diffusion language models for text generation.ArXiv preprint, abs/2410.21357, 2024

    Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovec, Stefano Ermon, and Arash Vahdat. Energy-based diffusion language models for text generation.ArXiv preprint, abs/2410.21357, 2024. URLhttps://arxiv.org/abs/2410.21357

    2024 arXiv

  62. [70]

    Seqgan: Sequence generative adversarial nets with policy gradient

    Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Satinder P. Singh and Shaul Markovitch, editors,Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, ...

    2017

  63. [71]

    Normalizing flows are capable generative models.ArXiv preprint, abs/2412.06329, 2024

    Shuangfei Zhai, Ruixiang Zhang, Preetum Nakkiran, David Berthelot, Jiatao Gu, Huangjie Zheng, Tianrong Chen, Miguel Angel Bautista, Navdeep Jaitly, and Josh Susskind. Normalizing flows are capable generative models.ArXiv preprint, abs/2412.06329, 2024. URL https: //arxiv.org/a...

    2024 arXiv

  64. [72]

    Learning structured latent factors from dependent data:a generative model framework from information-theoretic perspective

    Ruixiang Zhang, Masanori Koyama, and Katsuhiko Ishiguro. Learning structured latent factors from dependent data:a generative model framework from information-theoretic perspective. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2...

    2020

  65. [73]

    Susskind

    Ruixiang Zhang, Shuangfei Zhai, Etai Littwin, and Joshua M. Susskind. Learning representation from neural fisher kernel with low-rank approximation. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net,

    2022

  66. [74]

    Robust and controllable object-centric learning through energy-based models

    Ruixiang Zhang, Tong Che, Boris Ivanovic, Renhao Wang, Marco Pavone, Yoshua Bengio, and Liam Paull. Robust and controllable object-centric learning through energy-based models. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May ...

    2023

  67. [75]

    Target concrete score matching: A holistic framework for discrete diffusion

    Ruixiang Zhang, Shuangfei Zhai, Yizhe Zhang, James Thornton, Zijing Ou, Joshua Susskind, and Navdeep Jaitly. Target concrete score matching: A holistic framework for discrete diffusion. ArXiv preprint, abs/2504.16431, 2025. URLhttps://arxiv.org/abs/2504.16431

    2025 arXiv

  68. [76]

    URLhttps://openreview.net/forum?id=J1rhANsCY9

  69. [77]

    A reparameterized discrete diffusion model for text generation.ArXiv preprint, abs/2302.05737, 2023

    Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation.ArXiv preprint, abs/2302.05737, 2023. URL https://arxiv.org/ abs/2302.05737

    2023 arXiv

  70. [78]

    Open-sora: Democratizing efficient video production for all,

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all,

  71. [79]

    Perceptual generative autoencoders

    Zijun Zhang, Ruixiang Zhang, Zongpeng Li, Yoshua Bengio, and Liam Paull. Perceptual generative autoencoders. InProceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 ofProceedings of Machine Learning Research...

    2020

  72. [80]

    Ziegler and Alexander M

    Zachary M. Ziegler and Alexander M. Rush. Latent normalizing flows for discrete sequences. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors,Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, 18 v...

    2019

  73. [83]

    Dvornek, Sekhar Tatikonda, and James S

    Juntang Zhuang, Nicha C. Dvornek, Sekhar Tatikonda, and James S. Duncan. MALI: A memory efficient and reverse accurate integrator for neural odes. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net,

    2021

  74. [86]

    Differentiating ycdf = Φ(u t,i) with respect to ut,i yields dycdf dut,i = N(u t,i; 0,1), where N(u t,i; 0,1) is the PDF of the standard normal distribution

    The derivative of the inverse standard normal CDF, dΦ−1(ycdf) dycdf : If ut,i = Φ −1(ycdf), then ycdf = Φ(u t,i). Differentiating ycdf = Φ(u t,i) with respect to ut,i yields dycdf dut,i = N(u t,i; 0,1), where N(u t,i; 0,1) is the PDF of the standard normal distribution. Theref...

  75. [87]

    The derivative of the mixture CDF with respect toz t,i, ∂Fmix-1(zt,i;z<t,zt,<i) ∂zt,i : ∂Fmix-1(zt,i;z <t,z t,<i) ∂zt,i = ∂ ∂zt,i VX k=1 π[k]Φ zt,i −m[k] σ[k] = VX k=1 π[k]N zt,i −m[k] σ[k] ; 0,1 · 1 σ[k] Recognizing that 1 σ[k] N zt,i−m[k] σ[k] ; 0,1 is the PDF N(z t,i;m[k],σ...

  76. [88]

    Sample a component indexK=kwith probabilityP(K=k) =π k

  77. [89]

    Samplezfrom the chosen component:p(z|K=k) =N(z;m k, σ2 kId). The target conditional density can be found by marginalizing over this latent variable using the law of total probability: p(zi|z<i) = VX k=1 p(zi|z<i, K=k)P(K=k|z <i).(24) We now analyze the two terms in the summati...

  78. [90]

    For a given component k, z follows a single multivariate Gaussian distribution N(m k, σ2 kId)

    The Component Density: p(zi|z<i, K=k) .This term is the conditional density of zi given z<i, under the condition that we know the sample originates from component k. For a given component k, z follows a single multivariate Gaussian distribution N(m k, σ2 kId). The covariance m...

  79. [91]

    We denote this by α(i) k (z<i) and compute it using Bayes’ rule: P(K=k|z <i) = p(z<i|K=k)P(K=k) p(z<i) = p(z<i|K=k)P(K=k)PV j=1 p(z<i|K=j)P(K=j)

    The Mixture Weights: P(K=k|z <i).This term represents the posterior probability of being in component k after observing the first i−1 dimensions. We denote this by α(i) k (z<i) and compute it using Bayes’ rule: P(K=k|z <i) = p(z<i|K=k)P(K=k) p(z<i) = p(z<i|K=k)P(K=k)PV j=1 p(z...

  80. [92]

    This corresponds to minimizingKL(q in∥pmodel)

    Model its input distribution: The layer’s parameters (defining pmodel(x|cond)) are adjusted so that pmodel(x|cond) accurately represents the distribution qin(x) of the data it receives. This corresponds to minimizingKL(q in∥pmodel)

  81. [93]

    Flexible Patch Size

    Transform its input to a target distribution: As a consequence of (1) and the construc- tion of f, the layer transforms its input x into an output y whose distribution qout(y) closely matches the predefined target distribution ptarget(y). This corresponds to minimizing KL(qout...

  82. [94]

    For a given true token xt, the sample zt ∼ Nxt (zt) will converge in probability to its mean: zt →µ xt

  83. [95]

    ,zt−1)will converge to(µ x1 ,

    Consequently, the historyz <t = (z1, . . . ,zt−1)will converge to(µ x1 , . . . ,µxt−1 )

  84. [96]

    equation 71) evaluated at zt ≈µ xt will behave as follows: If µk are distinct, then for zt ≈µ xt, Nxt (zt) will be large, while Nj(zt) for j̸=x t will be very small

    The decoder probability p(xt =k|z t) (Eq. equation 71) evaluated at zt ≈µ xt will behave as follows: If µk are distinct, then for zt ≈µ xt, Nxt (zt) will be large, while Nj(zt) for j̸=x t will be very small. Thus,p(x t =k|z t ≈µ xt )→ 1ifk=x t 0ifk̸=x t . Substituting these li...

  85. [97]

    RealNVP [17] built upon this by introducing scaling and shifting operations, thereby increasing model flexibility

    pioneered the use of additive coupling layers, which made the Jacobian determinant computation straightforward. RealNVP [17] built upon this by introducing scaling and shifting operations, thereby increasing model flexibility. Glow [37] further improved these models by incorpo...

  86. [98]

    Other prominent generative models include Variational Autoencoders (V AEs) [38] and Generative Adversarial Networks (GANs) [23]

    have demonstrated the ability of diffusion models to generate extremely high-dimensional data. Other prominent generative models include Variational Autoencoders (V AEs) [38] and Generative Adversarial Networks (GANs) [23]. VQ-V AE [65] addresses posterior collapse and achieve...

  87. [99]

    analog bits,

    applied continuous diffusion to word embeddings, allowing for controllable text generation via gradient-based guidance.Plaid[ 27] focused on likelihood-based text modeling, jointly optimizing embeddings and model parameters using the VLB, categorical reparameterization, an out...

  88. [2015]

    URLhttp://proceedings.mlr.press/v37/germain15.html

  89. [2021]

    URLhttps://openreview.net/forum?id=blfSjHeFM_e

  90. [2022]

    URLhttps://openreview.net/forum?id=T0GpzBQ1Fg6

  91. [2024]

    URLhttps://github.com/hpcaitech/Open-Sora

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.