Pith. sign in

REVIEW 3 major objections 4 minor 63 references

On the Separability of Information in Diffusion Models

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Diffusion models store most of their information in fine texture, not in what identifies the image.

desk verdict A genuinely interesting toy-model story about where information lives in diffusion models, but the headline image-domain I(X;Y) numbers violate the label entropy bound, so the quantitative claim needs a working estimator before it can be trusted. read the letter →

arxiv 2509.23937 v5 pith:6NSW3RAY submitted 2025-09-28 cs.LG cond-mat.stat-mechcs.AIcs.ITmath.IT

Akhil Premkumar This is my paper
classification cs.LGcond-mat.stat-mechcs.AIcs.ITmath.IT
keywords diffusionmodelsneuralentropymutualinformationtotalcorrelationclassifier-freeguidanceperceptualversussemanticbottleneckscore-basedgenerative
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the information stored in a diffusion model's neural network is almost entirely spent on reproducing small-scale perceptual detail, while the information that connects images to their class labels is tiny by comparison. Concretely, the paper reports a neural entropy of about 2,018 nats versus a label mutual information of about 4 nats on a handwritten-digit benchmark, and about 7,776 versus 7.5 nats on a small-image benchmark. The explanation is structural: neural entropy decomposes into a huge total-correlation term that reconstructs pixel correlations, and a small I(X;Y) term that binds images to labels. The paper claims these two amounts of information are sourced from different stages of the reverse diffusion process, which is why classifier-free guidance works: the guidance vector lives in the stage where scores are well-behaved and amplifies the small semantic signal. If the paper is right, the difficulty of conditional image generation is in the information structure of the data itself, not in a model deficiency.

What carries the argument

The load-bearing objects are the neural entropy S_NN (the squared norm of the learned drift, integrated over the diffusion schedule), the total-correlation term TC(X) that appears when the data distribution is compared with the product of its marginals, and the mutual-information integral I(X;Y) written as the expected squared difference of conditional and unconditional scores. The identity that the extra information a conditional model stores over an unconditional one equals I(X;Y) turns a training-time entropy difference into the side information a conditional model must store. The probe is a split-band diffusion autoencoder: it partitions the denoising loss so that a 'perceptual' latent s

What would settle it

Take a synthetic image distribution whose class label is defined entirely by a fine texture pattern, with identical large-scale structure across classes. Train the split-band autoencoder with small τ and measure I(Z_per;Y): the paper's claim predicts it stays near zero; a clearly nonzero value would falsify the claim that early perceptual detail carries no class information. A second check is to artificially shuffle Z_per across classes: if reconstructions remain class-consistent, the separation holds; if they become class-inconsistent, leakage is present.

Watch

Extended reading notes

Core claim

The paper's central claim is that the information a diffusion model stores—its neural entropy S_NN—is dominated by total correlation, TC(X), the joint statistical dependence among pixels that must be reinstated to reproduce fine textures. In pixel-space image models this term is vast because natural images lie on a low-dimensional manifold, so locating the manifold precisely consumes most of the information budget. The mutual information I(X;Y) that binds images to class labels is orders of magnitude smaller, and it enters through a different, later stage of the denoising process than the early manifold-pinning stage. Because the classifier-free guidance vector is the difference of condition

Load-bearing premise

The load-bearing assumption is that the split-band diffusion autoencoder cleanly separates the model's information into perceptual and semantic channels with no leakage, and that the mutual-information estimator remains accurate at the very low values where class information lives.

Editorial extensions

If this is right

  • If the claim is right, compressing or pruning a pixel-space diffusion model will first erode texture and high-frequency detail; semantic class structure will survive until much more information is removed.
  • Score-difference estimates of I(X;Y) remain reliable even when the neural entropy is enormous, because the manifold-pinning parts of the conditional and unconditional scores cancel.
  • Classifier-free guidance raises I(X;Y) but only up to a plateau set by the conditioning variable's own capacity—the information-bottleneck saturation visible in the paper's Gaussian experiments.
  • Weak label-image binding is a property of the data, so guidance cannot manufacture class information that was never present; it amplifies a small signal along with noise.
  • The temporal separation predicts that conditioning information matters most in the later semantic stage of generation, not in the early texture-resolving stage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to condition only the semantic sub-interval of the denoising schedule and leave the perceptual sub-interval unconditional; the paper's account predicts little or no loss in class fidelity and a possible saving in compute.
  • If textures are largely class-agnostic, then swapping perceptual latents between images of different classes while keeping semantic latents fixed should preserve class-conditional appearance; a failure would indicate leakage between the split bands.
  • The same score-difference estimator used for CFG could be turned into a calibration tool: the guidance strength at which I(X;Y) saturates would estimate the effective information capacity of the conditioning channel.
  • One could read the paper's temporal decomposition as a principled way to design hierarchical or multi-stage generative models, where early stages model manifold geometry and later stages model semantics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces a Kelly-criterion interpretation of diffusion models, in which the information a diffusion model must inject during generation is measured by a 'neural entropy' S_NN, and the extra information needed for conditional generation equals the mutual information I(X;Y). The authors derive an estimator for I(X;Y) based on the difference between conditional and unconditional score networks (MINDE, Eq. (14)), validate it on a joint Gaussian model with known ground truth, and then apply it to MNIST and CIFAR-10. They report that in image models I(X;Y) is only about 10^-3 of S_NN, that the bulk of S_NN is a total-correlation term TC(X) tied to low-level perceptual detail, and that the small class-specific information is sourced from a later, 'semantic' diffusion stage. This is used to explain why classifier-free guidance works. The Gaussian toy model and the derivation of MINDE are the sound core of the paper; the image-domain quantitative claims are the point of failure.

Significance. If correct, the paper would provide a useful information-theoretic decomposition of what diffusion models learn, and a novel explanation for the effectiveness of classifier-free guidance in terms of where class information lives in the diffusion process. The Kelly-gambling framing is pedagogically attractive and connects several existing results (Premkumar 2025; Franzese et al. 2024). The Gaussian experiments are a genuine strength: they compare neural estimates against analytic ground truth and honestly display the estimator's failure regime. The central image-domain claim, however, is currently supported by an estimator that produces information-theoretically impossible values, so the significance of the paper depends on whether that claim can be repaired.

major comments (3)
  1. [Sec. 6, Fig. 14, Eq. (14)] The reported image-domain mutual information values violate a hard information-theoretic bound. For MNIST and CIFAR-10, Y is a 10-class label, so I(X;Y) ≤ H(Y) ≤ log(10) ≈ 2.30 nats. Fig. 14 reports I(X;Y)=4.1 nats for MNIST and 7.54 nats for CIFAR-10. Eq. (13) is exact and cannot violate this bound, so the error lies in the neural-network approximation Eq. (14) as applied in the image regime. Since the 10^-3 ratio S_NN/I(X;Y) and the CFG explanation in Secs. 4 and 6 are built on these numbers, the central quantitative claim is currently unsupported. The authors should either replace Eq. (14) by an estimator whose calibration is demonstrated in a setting where the true I is known (including a discrete-label setting), or report upper bounds/qualitative statements only.
  2. [Sec. F.3, Eqs. (19) and (74)] The split-band DAE conclusion that z_per contains no class-specific information assumes a clean separation that the objective does not enforce. Equation (19) trains a single shared denoising network e_θ; z_per is used only for s∈(0,τ) in the loss, but nothing prevents z_per from encoding global/semantic structure that is useful for small-s denoising, nor prevents z_sem from indirectly influencing early denoising through the shared network. The statement that 'Z_per encodes only the information that was absorbed from s∈(0,τ)' is therefore an assumption, not a consequence of the loss. Moreover, I(Z_per;Y) and I(Z_sem;Y) are estimated with the same MINDE estimator, so Fig. 16 inherits the validity problem of Major Comment 1. The claim that textures carry no class information needs a probe whose information separation is independently verified.
  3. [Sec. 3, Eq. (12)] The identification of S_NN with 'information stored in the network' is definitional rather than empirical. S_NN is defined as an integral of ||e_θ||^2, and S_NN ≈ S_tot holds only for an ideal entropy-matching network. The Gaussian experiments in Figs. 9-11 show that for real networks S_NN can deviate significantly from S_tot in exactly the regimes relevant to images (low-rank/flattened data and strongly correlated Y). In the image experiments there is no ground truth, so the absolute values S_NN=2017.9 and 7776.2 nats are not established as information content. Since Fig. 14 compares these absolute values with I(X;Y), the ratio S_NN/I(X;Y) is not yet a measured quantity. A calibrated or relative interpretation is needed.
minor comments (4)
  1. [Eq. (1)] The limit in the definition of the doubling rate R is written as n→0; it should be n→∞.
  2. [Sec. 4, after Eq. (14)] The text says the approximation holds 'up to terms that vanish as T→0'; from the derivation in Sec. D.1 and the assumption p_0≈p_eq, the correct statement is for large T (T→∞).
  3. [Sec. F.3, Fig. 16 caption/text] The sentence 'whereas Z_sem does the opposite' appears to refer to Z_per; the caption and the surrounding text should be reconciled.
  4. [Sec. 6] The claim that 'the CFG vector is strongest in the interval where the scores are well-behaved' is stated as a consequence, but the paper does not directly measure the CFG vector's strength or the score behavior in image models. Adding a direct image experiment, or clearly labeling this as a conjecture, would strengthen the paper.

Circularity Check

1 steps flagged · score 2.0 of 10

No material circularity; one minor self-citation for the neural-entropy measure. The derivation chain is self-contained, but the central image-domain MI estimate is invalidated by exceeding the discrete-label entropy bound (a correctness risk, not a circular step).

  1. self citation load bearing [Sec. 3 (Neural Entropy), Eq. (12) and following paragraph]
    "However, Premkumar (2025) demonstrates that the amount of information stored in a real network eθ is measured through its neural entropy, S X NN := Z T 0 ds σ(s)2 2 Ep h ∥eθ( ˜xs, s)∥2 i ≈ S X tot."

    The paper's quantitative framework depends on identifying the L2 norm of the network output with 'information stored in a real network,' and this identification is imported from the author's own prior work. The internal ideal-limit argument (eθ→e*θ ⇒ S_NN→S_tot) and Gaussian checks give partial support, but for image models the paper explicitly says it cannot gauge whether S_NN and I(X;Y) are close to target values. Thus a load-bearing premise of the central claim rests on a self-citation plus an unvalidated limit, rather than on an independent check. This is not a formal reduction of a prediction to a fitted parameter, so it is scored as minor.

full rationale

The derivation chain S_tot (Eq. 4) -> factorized Stot with TC(X) (Eq. 8) -> conditional entropy difference R = S(X)-S(X|Y) = I(X;Y) (Eq. 11) -> MINDE (Eq. 13) -> neural-score estimator (Eq. 14) is mathematically self-contained: Eq. (13) is re-derived in Sec. D.1 from the path-integral identity and Eq. (14) is the standard substitution of fitted scores for exact scores. No fitted parameter is renamed as a prediction, and the CFG explanation is not tautological because the paper explicitly rejects the naive substitution of the modified score into Eq. (13) and instead measures I under CFG on Gaussian data. The split-band DAE (Eqs. 19/74) is an experimental probe whose latents are not fit to Y; the finding that I(Z_per;Y)≈0 is an empirical result, not a construction. The only circular-adjacent element is the self-citation used to define neural entropy, which is why the score is 2 rather than 0. Separately, the paper's reported image-domain values (MNIST 4.1 nats, CIFAR-10 7.54 nats) violate the hard bound I(X;Y) <= H(Y) <= log(10) ≈ 2.30 nats for 10-class labels; the paper itself admits it has no true image scores to validate the estimator. That is a serious correctness risk for the central quantitative claim, but it is not a circularity and does not raise the circularity score.

Assumptions & free parameters 3 free parameters · 6 assumptions · 3 invented entities

The ledger shows the central claim rests on three load-bearing imports: the entropy-matching identification of S_NN with stored information (the author's own prior work), the accuracy of the score-difference MI estimator in the image regime (an external theorem, unverified here), and the split-band DAE probing assumption (constructed in this paper). Free knobs include the band split tau, the objective weighting lambda(s), and the VP schedule, all of which change the reported information-budget numbers. No new explanatory entities such as particles or forces are invented; the new objects are measurement constructs.

free parameters (3)
  • tau (DAE band split) = scanned 0.1T to 0.9T
    Defines which information is 'perceptual' (s in (0,tau)) versus 'semantic' (s in (tau,T)); the finding I(Z_per;Y) near 0 holds only for small tau, and no principled criterion for tau is given.
  • Objective weight lambda(s) = lambda=1 for training; lambda=sigma^2/2 for evaluation
    The network is trained with variance dropping but S_NN and I are evaluated with sigma^2/2 weights (Eqs. 12-13); the S_NN = S_tot correspondence presumes the sigma^2/2-weighted optimum, so the weighting choice affects the reported numbers.
  • VP noise schedule beta(s) = not stated in text
    S_tot and S_NN are KL quantities defined relative to the chosen forward process; the absolute nats (2017.9, 7776.2) and hence the 'large fraction' claim change with the schedule.
assumptions (6)
  • ad hoc to paper Neural entropy S_NN = S_tot and its interpretation as information stored in the network (Eq. 12)
    Imported from the author's prior preprint (Premkumar 2025); the identification of a norm-squared integral with stored information is the paper's own framework, not an independently established theorem.
  • domain assumption The MINDE score-difference estimator (Eq. 14) is accurate for images
    Franzese et al. (2024) prove the identity for exact scores; the paper's Figs. 10-11 show the learned estimate degrades in degenerate regimes, and the reported MNIST/CIFAR values exceed H(Y) <= log(10), so image-regime accuracy is assumed, not shown.
  • ad hoc to paper Split-band DAE conditioning cleanly separates model information into perceptual (0,tau) and semantic (tau,T) channels (Eqs. 19 and 74)
    The engineered loss split presumes no leakage through the shared decoder and that I(Z_per;Y) faithfully measures class information in early diffusion steps; the probe is validated only by its own outputs.
  • domain assumption Image densities are supported on a lower-dimensional manifold, so TC(X) and S_tot diverge, with the divergence concentrated near s=0
    Sec. 5 and Fig. 14; the claim that texture dominates the budget relies on this singular-structure assumption about natural images, which also implies S_NN cannot equal the divergent S_tot.
  • standard math Large-T limit: p0 = p_eq(T), boundary terms and S0 shifts vanish in Eqs. (7), (10), (49)
    Standard in diffusion theory; finite-T corrections are not quantified for the image experiments.
  • standard math Textbook identities: KL factorization (Eq. 8), MI bounds for discrete Y, Tweedie/Miyasawa relation, pointwise convex minimization
    Used in Secs. 3, B, and D; these are established results, including I(X;Y) <= H(Y) <= log(10) for 10-class labels.
invented entities (3)
  • Neural entropy S_NN as 'information stored in the network' independent evidence
    purpose: Quantify the network's information budget for comparison with I(X;Y) and TC(X)
    Operationally defined (Eq. 12) and matched against analytic S_tot in the Gaussian toy, so others can falsify the numbers; the 'stored information' reading is an interpretation, not a proved fact.
  • Perceptual latent z_per (split-band DAE probe)
    purpose: Measure class-specific information in the early-s (fine-detail) diffusion band
    Its key property (no class structure) is established only by in-paper t-SNE and I(Z_per;Y) estimates without error bars and with an unvalidated estimator; it is a probe construct, not an explanatory entity.
  • Semantic latent z_sem (split-band DAE probe) independent evidence
    purpose: Measure class-relevant information in the late-s band
    Semantic latents in DAEs have prior support (Preechakul et al. 2022) and the class clusters reproduced here match that prior work, giving it partial independent grounding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Separability of Information in Diffusion Models." pith.science (2026). https://pith.science/paper/6NSW3RAY

@misc{pith2026250923937,
  author       = {Pith},
  title        = {Pith review of: On the Separability of Information in Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6NSW3RAY}},
  note         = {Machine review of arXiv:2509.23937}
}
read the original abstract

Diffusion models transform noise into data by injecting information that was captured in their neural network during the training phase. In this paper, we ask: \textit{what} is this information? We find that, in pixel-space diffusion models, (1) a large fraction of the total information in the neural network is committed to reconstructing small-scale perceptual details of the image, and (2) the correlations between images and their class labels are informed by the semantic content of the images, and are largely agnostic to the low-level details. We argue that these properties are intrinsically tied to the manifold structure of the data itself. Finally, we show that these facts explain the efficacy of classifier-free guidance: the guidance vector amplifies the mutual information between images and conditioning signals early in the generative process, influencing semantic structure, but tapers out as perceptual details are filled in.

Figures

Figures reproduced from arXiv: 2509.23937 by the authors.

Figure 2
Figure 2. Left: Samples generated by a CFG-style modification to the conditional score ∇ log p(xt, t|y) of a joint Gaussian, Y = AX + ε (cf. Eqs. (16) and (53)). CFG strengthens the correlation between X and Y , increasing their mutual information. But it also alters the re￾lationship between them. Right: Mutual information under CFG for the joint Gaussian. We fix DX = 25 and repeat the experiment with DY = 5, 10, 15. Notice … view at source ↗
Figure 3
Figure 3. Two ways of having a large S X|Y tot . (Left) Flattening: I(X;Y ) stays finite while SX tot blows up due to lower intrinsic dimensionality of X. (Right) Determinism: I(X;Y ) diverges when Y is strongly correlated with X, but SX tot is under control since its covariance is full-rank. The entropy rate curves are the time derivatives of the corresponding entropy or mutual information. Similar plots for different degree… view at source ↗
Figure 4
Figure 4. Probing the information stored in a diffusion model using a diffusion autoencoder. (Left [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figures from the paper (13 more)
Figure 5
Figure 5. Figure 5: The linear Gaussian model from Eq. (22) with (a) higher noise/larger [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 7
Figure 7. Figure 7: Samples generated by a CFG-style modification to the conditional score [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: Mutual information under CFG for the joint Gaussian model from Eq. (53). We fix [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: Flattening: Evolution of total entropy, neural entropy, and the mutual information under the forward process, for a joint Gaussian with DX = 25, DY = 15, as the effective rank of ΣX is made smaller. We keep DX = 25, DY = 15, σε = 1.0 but λδ is lowered from top to botto…
Figure 10
Figure 10. Figure 10: Determinism: Entropy curves for a joint Gaussian with DX = 25, DY = 15 and a full-rank ΣX. As σε is lowered X and Y become more correlated, which causes I(X;Y ) to grow while S X tot remain fixed. Notice also how the S X|Y tot and I(X;Y ) curves become more concentrat…
Figure 11
Figure 11. Figure 11: Mutual information and neural entropy for the joint Gaussian model as [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]
Figure 12
Figure 12. Figure 12: Images reconstructed by a DAE and VAE. Both of them have the same encoder architec [PITH_FULL_IMAGE:figures/full_fig_p030_12.png]
Figure 13
Figure 13. Figure 13: Latents from a VAE and DAE trained to reconstruct MNIST digits. Distinct clusters [PITH_FULL_IMAGE:figures/full_fig_p030_13.png]
Figure 14
Figure 14. Figure 14: Neural entropy profiles for two image diffusion models trained on the MNIST and [PITH_FULL_IMAGE:figures/full_fig_p032_14.png]
Figure 16
Figure 16. Figure 16: Mutual information between the image labels [PITH_FULL_IMAGE:figures/full_fig_p034_16.png]
Figure 17
Figure 17. Figure 17: A schematic of the forward and reverse diffusion processes. [PITH_FULL_IMAGE:figures/full_fig_p035_17.png]
Figure 18
Figure 18. Figure 18: A breakdown of how − log pd(x) is computed in in Eq. (45). The denoised means xˆ(x˜s) from many x˜s give a sense of the regions in pd that are most like x. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_18.png]
Figure 19
Figure 19. Figure 19: Iterative resolution of a sample from pd. The blue points are forward diffused version of the black one. The red curve shows the density of denoised means computed from the blue points using the Miyasawa relation, Eq. (40). As we reverse diffuse in the t-direction, th…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 2 canonical work pages

  1. [1]

    Alemi, Ian Fischer, Joshua V

    Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. CoRR, abs/1612.00410, 2016. URL http://arxiv.org/abs/1612.00410

    arXiv 2016

  2. [2]

    Christopher M. Bishop. Pattern Recognition and Machine Learning. Information Science and Statistics. Springer, New York, 2006. ISBN 978-0387310732

    2006

  3. [3]

    JAX : composable transformations of P ython+ N um P y programs, 2018

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake Vander P las, Skye Wanderman- M ilne, and Qiao Zhang. JAX : composable transformations of P ython+ N um P y programs, 2018. URL http://github.com/jax-ml/jax

    2018

  4. [4]

    Classifier-free guidance is a predictor-corrector

    Arwen Bradley and Preetum Nakkiran. Classifier-free guidance is a predictor-corrector. In NeurIPS Workshop on Score-Based Methods, 2024. URL https://arxiv.org/abs/2408.09000

    arXiv 2024

  5. [5]

    Bradley C. A. Brown, Anthony L. Caterini, Brendan Leigh Ross, Jesse C. Cresswell, and Gabriel Loaiza - Ganem. Verifying the union of manifolds hypothesis for image data. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=Rvee9CAX4fi

    2023

  6. [6]

    e l Chetrite, Paolo Muratore-Ginanneschi, and Kay Schwieger. E. S chr \

    Rapha \"e l Chetrite, Paolo Muratore-Ginanneschi, and Kay Schwieger. E. S chr \"o dinger's 1931 paper `` On the Reversal of the Laws of Nature '' [`` \"U ber die Umkehrung der Naturgesetze '', Sitzungsberichte der preussischen Akademie der Wissenschaften, physikalisch-mathematische Klasse, 8 N9 144--153] . The European Physical Journal H, 46 0 (1): 0 28, ...

    doi:10.1140/epjh/s13129-021-00032-7 1931

  7. [7]

    Text-to-Image Diffusion Models are Zero-Shot Classifiers

    Kevin Clark and Priyank Jaini. Text-to-Image Diffusion Models are Zero-Shot Classifiers . In Alice Oh, Taesup Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ 58921--58937. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023...

    2023

  8. [8]

    Cover and Joy A

    Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006. ISBN 0471241954

    2006

Show all 63 references
  1. [9]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 8780--8794. Curran Associates, Inc., ...

    2021

  2. [10]

    Tweedie's formula and selection bias

    Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106 0 (496): 0 1602--1614, 2011

    2011

  3. [11]

    MINDE : Mutual information neural diffusion estimation

    Giulio Franzese, Mustapha Bounoua, and Pietro Michiardi. MINDE : Mutual information neural diffusion estimation. In Proceedings of the International Conference on Learning Representations (ICLR), pp.\ 16685--16716, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/f...

    2024

  4. [12]

    Generalized jarzynski's equality of inhomogeneous multidimensional diffusion processes

    Hao Ge and Da-Quan Jiang. Generalized jarzynski's equality of inhomogeneous multidimensional diffusion processes. Journal of Statistical Physics, 131 0 (4): 0 675--689, 5 2008. ISSN 1572-9613. doi:10.1007/s10955-008-9520-4. URL https://doi.org/10.1007/s10955-008-9520-4

    2008 doi

  5. [13]

    Burgess, Xavier Glorot, Matthew M

    Irina Higgins, Lo \" c Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. -vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Representat...

    2017

  6. [14]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CoRR, abs/2207.12598, 2022. doi:10.48550/ARXIV.2207.12598. URL https://doi.org/10.48550/arXiv.2207.12598

    2022 doi arXiv

  7. [15]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria - Florina Balcan, and Hsuan - Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural I...

    2020

  8. [16]

    Courville

    Chin - Wei Huang, Jae Hyun Lim, and Aaron C. Courville. A variational perspective on diffusion-based generative models and score matching. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Informat...

    2021

  9. [17]

    David A. Huffman. A method for the construction of minimum-redundancy codes. Proceedings of the IRE, 40 0 (9): 0 1098--1101, 1952. doi:10.1109/JRPROC.1952.273898

    1952

  10. [18]

    Simoncelli, and St \' e phane Mallat

    Zahra Kadkhodaie, Florentin Guth, Eero P. Simoncelli, and St \' e phane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representation. CoRR, abs/2310.02557, 2023. doi:10.48550/ARXIV.2310.02557. URL https://doi.org/10.48550/arXiv.2310.02557

    2023 doi arXiv

  11. [19]

    Elucidating the Design Space of Diffusion-Based Generative Models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the Design Space of Diffusion-Based Generative Models . In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Confe...

    2022

  12. [20]

    J. L. Kelly. A New Interpretation of Information Rate . The Bell System Technical Journal, 35 0 (4): 0 917--926, 1956. doi:10.1002/j.1538-7305.1956.tb03809.x

    1956

  13. [21]

    Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation

    Diederik Kingma and Ruiqi Gao. Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation . In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ 65484--65516. Curran ...

    2023

  14. [22]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014. URL http://arxiv.org/abs...

    2014 arXiv

  15. [23]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. https://www.cs.toronto.edu/ kriz/learning-features-2009-TR.pdf

    2009

  16. [24]

    Gradient-based learning applied to document recognition

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 0 (11): 0 2278--2324, 1998. doi:10.1109/5.726791

    1998 doi

  17. [25]

    Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak

    Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 2206--2217, October 2023

    2023

  18. [26]

    Critical windows: Non-asymptotic theory for feature emergence in diffusion models

    Marvin Li and Sitan Chen. Critical windows: Non-asymptotic theory for feature emergence in diffusion models. In Proceedings of the 41st International Conference on Machine Learning (ICML), ICML'24, pp.\ 1097:1--1097:25. JMLR.org, 2024

    2024

  19. [27]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben - Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openrev...

    2023

  20. [28]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024 a . URL https://openrevie...

    2024

  21. [29]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024 b . URL https://openrevie...

    2024

  22. [30]

    David J. C. MacKay. Information Theory, Inference & Learning Algorithms. Cambridge University Press, USA, 2002. ISBN 0521642981

    2002

  23. [31]

    Interacting particle solutions of fokker--planck equations through gradient--log--density estimation

    Dimitra Maoutsa, Sebastian Reich, and Manfred Opper. Interacting particle solutions of fokker--planck equations through gradient--log--density estimation. Entropy, 22 0 (8): 0 802, 2020. doi:10.3390/e22080802. URL https://www.mdpi.com/1099-4300/22/8/802

    2020 doi

  24. [32]

    An empirical Bayes estimator of the mean of a normal population

    Koichi Miyasawa. An empirical Bayes estimator of the mean of a normal population . Bulletin of the International Statistical Institute, 38: 0 181--188, 1961

    1961

  25. [33]

    Large Language Diffusion Models , 2025

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large Language Diffusion Models , 2025. URL https://arxiv.org/abs/2502.09992

    2025 arXiv

  26. [34]

    Stochastic control and nonequilibrium thermodynamical systems

    Michele Pavon. Stochastic control and nonequilibrium thermodynamical systems. Applied Mathematics and Optimization, 19 0 (1): 0 187--202, 1989. doi:10.1007/BF01448198. URL https://doi.org/10.1007/BF01448198

    1989 doi

  27. [35]

    Diffusion Autoencoders: Toward a Meaningful and Decodable Representation

    Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion Autoencoders: Toward a Meaningful and Decodable Representation . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 10619--10629, June 2022

    2022

  28. [36]

    Neural Entropy , 2025

    Akhil Premkumar. Neural Entropy , 2025. URL https://arxiv.org/abs/2409.03817

    2025

  29. [37]

    Tyrrell Rockafellar and Roger J.-B

    R. Tyrrell Rockafellar and Roger J.-B. Wets. Variational analysis / R. Tyrrell Rockafellar, Roger J.-B. Wets. Grundlehren der mathematischen Wissenschaften, 317. Springer, Berlin ;, 1998. ISBN 3540627723. URL http://swbplus.bsz-bw.de/bsz063165805cov.htm

    1998

  30. [38]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \" o rn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pp.\ 1067...

    2022

  31. [39]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. CoRR, abs/1505.04597, 2015. URL http://arxiv.org/abs/1505.04597

    2015 arXiv

  32. [40]

    D. L. Ruderman. The statistics of natural images. Network: Computation in Neural Systems, 5 0 (4): 0 517--548, November 1994. doi:10.1088/0954-898X/5/4/006. URL https://dx.doi.org/10.1088/0954-898X/5/4/006

    1994 doi

  33. [41]

    J. J. Sakurai and Jim Napolitano. Modern Quantum Mechanics. Cambridge University Press, Cambridge, 3 edition, 2020

    2020

  34. [42]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. CoRR, abs/2202.00512, 2022. URL https://arxiv.org/abs/2202.00512

    2022 arXiv

  35. [43]

    Entropy production along a stochastic trajectory and an integral fluctuation theorem

    Udo Seifert. Entropy production along a stochastic trajectory and an integral fluctuation theorem. Phys. Rev. Lett., 95: 0 040602, Jul 2005. doi:10.1103/PhysRevLett.95.040602. URL https://link.aps.org/doi/10.1103/PhysRevLett.95.040602

    2005 doi

  36. [44]

    Stochastic Thermodynamics, Fluctuation Theorems and Molecular Machines

    Udo Seifert. Stochastic Thermodynamics, Fluctuation Theorems and Molecular Machines . Reports on Progress in Physics, 75 0 (12): 0 126001, Nov 2012. doi:10.1088/0034-4885/75/12/126001. URL https://dx.doi.org/10.1088/0034-4885/75/12/126001

    2012 doi

  37. [45]

    C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27 0 (3): 0 379--423, 1948. doi:10.1002/j.1538-7305.1948.tb01338.x

    1948

  38. [46]

    Noiseless coding of correlated information sources

    David Slepian and Jack Wolf. Noiseless coding of correlated information sources. IEEE Transactions on Information Theory, 19 0 (4): 0 471--480, 1973. doi:10.1109/TIT.1973.1055037

    1973

  39. [47]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of ...

    2015

  40. [48]

    Maximum Likelihood Training of Score-Based Diffusion Models

    Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum Likelihood Training of Score-Based Diffusion Models . In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Syste...

    2021

  41. [49]

    Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole

    Yang Song, Jascha Sohl - Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations . In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May...

    2021

  42. [50]

    Edward O. Thorp. The Kelly Criterion in Blackjack Sports Betting, and the Stock Market . In Leonard C MacLean, Edward O Thorp, and William T Ziemba (eds.), THE KELLY CAPITAL GROWTH INVESTMENT CRITERION THEORY and PRACTICE , World Scientific Book Chapters, chapter 54, pp.\ 789-...

    2011

  43. [51]

    Naftali Tishby, Fernando C. N. Pereira, and William Bialek. The Information Bottleneck Method . CoRR, physics/0004057, 2000. URL http://arxiv.org/abs/physics/0004057

    2000 arXiv

  44. [52]

    Score-based Generative Modeling in Latent Space

    Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based Generative Modeling in Latent Space . In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 11287--1...

    2021

  45. [53]

    Vaikuntanathan and C

    S. Vaikuntanathan and C. Jarzynski. Dissipation and Lag in Irreversible Processes . Europhysics Letters, 87 0 (6): 0 60005, oct 2009. doi:10.1209/0295-5075/87/60005. URL https://dx.doi.org/10.1209/0295-5075/87/60005

    2009 doi

  46. [54]

    Visualizing data using t-sne

    Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9 0 (86): 0 2579--2605, 2008. URL http://jmlr.org/papers/v9/vandermaaten08a.html

    2008

  47. [55]

    Zhou Wang and Alan C. Bovik. Mean Squared Error: Love It or Leave It? A New Look at Signal Fidelity Measures . IEEE Signal Processing Magazine, 26 0 (1): 0 98--117, 2009. doi:10.1109/MSP.2008.930649

    2009

  48. [56]

    Information theoretical analysis of multivariate correlation

    Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of Research and Development, 4 0 (1): 0 66--82, 1960. doi:10.1147/rd.41.0066

    1960 doi

  49. [57]

    Energy-based diffusion language models for text generation

    Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovec, Stefano Ermon, and Arash Vahdat. Energy-based diffusion language models for text generation. CoRR, abs/2410.21357, 2024. doi:10.48550/ARXIV.2410.21357. URL https://doi.org/10.48550/arXiv.2410.21357

    2024 doi arXiv

  50. [58]

    A theory of usable information under computational constraints

    Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. U...

    2020

  51. [59]

    Characteristic guidance: Non-linear correction for diffusion model at large guidance scale

    Candi Zheng and Yuan Lan. Characteristic guidance: Non-linear correction for diffusion model at large guidance scale. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net/fo...

    2024

  52. [60]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  53. [61]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  54. [62]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  55. [63]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.