Pith. sign in

REVIEW 4 major objections 5 minor 14 references

Measuring Semantic Information Production in Generative Diffusion Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The rate at which entropy drops reveals when a diffusion model commits to a class.

desk verdict A useful framing for timing semantic decisions in diffusion, but the CIFAR10 results rest on a posterior estimator that is inconsistent with the paper's own equation, so the class-specific peaks may be artifacts. read the letter →

arxiv 2506.10433 v1 pith:7477FWAU submitted 2025-06-12 stat.ML cs.LG

Florian Handke , Félix Koulischer , Gabriel Raya , Luca Ambrogioni This is my paper · ORCID
classification stat.MLcs.LG MSC 94A1768T07
keywords semanticinformationdiffusionmodelsconditionalentropyratedecisionwindowsphasetransitionsclassifier-freeguidanceCIFAR10
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the temporal derivative of the conditional entropy H(c|x_t), which measures remaining uncertainty in the class label given the current noisy image, pinpoints the moments during reverse diffusion when class-semantic information is actually produced. The authors estimate this derivative without knowing the data distribution, using an online Bayesian update of the class posterior along generated trajectories, and demonstrate it on a Gaussian mixture toy model and on DDPM models trained on CIFAR10. Their central finding is that semantic information transfer is concentrated in intermediate diffusion stages and vanishes near the end, but the peak times differ between classes, suggesting that different 'semantic decisions' occur at different intermediate times. If correct, this yields a scalable, distribution-agnostic tool to localize decision windows in any trained diffusion model.

What carries the argument

The central quantity is the conditional entropy of a binary decision variable z that partitions the class set, H(z|x_t) = -∫ p_z(x_t) Σ_z P(z|x_t)  log₂ P(z|x_t) dx_t, together with its temporal derivative Ḍ(z|x_t). The estimator rests on the iterative Bayesian update (Eq. 3): log P(c|x_t) = log P(c|x_{t+1}) + [log p(x_t|x_{t+1}, c) − log p(x_t|x_{t+1})], which converts the difference between a model's class-conditional and unconditional noise predictions into an approximate posterior ratio, and so yields entropy rates from a pair of forward passes.

What would settle it

Train a diffusion model on a Gaussian mixture with known analytic posteriors, compute the exact entropy-rate profile, and compare it with the estimator's profile on model-generated samples; a mismatch in peak location or a violation of 0 ≤ H(c|x_t) ≤ H(c) would falsify the claim. Alternatively, an external classifier trained on corrupted real data could provide a ground-truth conditional entropy to compare against.

Watch

Extended reading notes

Core claim

The paper claims that a precise information-theoretic measure of semantic generation can be extracted from a trained diffusion model without any external classifier or training-time labels. Treating generation as information transmission through the forward noising channel, the conditional entropy H(c|x_t) quantifies remaining class uncertainty, and its time derivative (the entropy rate) measures bits of semantic information restored per denoising step. Using an iterative posterior update from Koulischer et al. (2025), the authors estimate these quantities from a model's own conditional and unconditional noise predictions. On CIFAR10 they find the derivative is highest at intermediate noise levels, with class-pair-specific peak locations that match intuitive feature similarity (e.g., deer vs car diverges earlier than deer vs bird).

Load-bearing premise

The iterative Bayesian posterior update yields accurate class posteriors P(c|x_t) along reverse-sampled trajectories, and the replacement of the complement-class score by the null-model score does not distort the entropy-rate profiles.

Editorial extensions

If this is right

  • The method yields class-specific decision-time profiles, so generation pipelines can know which classes commit early and which commit late.
  • Restricting classifier-free guidance to the measured high-information interval could preserve diversity while improving alignment, extending the limited-interval guidance result of Kynkäänniemi et al. (2024).
  • The entropy-rate peak provides an empirical signature of the predicted phase transition or critical window, connecting information theory to the symmetry-breaking picture.
  • The approach transfers to any conditional generation model for which conditional and unconditional score estimates are available, not just diffusion on images.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If entropy-rate peaks differ by class, scheduling guidance strength per class rather than globally could improve image fidelity; the paper hints at this direction but does not test it.
  • The null-model approximation for ¬c should become exact as the number of classes grows, so a direct comparison between the pairwise two-class estimator and the null-model estimator would quantify the bias.
  • The same derivative could be applied to other semantic attributes, such as pose or colour, by redefining the partition z, yielding a general 'semantic decision atlas' for a trained generative model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces an information-theoretic method for measuring when class-semantic information is produced during the reverse diffusion process of generative models. The key object is the conditional entropy H(c|x_t) of a binary decision variable z given the noisy state, and its time derivative (entropy rate). The method is exact for Gaussian mixture models (Appendix B). For a DDPM trained on CIFAR10, the posterior is estimated with an iterative scheme from Koulischer et al. and the complement class is approximated by the unconditional model. The paper reports that entropy rate peaks in intermediate diffusion stages and differs across classes, implying class-specific decision windows.

Significance. If valid, the method would provide a distribution-agnostic, empirical tool to localize decision windows in diffusion models, complementing theoretical phase-transition results. The GMM derivations are clean and reproducible. However, the empirical claims rest on a posterior estimator that is not validated on a ground truth, and the CIFAR10 experiments lack statistical controls. The central idea is promising, but the current evidence is not sufficient to support the strong claims.

major comments (4)
  1. [Section 3.1, Eq. (3), Appendix C.1, Algorithm 1] The iterative posterior update is stated in three inconsistent forms. Eq. (3) gives log P(c|x_t) = log P(c|x_{t+1}) - 1/(2 sigma_t^2) Delta, Algorithm 1 line 16 uses the factor 1/(1-beta_t) instead of 2 sigma_t^2, and the prose before Algorithm 1 omits the minus sign. Since the scale factor changes the posterior by orders of magnitude for standard DDPM beta_t, the entropy-rate peaks in Fig. 3 may be artifacts of the update's normalization. The authors must specify sigma_t^2, reconcile these equations, and validate the estimator against a known ground truth.
  2. [Section 4, Fig. 3] The approximation ∇ log p(x_t|¬c) ≈ ∇ log p(x_t|∅) is used for the 'deer vs. not deer' curve. The unconditional model is a mixture over all ten classes including deer, so 'not deer' is contaminated by the target class. This is not a small error for a 10-class problem, and it directly affects the class-specific curves that are the paper's central empirical finding. A validation experiment (e.g., training a proper 9-class complement model or ablating the target class from the null model) is needed before the class-comparison claims can be trusted.
  3. [Section 4, Fig. 3] The CIFAR10 entropy-rate profiles are single estimates with no error bars, no repeated runs, and no null baseline. The claim that 'sizable differences' between classes exist is not supported by any statistical analysis, and the intermediate peaks are not compared against a null model (e.g., a model with randomized class labels). Without these controls, the observed profiles could reflect estimator noise rather than semantic information production.
  4. [Appendix B] The GMM validation computes exact posteriors in closed form and never runs Algorithm 1. Since Algorithm 1 is the tool used for the CIFAR10 claims, the paper should demonstrate on the GMM (or another ground-truth setting) that the iterative posterior estimator is accurate along reverse-sampled trajectories. The current GMM experiment is therefore not a validation of the central estimator.
minor comments (5)
  1. [Appendix C.1] The formula immediately before Algorithm 1 is missing the minus sign present in both Eq. (3) and Algorithm 1 line 16; please correct this typo.
  2. [Section 3.1 / 4] It is not stated how the temporal derivative \dot H(c|x_t) is computed from the discrete H_t values; please describe the differentiation procedure and any smoothing.
  3. [Figure 3] The right panel lacks axis labels; please specify the units of the entropy rate and the time axis.
  4. [Abstract] The abstract says the method is 'general', but the experiments cover only one dataset and a few class pairs; please temper the wording or expand the experiments.
  5. [Eq. (3)] Please define sigma_t^2 explicitly; in DDPM notation it could be beta_t, 1-alpha_t, or the reverse variance, and the value changes the numerical results.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the entropy-rate estimates are observations of model behavior, not fits, and the self-cited posterior estimator is re-derived in Eq. 3 rather than imported as a black box.

full rationale

The paper's central derivation chain is self-contained at the level of equations. Eq. 1 and Eq. 2 define conditional entropy and the binary-partition version; Eq. 3 derives the posterior update from Bayes rule and Gaussian transition assumptions, rather than merely importing it. Appendix B derives the GMM marginals, likelihoods, and posteriors in closed form, and the GMM entropy estimates use those exact quantities, not a fitted surrogate. The CIFAR10 procedure applies Algorithm 1, which is based on the same Eq. 3 update, to the trained model's own conditional and unconditional denoisers; the resulting entropy-rate profiles are observations about that model, not predictions obtained by fitting parameters to the quantity being reported. The self-citation to Koulischer et al. (2025) is present and involves overlapping authors, but it is not load-bearing in the circular sense: the estimator is parameter-free, its stated assumptions (Markov forward process, Gaussian noise) do not include the target result, and the present paper re-derives the update. The reader's flagged weaknesses are correctness and robustness risks rather than circularity: the GMM section validates with closed-form posteriors instead of running Algorithm 1; the approximation nabla log p(x_t|not c) approx nabla log p(x_t|empty) is uncontrolled; and Algorithm 1 line 16 uses exp(-1/(1-beta_t) Delta) while Eq. 3 has exp(-1/(2 sigma_t^2) Delta), an inconsistency that could affect the estimated profiles but does not make the output equal to the input. The conclusion itself concedes that the method 'remains to be extensively validated' on more complex datasets, which is an acknowledged limitation, not a circular step. Overall, no specific reduction of a claimed prediction to a fitted input or self-citation chain is exhibited, so the paper receives a low score reflecting only the minor self-citation and unvalidated estimator.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central measurement rests on the posterior estimator from Koulischer et al. (2025) and the complement-class approximation; the only hand-tuned analysis parameter is the binary decision prior. No new entities are introduced.

free parameters (1)
  • Binary decision prior P(z0) = 0.1 for class vs. complement; 0.5 for class vs. class
    Hand-chosen priors in Section C.1. The entropy H(z|x_t) and its derivative depend on this prior; the paper does not report sensitivity.
assumptions (3)
  • standard math The forward diffusion process is Markovian, so P(c|x_{t:T}) = P(c|x_t).
    Invoked in Section 2 after Eq. 1. Standard property of the forward noising process.
  • domain assumption The iterative posterior update (Eq. 3) from Koulischer et al. (2025) correctly estimates P(c|x_t) along reverse trajectories.
    Section 3.1. The paper relies on this estimator without independent validation in this setting.
  • ad hoc to paper The unconditional model's score approximates the complement class score: ∇ log p(x_t|¬c) ≈ ∇ log p(x_t|∅).
    Section 3.1 and C.1. Used for all 'not deer' computations; approximation quality grows with number of classes but is unquantified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Measuring Semantic Information Production in Generative Diffusion Models." pith.science (2026). https://pith.science/paper/7477FWAU

@misc{pith2026250610433,
  author       = {Pith},
  title        = {Pith review of: Measuring Semantic Information Production in Generative Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7477FWAU}},
  note         = {Machine review of arXiv:2506.10433}
}
read the original abstract

It is well known that semantic and structural features of the generated images emerge at different times during the reverse dynamics of diffusion, a phenomenon that has been connected to physical phase transitions in magnets and other materials. In this paper, we introduce a general information-theoretic approach to measure when these class-semantic "decisions" are made during the generative process. By using an online formula for the optimal Bayesian classifier, we estimate the conditional entropy of the class label given the noisy state. We then determine the time intervals corresponding to the highest information transfer between noisy states and class labels using the time derivative of the conditional entropy. We demonstrate our method on one-dimensional Gaussian mixture models and on DDPM models trained on the CIFAR10 dataset. As expected, we find that the semantic information transfer is highest in the intermediate stages of diffusion while vanishing during the final stages. However, we found sizable differences between the entropy rate profiles of different classes, suggesting that different "semantic decisions" are located at different intermediate times.

Figures

Figures reproduced from arXiv: 2506.10433 by the authors.

Figure 1
Figure 1. Generative information transfer in 1D Gaussian-mixture diffusion. Left) Diffusion of a one-dimensional equally weighted Gaussian-mixture with four data points (classes) at (-8,-4, 6, 8) along with H˙ (z|xt) for different decision problems. Right) Posterior evolu￾tions of classes 3 (top left), 2 (top right), 1 (bottom left), 0 (bottom right). The time axes represent the normalized number of noise additions according … view at source ↗
Figure 2
Figure 2. Transmitting class-semantic information in generative diffusion. Information transmission in generative diffusion from the perspective of the class random variable c. In contrast to the conventional description, the receiver p(x0|xt) is exchanged with a classifier P(c|xt), and the destination p(x0) is replaced by P(c). As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. shows the estimated temporal derivatives of the conditional entropy for the decision prob￾lems just described for a model trained on CIFAR10 (Appendix C.3). Here, we use ”deer” as the target class (z0) and ”airplane”, ”bird”, ”cat”, ”car”, and ”not deer” as the counterparts (z1) respec￾tively [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: GMM with mixture weights (π0, π1) = (0.5, 0.5) and deltas at (µ0, µ1) = (−1, 1) [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: GMM with mixture weights (π0, π1) = (1/3, 2/3) and deltas at (µ0, µ1) = (−1, 1) [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: GMM with mixture weights (π0, π1, π2) = (1/3, 1/3, 1/3) and deltas at (µ0, µ1, µ2) = (−2, 0, 2) [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: GMM with mixture weights (π0, π1, π2) = (0.25, 0.25, 0.5) and deltas at (µ0, µ1, µ2) = (−2, 0, 2). 9 [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: GMM with mixture weights (π0, π1, π2) = (1/3, 1/3, 1/3) and deltas at (µ0, µ1, µ2) = (−2, 1, 2) [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: GMM with mixture weights (π0, π1, π2, π3) = (0.25, 0.25, 0.25, 0.25) and deltas at (µ0, µ1, µ2, µ3) = (−8, −4, 4, 8). C DIFFUSION ON CIFAR10 C.1 ESTIMATING THE CONDITIONAL ENTROPY To estimate Eq. 2 we need to approximate the sampler, pz(xt) and the posterior, P(z0|xt).…
Figure 10
Figure 10. Figure 10: Experiments using the target class ”cat”. [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Experiments using the target class ”airplane”. [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: Experiments using the target class ”car”. [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 13
Figure 13. Figure 13: Experiments using the target class ”bird”. [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 5 canonical work pages

  1. [1]

    We will derive them in the following

    ·plogp +p(x t|z1)·log(1−p)dx t +c = 1 2 Z p(xt|z0) +p(xt|z1) ·plogp+p(x t|z1)·log(1−p)dx t +c Likewise, one can transform the right summand: p(xt|z1)·log(1−p) = p(xt|z0) +p(xt|z1) (1−p) log(1−p) The final result then yields: J SD(p(xt|z0)||p(xt|z1)) = 1 2 Z p(xt|z0) +p(xt|z1) ·plogp + p(xt|z0) +p(xt|z1) ·(1−p) log(1−p)dx t +c = 1 2 Z p(xt|z0)· plogp+ (1−p...

    work page 2025

  2. [5]

    Marvin Li and Sitan Chen

    URLhttps://arxiv.org/abs/2404.07724. Marvin Li and Sitan Chen. Critical windows: non-asymptotic theory for feature emergence in diffusion models,

    arXiv

  3. [6]

    Gabriel Raya and Luca Ambrogioni

    URLhttps://arxiv.org/abs/2403.01633. Gabriel Raya and Luca Ambrogioni. Spontaneous symmetry breaking in generative diffusion mod- els,

    arXiv

  4. [13]

    We estimate the integral using a discretization forx t and then taking the Riemann sum

    +P(c2) Z p(xt|ci) 2X j P(c j|xt) P(c 1|xt) +P(c2|xt) log P(c j|xt) P(c 1|xt) +P(c2|xt) dxt The likelihoodsp(x t|ci)and the posteriorsP(c j|xt)were derived in the previous section and are given by: p(xt|ci) =N(x t|µit, σ2 it) P(c j|xt) = πjN(x t|µjt, σ2 jt) π1N(x t|µ1t, σ2 1t) +π2N(x t|µ2t, σ2 2t) The priors are simplyP(c i) = πi π1+π2 . We estimate the in...

    work page 2025

  5. [14]

    11 Published as a DeLTa Workshop Paper at ICLR 2025 Figure 11: Experiments using the target class ”airplane”

    +ϵ 16:P(z 0|xt−1) =P(z 0|xt) exp − 1 1−βt ||xt−1 −µ θ(xt;z 0)||2 − ||xt−1 −µ θ(xt;z 1)||2 17:P(z 1|xt−1) = 1−P(z0|xt−1) 18:x z t−1 =x t−1 19:H z t−1 = 1 Nz P xt−1 P(z 0|xt−1) logP(z0|xt−1) +P(z1|xt−1) logP(z1|xt−1) 20:end for 21:H t−1 =− P(z 0)H z0 t + (1−P(z 0))H z1 t 22:end for 23:output:H T:0 C.2 ADDITIONALEXAMPLES Figure 10: Experiments using the targ...

    work page 2009

  6. [1948]

    Jascha Sohl-Dickstein, Eric A

    doi: 10.1002/j.1538-7305.1948.tb01338.x. Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics,

    arXiv 1948

  7. [2015]

    Yang Song, Jascha Sohl-Dickstein, Diederik P

    URLhttps://arxiv.org/ abs/1503.03585. Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations,

    arXiv

  8. [2016]

    5 Published as a DeLTa Workshop Paper at ICLR 2025 A JENSEN-SHANNONDIVERGENCE ANDCONDITIONALENTROPY The Jensen-Shannon Divergence (J SD) can be defined using Kullback-Leibler Divergences be- tween two distributionsp,qand their mixturem= p+q 2 . Withp=p(x t|z0)andq=p(x t|z1)J SD can accordingly be written as: J SD(p(xt|z0)||p(xt|z1)) = 1 2 DKL p(xt|z0)||m ...

    work page 2025

Show all 14 references
  1. [2020]

    Felix Koulischer, Johannes Deleu, Gabriel Raya, Thomas Demeester, and Luca Ambrogioni

    URL https://arxiv.org/abs/2006.11239. Felix Koulischer, Johannes Deleu, Gabriel Raya, Thomas Demeester, and Luca Ambrogioni. Dy- namic negative guidance of diffusion models,

    2006 arXiv

  2. [2021]

    Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, and Alex Graves

    URL https://arxiv.org/abs/2011.13456. Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, and Alex Graves. Con- ditional image generation with pixelcnn decoders.Advances in neural information processing systems, 29,

    2011 arXiv

  3. [2022]

    org/abs/2207.12598

    URLhttps://arxiv. org/abs/2207.12598. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models,

    arXiv

  4. [2023]

    Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma

    URLhttps://arxiv.org/abs/2305.19693. Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications.arXiv preprint arXiv:1701.05517,

    arXiv

  5. [2024]

    doi: 10.1038/ s41467-024-54281-3

    ISSN 2041-1723. doi: 10.1038/ s41467-024-54281-3. URLhttp://dx.doi.org/10.1038/s41467-024-54281-3. Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance,

    doi

  6. [2025]

    Alex Krizhevsky, Geoffrey Hinton, et al

    URLhttps://arxiv.org/abs/ 2410.14398. Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.