Pith. sign in

REVIEW 31 references

MAD: Manifold Attracted Diffusion

T0 review · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Score-based diffusion can be modified at inference time so that samples from noisy training data are pulled toward the clean data manifold.

arxiv 2509.24710 v2 pith:MMIVV7TK submitted 2025-09-29 stat.ML cs.LGcs.NAmath.NA

Dennis Elbrächter , Giovanni S. Alberti , Matteo Santacesaria This is my paper
classification stat.MLcs.LGcs.NAmath.NA
keywords datadistributionmanifoldapproximationdiffusiondirectionsefficientlymostly
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Diffusion models are trained to turn pure noise into images by slowly removing noise, guided by a 'score' that points toward the data. If the training images were already noisy, the model learns to reproduce that noise. MAD changes only the generation step, not the training. The idea is that real images live on a low-dimensional manifold, while noise wiggles in the many other directions. MAD defines an 'extended score' that behaves like the usual score for smooth data but treats perfect points (Dirac deltas) as if they were little Gaussian blobs. This makes the score point straight to the nearest data point, so small off-manifold wiggles are collapsed, while large along-manifold changes survive. The extended score is approximated by evaluating the same pretrained network at a slightly higher noise level, so it can be used with existing diffusion models. In toy examples, MAD shrinks low-variance directions to zero. On synthetic shapes corrupted by blur and pixel noise, MAD-generated samples clearly show the original shapes, which standard inference does not. On real cryo-EM images, MAD produces shapes resembling the known particle structure. However, all these results are shown as pictures, not measured numerically, and the method's key parameters (a, b, p, delta) were chosen by hand for each dataset. The paper's math proves the behavior only for very simple distributions, not for the general noisy-manifold case.
Extended reading notes

Core claim

The central claim is that MAD, an inference-time modification using the extended score, generates samples approximately from the clean distribution when trained on noisy data. The abstract states: 'we present an efficiently implementable modification of the inference procedure to generate noiseless samples' and that 'in a simplified setting, it can be used to reduce small variations to zero, while leaving large variations mostly unchanged.' If the paper is correct, one can take a score network trained on corrupted samples and produce clean samples without any training-phase changes, just extra score evaluations.

Load-bearing premise

Algorithm 1 approximates the derivative of the true score with respect to noise level by a finite difference of a neural network score approximator, d/dσ S_theta(t,x). The paper provides no argument that this network derivative is a faithful approximation to the true derivative of the score, nor that the correction factor m(t), derived only for a Dirac-delta target distribution, transfers to general manifold data. If this derivative is inaccurate, the extended-score estimate is wrong and the claimed denoising behavior would not occur. This is an unflagged assumption about the smoothness and trainability of score gradients.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method rests on the manifold hypothesis and on the availability of an accurate score network derivative. The only numbers fitted to data are the hand-chosen hyperparameters a,b,p,delta. No new physical entities are introduced. The main unfounded assumption is that the network's derivative w.r.t. noise level is reliable.

free parameters (4)
  • a = dataset-dependent: 2.5 for FFHQ/AFHQ, 8 for ImageNet, 0.002 for synthetic, 0.01 for EMPIAR-11618
    Scales the mapping between t and gamma(t); chosen by hand per dataset to obtain good visual results.
  • b = dataset-dependent: 2-80 for FFHQ/AFHQ, 5-250 for ImageNet, 15 for synthetic, 2 for EMPIAR-11618
    Controls the strength of manifold attraction in Algorithm 1; selected manually for each dataset.
  • p = dataset-dependent: 8 for FFHQ/AFHQ and synthetic, 12 for ImageNet
    Sets gamma(t) = t^p, regulating the speed at which the extended score limit is approached; chosen by hand.
  • delta = 0.0001 to 0.02 depending on dataset
    Step size for the finite-difference approximation of d/dsigma S_theta in Algorithm 1; not justified analytically.
assumptions (4)
  • domain assumption Data lie on a low-dimensional manifold and noise is small in off-manifold directions.
    Basis for the entire approach, stated in Section 1. If this is false, MAD has no grounding.
  • domain assumption A well-trained score network S_theta approximates the true score S_{p_sigma} for all sigma in the training range.
    Assumed in Section 2 to make the method work; the paper explicitly says 'we will simply assume that we can obtain a suitable approximation to the score'.
  • ad hoc to paper The finite-difference derivative d/dsigma S_theta(t,x) is a faithful approximation to the true derivative of the score.
    Algorithm 1 uses this derivative; no justification is given that the network's derivative is accurate, only that a second evaluation is cheap. This is a load-bearing assumption.
  • standard math Standard approximate identity results (Grafakos) for convolutions with Gaussians.
    Used in the proof of Lemma 3.2 in Appendix A.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MAD: Manifold Attracted Diffusion." pith.science (2026). https://pith.science/paper/MMIVV7TK

@misc{pith2026250924710,
  author       = {Pith},
  title        = {Pith review of: MAD: Manifold Attracted Diffusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MMIVV7TK}},
  note         = {Machine review of arXiv:2509.24710}
}
read the original abstract

Score-based diffusion models are a highly effective method for generating samples from a distribution of images. We consider scenarios where the training data comes from a noisy version of the target distribution, and present an efficiently implementable modification of the inference procedure to generate noiseless samples. Our approach is motivated by the manifold hypothesis, according to which meaningful data is concentrated around some low-dimensional manifold of a high-dimensional ambient space. The central idea is that noise manifests as low magnitude variation in off-manifold directions in contrast to the relevant variation of the desired distribution which is mostly confined to on-manifold directions. We introduce the notion of an extended score and show that, in a simplified setting, it can be used to reduce small variations to zero, while leaving large variations mostly unchanged. We describe how its approximation can be computed efficiently from an approximation to the standard score and demonstrate its efficacy on toy problems, synthetic data, and real data.

Figures

Figures reproduced from arXiv: 2509.24710 by the authors.

Figure 1
Figure 1. Comparison of inference trajectories (red) and samples (blue) for Gaussian [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Target distribution is displayed as a histogram colormap. Red crosses indicate [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. In both subfigures, all images in each row start from the same latent noise [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Each row starts from the same latent noise sample, and the leftmost column [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Generated with Algorithm 1 from the same latent noise sample with [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Grid of clean samples from the synthetic data set (left); grid of corresponding [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Grid of samples generated from consecutive random seeds by Algorithm 1 with [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Images from the EMPIAR-11618 dataset (left). Samples generated by Algo [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: In both subfigures, all images in each row starts from the same latent noise sam [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: All images in each row starts from the same latent noise sample and the left [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Generated with Algorithm 1 from the same latent noise [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Generated with Algorithm 1 from the same latent noise sample with [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Images from the EMPIAR-11618 dataset (left). Images generated by standard [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 1 linked inside Pith

  1. [1]

    C., Brackmann, K., Dimakou, D., Halic, M., Hewitt, G., Boulton, S

    Bacic, L., Gaullier, G., Sabantsev, A., Lehmann, L. C., Brackmann, K., Dimakou, D., Halic, M., Hewitt, G., Boulton, S. J., and Deindl, S. (2021). Structure and dynamics of the chromatin remodeler alc1 bound to a parylated nucleosome. eLife , 10:e71420

    2021

  2. [2]

    Bengio, Y., Courville, A., and Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798--1828

    2013

  3. [3]

    Bortoli, V. D. (2022). Convergence of denoising diffusion models under the manifold hypothesis. Transactions on Machine Learning Research

    2022

  4. [4]

    and De Vleeschouwer, C

    Brummer, B. and De Vleeschouwer, C. (2019). Natural image noise dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops

    2019

  5. [5]

    A., and Walz, T

    Cheng, Y., Grigorieff, N., Penczek, P. A., and Walz, T. (2015). A primer to single-particle cryo-electron microscopy. Cell , 161:438--449

    2015

  6. [6]

    Choi, Y., Uh, Y., Yoo, J., and Ha, J.-W. (2020). Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    2020

  7. [7]

    T., Klasky, M

    Chung, H., Kim, J., Mccann, M. T., Klasky, M. L., and Ye, J. C. (2023). Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations

    2023

  8. [8]

    Chung, H., Sim, B., Ryu, D., and Ye, J. C. (2022). Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems , 35:25683--25696

    2022

Show all 31 references
  1. [9]

    Daras, G., Shah, K., Dagan, Y., Gollakota, A., Dimakis, A., and Klivans, A. (2023). Ambient diffusion: Learning clean distributions from corrupted data. Advances in Neural Information Processing Systems , 36:288--313

    2023

  2. [10]

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248--255

    2009

  3. [11]

    and Nichol, A

    Dhariwal, P. and Nichol, A. (2021). Diffusion models beat gans on image synthesis. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems , volume 34, pages 8780--8794. Curran Associates, Inc

    2021

  4. [12]

    and Levin, D

    Faigenbaum-Golovin, S. and Levin, D. (2023). Manifold reconstruction and denoising from scattered data in high dimension. J. Comput. Appl. Math. , 421:Paper No. 114818, 24

    2023

  5. [13]

    Fefferman, C., Ivanov, S., Kurylev, Y., Lassas, M., and Narayanan, H. (2018). Fitting a putative manifold to noisy data. In Conference On Learning Theory , pages 688--720. PMLR

    2018

  6. [14]

    Fefferman, C., Mitter, S., and Narayanan, H. (2016). Testing the manifold hypothesis. Journal of the American Mathematical Society , 29(4):983--1049

    2016

  7. [15]

    Gong, D., Sha, F., and Medioni, G. (2010). Locally linear denoising on image manifolds. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages 265--272. JMLR Workshop and Conference Proceedings

    2010

  8. [16]

    Grafakos, L. (2014). Classical F ourier analysis , volume 249 of Graduate Texts in Mathematics . Springer, New York, third edition

    2014

  9. [17]

    and Gupta, A

    Gupta, S. and Gupta, A. (2019). Dealing with noise problem in machine learning data-sets: A systematic review. Procedia Computer Science , 161:466--474. The Fifth Information Systems International Conference, 23-24 July 2019, Surabaya, Indonesia

    2019

  10. [18]

    and Maier, M

    Hein, M. and Maier, M. (2006). Manifold denoising. Advances in neural information processing systems , 19

    2006

  11. [19]

    Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems , volume 33, pages 6840--6851

    2020

  12. [20]

    Karras, T., Aittala, M., Aila, T., and Laine, S. (2022). Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems , volume 35, pages 26565--26577

    2022

  13. [21]

    Karras, T., Laine, S., and Aila, T. (2019). A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2019

  14. [22]

    Lu, H., Wu, Q., and Yu, Y. (2025). Stochastic forward backward deconvolution: Training diffusion models with finite noisy datasets. In Forty-second International Conference on Machine Learning

    2025

  15. [23]

    Pope, P., Zhu, C., Abdelkader, A., Goldblum, M., and Goldstein, T. (2021). The intrinsic dimension of images and its impact on learning. In International Conference on Learning Representations

    2021

  16. [24]

    Potaptchik, P., Azangulov, I., and Deligiannidis, G. (2024). Linear convergence of diffusion models under the manifold hypothesis. arXiv preprint arXiv:2410.09046

    2024 arXiv

  17. [25]

    Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In Navab, N., Hornegger, J., Wells, W. M., and Frangi, A. F., editors, Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2015 , pages 234--241...

    2015

  18. [26]

    and Ermon, S

    Song, Y. and Ermon, S. (2020). Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations

    2020

  19. [27]

    P., Kumar, A., Ermon, S., and Poole, B

    Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. (2021). Score-based generative modeling through stochastic differential equations

    2021

  20. [28]

    P., Batzolis, G., Deveney, T., and Sch \"o nlieb, C.-B

    Stanczuk, J. P., Batzolis, G., Deveney, T., and Sch \"o nlieb, C.-B. (2024). Diffusion models encode the intrinsic dimension of data manifolds. In Forty-first International Conference on Machine Learning

    2024

  21. [29]

    and Yang, Y

    Tang, R. and Yang, Y. (2024). Adaptivity of diffusion models to manifold structures. In Dasgupta, S., Mandt, S., and Li, Y., editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , volume 238 of Proceedings of Machine Learning Rese...

    2024

  22. [30]

    and Carreira-Perpin \'a n, M

    Wang, W. and Carreira-Perpin \'a n, M. A. (2010). Manifold blurring mean shift algorithms for manifold denoising. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 1759--1766. IEEE

    2010

  23. [31]

    wwPDB Consortium, T. (2023). Emdb—the electron microscopy data bank. Nucleic Acids Research , 52(D1):D456--D465

    2023

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.