REVIEW 4 major objections 5 minor 16 references
Clustering via Self-Supervised Diffusion
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read CLUDI repurposes a diffusion model as a clustering engine and beats all compared unsupervised classifiers.
desk verdict New clustering method that couples diffusion-sampled assignment embeddings with self-distillation; the diffusion component is plausible but the paper never tests whether it actually drives the gains. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a feature-conditioned diffusion model acting as a stochastic cluster-assignment sampler. Starting from $z_T \sim \mathcal{N}(0, F^2 I_d)$, a 25-step stochastic DDIM reverse process (Eqs. 7-9) denoises to an assignment embedding $z_0$, which a logit projection $L$ and tempered softmax convert into cluster probabilities. Three pieces carry the argument: (i) the Monte Carlo averaging of $B$ sampled probability vectors (Eq. 3), which stabilizes predictions; (ii) the teacher-student self-distillation scheme with SimSiam-style shared weights, where the student must predict both the denoised embedding $z_0$ (MSE loss) and the cluster probabilities $u$ (cross-entropy with a uniform-prior regularizer drawn from Self-Classifier); and (iii) the auxiliary embedding matrix $E$ that maps probabilities back into the embedding space, playing the role of the predictor network in BYOL and SimSiam and enabling the embedding targets in Eq. 13. The sqrt noise schedule and Min-SNR-$\gamma$ weighting control how different noise levels contribute to training.
What would settle it
Train CLUDI's identical student and losses but replace the diffusion teacher with a non-diffusion stochastic teacher that emits the same number $B$ of diverse assignment embeddings per image (for example, noise-injected MLP outputs with the same target normalization); if it matches CLUDI's accuracy on ImageNet-100 validation, the diffusion mechanism is not the load-bearing ingredient. A second check: compare CLUDI at $B=1$ against the averaged $B>1$ prediction — if single-sample accuracy already matches the averaged accuracy, the Monte Carlo averaging in Eq. 3 contributes nothing.
Extended reading notes
Core claim
The central claim is that the stochasticity of a diffusion model can serve as a structured data-augmentation mechanism for cluster assignment, not just for image generation. Concretely, CLUDI defines the cluster-posterior $p_\theta(k|x)$ as a Monte Carlo average over $B$ samples from a data-conditioned diffusion model, $p_\theta(k|x) \approx \frac{1}{B}\sum_{i=1}^B p(k|z_0^i)$, where each $z_0^i \in \mathbb{R}^d$ is an assignment embedding obtained by stochastic DDIM reverse sampling starting from pure Gaussian noise. The teacher's diverse assignments become training targets for a student network, and training combines an MSE loss on denoised embeddings with a cross-entropy loss on cluster probabilities that is regularized by a uniform prior over the mini-batch to prevent collapse. The paper argues that this yields the best reported NMI, ACC, and ARI on all tested benchmarks, outperforming SCAN, ProPos, TSP, TEMI, and Self-Classifier variants, and that averaging over many samples makes predictions on heavily corrupted inputs nearly as accurate as on clean inputs.
Load-bearing premise
The load-bearing premise is that stochastic diversity from the diffusion sampler itself — 25 DDIM steps starting from pure noise — is what makes the teacher's assignment targets informative and accurate, yet the paper never tests a non-diffusion teacher, and its own ablations show results hinge on tuning the noise scale $F^2$ and embedding dimension $d$.
Editorial extensions
If this is right
- On every setting the paper reports — CIFAR-10, STL-10, Oxford-IIIT Pets, Oxford 102 Flower, Caltech-101, and the ImageNet-50/100/200 subsets — CLUDI reports the highest NMI, ACC, and ARI among the compared self-supervised clustering methods, with its largest margins on the ImageNet subsets.
- Accuracy on inputs corrupted by feature dropout and Gaussian noise approaches clean-input accuracy as the number of averaged assignment samples $B$ grows (Figure 2), so the method's robustness comes from sampling diversity rather than from extra augmentation pipelines or neighbor mining.
- Because CLUDI works on frozen pre-trained features and needs no multi-head selection or pseudo-label refinement, the clustering stage is comparatively lightweight, inheriting the efficiency of pre-trained-feature pipelines while adding only the diffusion head.
- The reported gains depend on the method's specific design choices — the $E$ matrix predictor, the uniform-prior classification loss, and the Min-SNR weighting — which together prevent the trivial solutions that earlier deep clustering methods struggled with.
Reading between the lines
- If diffusion noise is the active ingredient, the same teacher-student scheme should transfer beyond images: any frozen embedding space (text, bioinformatics, graphs) could be clustered with a small diffusion head, which would be a cheap experiment to run.
- A sharper test the authors leave implicit: replace the diffusion teacher with a non-diffusion stochastic sampler, such as an MLP emitting equally diverse noise-injected assignment embeddings with the same targets and losses; comparable accuracy would mean the sampling mechanism, not diffusion as such, is doing the work.
- The sampling count $B$ acts as an inference-time compute knob — a deployment could run at low $B$ where latency matters and upgrade to high $B$ where accuracy matters, without retraining.
- The paper's limitation section reports degradation as the number of clusters $K$ grows; a natural stress test is whether hierarchical or multi-scale diffusion with $K$-dependent noise schedules preserves cluster separability at large $K$, the direction the authors point to in their conclusion.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CLUDI, a self-supervised clustering method that operates on pre-trained DINO ViT features. The core idea is to use a diffusion model as a teacher: conditioned on a feature vector x, the teacher runs a stochastic DDIM sampling procedure (25 training steps, 100 inference steps) from Gaussian noise to produce multiple assignment embeddings z0, which are converted to cluster probabilities via a trainable logit layer. The student, fed noisy/augmented versions of the input and of the teacher's embeddings, is trained to predict both the embeddings and the cluster probabilities under an MSE loss and a regularized cross-entropy loss with a uniform-prior term. At inference, the probabilities from B sampled embeddings are averaged. Experiments on CIFAR-10, STL-10, ImageNet subsets, and several fine-grained datasets report state-of-the-art clustering accuracy, with ablations over the loss weight λ, the noise scale F², the embedding dimension d, and the number of averaged samples B.
Significance. If the reported results hold, CLUDI would be a novel and effective application of diffusion models to clustering, and the paper's central claim that stochastic diffusion-based assignment generation improves unsupervised classification would be of clear interest to the deep clustering community. The paper is also commendable for providing code, for comparing against a frozen-feature Self-Classifier baseline on identical DINO features, and for running several hyperparameter ablations. However, the significance is currently qualified by the absence of a control that isolates the diffusion component from the other self-distillation and regularization ingredients, and by the lack of error bars and non-neural baselines; these omissions make the state-of-the-art claim and the attribution of the gain to diffusion uncertain.
major comments (4)
- [Section 5.1, 5.4 and Section 6] The paper claims that stochastic diffusion-based sampling is what drives performance, but no ablation removes or replaces the diffusion teacher while keeping all other components fixed. The only non-diffusion comparison, Self-Classifier*, differs from CLUDI in many simultaneous ways: it has no embedding MSE loss (Eq. 18), no student feature/embedding noise (Section 5.2), no E-projection (Eq. 13), and no B-averaging at inference. The ablations in Figures S1–S3 and Figure 2 vary λ, F², d, and B, but never replace the teacher with a deterministic or single-step denoiser. Therefore the reported gains could in principle come entirely from the two losses and the uniform-prior regularization applied to any stochastic embedding generator, with the 25-step diffusion process acting only as an expensive source of randomness. A direct control is needed: keep the student and losses identical, but replace the 25-step DDIM teacher with, for example, a single-step denoiser that maps (x, Gaussian noise) to an embedding, or with a deterministic MLP; if accuracy is unchanged, the load-bearing attribution to diffusion is not established.
- [Tables 1 and 2, Section 6] The main results are reported as single numbers with no error bars or number of seeds. Since several claimed wins are small (e.g., CIFAR-10 NMI 89.6 vs. TEMI 88.6, ImageNet-200 ACC 73.7 vs. 73.2), single runs cannot support a state-of-the-art claim. Additionally, there is no k-means or Gaussian-mixture baseline on the same DINO features; without such baselines, the reader cannot tell how much of the improvement comes from the clustering model rather than from the quality of the pre-trained features. Please report mean±standard deviation over at least three seeds for all tables and add k-means/GMM on the same frozen DINO features as lower-bound references.
- [Section 3 and Section 4, Eqs. (7)–(10)] The DDIM sampling equations are not self-consistent as written. Section 3 defines α_t := 1−β_t and ar α_t as the cumulative product, but Eq. (8) uses α_s and α_t in the standard DDIM update, which requires the cumulative product (usually denoted ar α_s and ar α_t), while Eq. (10) correctly uses ar α_t. As written, the variance terms and the noise prediction in Eq. (8) are inconsistent with the forward process in Eq. (2). Please correct the notation (either redefine α_t as the cumulative product in Section 4 or replace α_s, α_t with ar α_s, ar α_t in Eq. (8)), so that the method is reproducible from the text alone.
- [Section 5.4, Eqs. (21)–(23)] The derivation of the regularized cross-entropy loss contains a garbled line. Eq. (23) reads 'p(k|\hat z) = p(k)p(k|\hat z)/p(k) = (NB/K) p(k|\hat z) Σ ...', which is tautological once p(k) is canceled. The intended expression should presumably define p(\hat z|k) via a column softmax analogous to Eq. (21) and then apply Bayes to obtain p(k|\hat z). As printed, the posterior probability formula is not well-defined and the loss in Eq. (24) cannot be reconstructed. Please rewrite Eqs. (21)–(23) with clear, correct definitions of the likelihood and posterior terms.
minor comments (5)
- [Section 6, Experimental Setup] The paper should state explicitly that all hyperparameters (d, F², λ) were selected on the ImageNet-100 validation set and then fixed for other datasets, and clarify whether the numbers in Tables 1–2 are test-set results or validation-set results for ImageNet-100.
- [Figure 2 caption] The caption says accuracy 'becomes similar as the number of data samples grows', but the x-axis appears to be B (number of averaged samples), not the number of data samples; please rephrase.
- [Section 5.4, Eq. (26)] The Min-SNR weighting as written uses max(SNR, γ)/(SNR+1). Please verify this against the cited source (Hang et al., 2023), which is commonly stated with a min(SNR, γ) factor; either the formula is a variant or the citation needs clarification.
- [Sections 5.1 and 6] The teacher uses 25 DDIM steps during training while all reported results use 100 inference steps; please state explicitly whether this difference is intentional and whether the student is trained only on targets produced by the 25-step teacher.
- [Section 5.1, Eq. (13)] The embedding matrix E is introduced but not explicitly defined as a learnable parameter; Figure 4 indicates it is trained, so the text should state its trainability and where its gradients flow.
Circularity Check
No significant circularity: CLUDI's derivation is self-contained; the self-distillation loop is explicit and standard, and the claims are grounded in external benchmarks.
full rationale
The paper's teacher-student setup is self-referential by design: the teacher and student share weights with a stop-gradient, and the target embeddings in Eq. 13 are derived from the teacher's own softmax outputs. However, this is explicitly described as self-distillation in Section 5, following BYOL/SimSiam/DINO, and is not presented as an independent ground truth. The load-bearing claim of state-of-the-art clustering is evaluated against external ground-truth labels on seven benchmarks, so the empirical result is not forced by construction. Hyperparameters d, F^2, and lambda are tuned on a validation split and then fixed, which is standard practice rather than a fitted input being renamed a prediction. The absence of a non-diffusion teacher control weakens the causal attribution of the gains to diffusion, but that is an experimental-support limitation, not circularity. Self-citations (Pakman et al. 2020; Chelly et al. 2025; Ronen et al. 2022) appear only in related work and are not load-bearing. No uniqueness theorem or ansatz is imported from the authors' prior work; the adopted diffusion components are drawn from external references (Ho et al., Song et al., Li et al., Salimans & Ho, Hang et al.).
Assumptions & free parameters
free parameters (6)
- F^2 =
25.0
- d =
64 (K>=100), 32 (K<100)
- lambda =
50
- B =
not reported in main text
- tau, tau_col =
not reported
- Min-SNR gamma =
5
assumptions (4)
- domain assumption DINO ViT features pre-trained on ImageNet provide sufficiently informative representations for clustering the target datasets.
- domain assumption The 25-step DDIM sampling that maps pure noise z_T ~ N(0,F^2 I) to assignment embeddings z_0 produces diverse, useful cluster targets for the student.
- standard math The uniform-prior cross-entropy loss of Amrani et al. prevents collapsed solutions as claimed.
- standard math DDPM and DDIM forward and reverse processes (Equations 1-11) correctly model the conditional distribution of assignment embeddings.
Cite this review
Pith. "Pith review of Clustering via Self-Supervised Diffusion." pith.science (2026). https://pith.science/paper/KW4HLYOC
@misc{pith2026250704283,
author = {Pith},
title = {Pith review of: Clustering via Self-Supervised Diffusion},
year = {2026},
howpublished = {\url{https://pith.science/paper/KW4HLYOC}},
note = {Machine review of arXiv:2507.04283}
}
read the original abstract
Diffusion models, widely recognized for their success in generative tasks, have not yet been applied to clustering. We introduce Clustering via Diffusion (CLUDI), a self-supervised framework that combines the generative power of diffusion models with pre-trained Vision Transformer features to achieve robust and accurate clustering. CLUDI is trained via a teacher-student paradigm: the teacher uses stochastic diffusion-based sampling to produce diverse cluster assignments, which the student refines into stable predictions. This stochasticity acts as a novel data augmentation strategy, enabling CLUDI to uncover intricate structures in high-dimensional data. Extensive evaluations on challenging datasets demonstrate that CLUDI achieves state-of-the-art performance in unsupervised classification, setting new benchmarks in clustering robustness and adaptability to complex data distributions. Our code is available at https://github.com/BGU-CS-VIL/CLUDI.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[3]
H., Doucet, A., Strudel, R., Dyer, C., Durkan, C., et al
Dieleman, S., Sartran, L., Roshannai, A., Savinov, N., Ganin, Y ., Richemond, P. H., Doucet, A., Strudel, R., Dyer, C., Durkan, C., et al. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.15089,
-
[7]
Empowering diffusion models on the embedding space for text generation
Gao, Z., Guo, J., Tan, X., Zhu, Y ., Zhang, F., Bian, J., and Xu, L. Empowering diffusion models on the embedding space for text generation. In Proceedings of the 2024 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4664–4683,
work page 2024
-
[9]
Varia- tional deep embedding: An unsupervised and generative approach to clustering
Jiang, Z., Zheng, Y ., Tan, H., Tang, B., and Zhou, H. Varia- tional deep embedding: An unsupervised and generative approach to clustering. arXiv preprint arXiv:1611.05148,
-
[10]
Understanding diffusion models: A unified perspec- tive
Luo, C. Understanding diffusion models: A unified perspec- tive. arXiv preprint arXiv:2208.11970,
-
[11]
Step- by-step diffusion: An elementary tutorial
Nakkiran, P., Bradley, A., Zhou, H., and Advani, M. Step- by-step diffusion: An elementary tutorial. arXiv preprint arXiv:2406.08929,
-
[12]
and Zisserman, A
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722–729. IEEE,
2008
-
[14]
E., Diaconu, C.-D., Markou, S., Shysheya, A., Foong, A
Turner, R. E., Diaconu, C.-D., Markou, S., Shysheya, A., Foong, A. Y ., and Mlodozeniec, B. Denoising diffusion probabilistic models in six simple steps. arXiv preprint arXiv:2402.04384,
-
[15]
Wang, X., Chen, X., Du, S. S., and Tian, Y . Towards de- mystifying representation learning with non-contrastive self-supervision. arXiv preprint arXiv:2110.04947,
Show all 16 references
-
[16]
and Zhang, N
Zhou, X. and Zhang, N. L. Deep clustering with fea- tures from self-supervised pretraining. arXiv preprint arXiv:2207.13364,
-
[2011]
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee,
2009
-
[2014]
Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recogni- tion workshop, pp. 178–178. IEEE,
2004
-
[2019]
An image is worth 16x16 words: Trans- formers for image recognition at scale
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929,
2010 arXiv
-
[2020]
M., Vedaldi, A., Zisserman, A., and Jawahar, C
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition , pp. 3498–3505. IEEE,
2012
-
[2021]
Chan, S. H. Tutorial on diffusion models for imaging and vision. arXiv preprint arXiv:2403.18103,
-
[2022]
Distributed mcmc inference in dirichlet process mixture models using julia
Dinari, O., Yu, A., Freifeld, O., and Fisher, J. Distributed mcmc inference in dirichlet process mixture models using julia. In 2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), pp. 518–525. IEEE,
2019
-
[2024]
Diffuseq: Sequence to sequence text generation with diffusion mod- els
Gong, S., Li, M., Feng, J., Wu, Z., and Kong, L. Diffuseq: Sequence to sequence text generation with diffusion mod- els. arXiv preprint arXiv:2210.08933,
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.