Pith. sign in

REVIEW 2 major objections 5 minor 29 references

Supercm: Revisiting Clustering for Semi-Supervised Learning

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SuperCM trains a semi-supervised model with a differentiable clustering module whose class centroids are moving averages of labeled features, improving supervised baselines and boosting existing SSL methods when labels are scarce.

desk verdict A simple, plausible SSL regularizer whose centroid update equation doesn't match the stated mechanism, and whose experimental detail is too sparse to accept as submitted. read the letter →

arxiv 2506.23824 v1 pith:VNAET2BQ submitted 2025-06-30 cs.LG cs.CV

Durgesh Singh , Ahcene Boubekki , Robert Jenssen , Michael C. Kampffmeyer This is my paper
classification cs.LGcs.CV
keywords semi-supervisedlearningclusteringGaussianmixturemodeldifferentiablemodulemoving-averagecentroidsCIFAR-10consistencyregularization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that semi-supervised learning can be improved by explicitly enforcing the clustering assumption through a differentiable clustering module, without complex consistency-regularization or pseudo-label heuristics. The proposed method, SuperCM, uses a Gaussian-mixture-based one-layer autoencoder whose class centroids are moving averages of labeled features, trained end-to-end with a cross-entropy loss on labeled examples and the clustering loss on all data. On CIFAR-10, SuperCM raises the supervised-only baseline from 56.94 to 62.14 at 600 labels and from 78.65 to 82.26 at 4000 labels. It also works as an add-on regularizer, lifting Pseudo-Label by 4.14 percentage points and VAT by 6.8 percentage points at 600 labels. The paper argues that SuperCM is valuable precisely where labels are scarce and existing SSL methods need a well-separated feature space.

What carries the argument

The central object is the Clustering Module (CM), a one-layer autoencoder that approximately maximizes the Q-function of a Gaussian mixture model, with cluster responsibilities from a softmax encoder and centroids as decoder weights. SuperCM modifies CM by setting each class centroid to an online moving average of labeled features, $\mu_k \leftarrow \frac{t-1}{t}\mu_k + \frac{1}{t}\frac{1}{n^{(l)}}\sum_{i}\mathbf{1}(y_i^{(l)}=k)\mathbf{x}_i^{(l)}$, and removing the Dirichlet prior. This moving-average centroid update is what carries the argument: it stabilizes the cluster centers from noisy mini-batches, keeps them tied to class identity, and lets the whole network -- feature extractor, encoder, and decoder -- train jointly with gradient descent while clustering all data.

What would settle it

A concrete test: in the 600-label CIFAR-10 setup, construct training mini-batches so that every 10th batch contains no examples of one fixed class, while keeping total label counts identical, and compare per-class accuracy of that class against the standard full-batch schedule. Since Eq. (2) gives the absent class's centroid a decay factor of $\frac{t-1}{t}$ with zero update, a significant drop in that class's accuracy would show the moving-average centroid assumption is fragile under exactly the scarcity the method is designed for.

Watch

Extended reading notes

Core claim

The core discovery is that replacing the supervised classifier's centroids with a class-wise moving average of labeled features prevents the trivial solutions that arise when a clustering module is naively combined with a cross-entropy loss, and yields a simple end-to-end SSL method. In SuperCM, labeled and unlabeled data pass through a feature extractor; the labeled features update each class centroid by a 1/t moving average, and both sets then feed a differentiable Gaussian-mixture autoencoder whose loss is added to the CE loss. The paper reports statistically significant gains over the supervised baseline at 600 and 4000 labels, and significant gains for Pseudo-Label and VAT at 600 labels, with the gains shrinking at 4000 labels. The authors interpret the result as evidence that a well-separated clustered embedding is the key ingredient, and that SuperCM supplies this embedding to base SSL methods.

Load-bearing premise

The load-bearing premise is that the moving-average class centroids computed from labeled mini-batches stay close to the true class centers even when some classes are missing from a batch and even when the whole labeled set has only 250 images.

Editorial extensions

If this is right

  • At 600 labeled CIFAR-10 images, SuperCM improves the cross-entropy baseline by about 5.2 percentage points, and by about 3.6 points at 4000 labels.
  • As a regularizer, SuperCM lifts Pseudo-Label accuracy by 4.14 points and VAT accuracy by 6.8 points at 600 labels.
  • SuperCM alone improves over the supervised baseline at every label count tested from 250 to 4000, though the gain shrinks as labels become very scarce.
  • Because the clustering loss is differentiable, SuperCM can be appended to any gradient-based SSL method as an extra loss term, not only to Pseudo-Label and VAT.
  • At 4000 labels the regularizer does not significantly improve over the base models, indicating its main benefit is in the low-label regime.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the moving-average update leaves a centroid decaying toward zero whenever its class is absent from a labeled mini-batch; this is not analyzed in the paper, so a stress test with class-imbalanced batches is a natural next step.
  • Inference: the method could be extended to class-imbalanced and out-of-distribution settings, where the gap between supervised and SSL baselines is often larger than on CIFAR-10.
  • Inference: the paper does not compare against more recent consistency-based SSL methods, so whether clustering losses add value on top of them remains untested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes SuperCM, a semi-supervised learning (SSL) method that extends the differentiable clustering module (CM) of Boubekki et al. by updating class-wise centroids as a moving average of labeled features and combining the CM loss on labeled and unlabeled data with a cross-entropy loss on labeled data. The method is evaluated on CIFAR-10 with 600 and 4000 labels, both as a standalone replacement for the supervised classifier and as a regularizer added to Pseudo-Label and Virtual Adversarial Training. The abstract and Section 4.2 claim that SuperCM improves the supervised-only baseline and boosts the performance of existing SSL methods at low label counts, with a beta-sensitivity analysis in Section 5.3 and a code release to support reproducibility.

Significance. If the claims hold, SuperCM offers a conceptually simple, end-to-end differentiable alternative to consistency-regularization and entropy-minimization SSL methods, and its plug-in regularizer form could be useful for other gradient-based SSL algorithms. The paper has several strengths: it reports mean and standard deviation over five runs, uses validation-based model selection, follows the evaluation protocol of Oliver et al. (2018), includes a sensitivity analysis for the main hyperparameter beta, and releases code. The central mechanism, however, rests on the moving-average centroid update in Eq. (2), and this update as written does not compute a class-wise mean and is not fully specified. Because the main experimental gains are attributed to these centroids, the current description prevents verification of the paper's central claim and requires a major revision.

major comments (2)
  1. [Section 3.2, Eq. (2)] The update rule is not a class-wise moving average as claimed. Let c_k = sum_i 1(y_i^{(l)}=k) be the number of labeled samples of class k in the mini-batch. The second term in Eq. (2) equals (c_k / n^{(l)}) times the empirical mean of the class-k features in that batch. In expectation, the centroid therefore converges to approximately (E[c_k] / n^{(l)}) E[x | y=k], i.e., a class-frequency-scaled mean, rather than E[x | y=k]. Moreover, when a labeled mini-batch contains no samples of class k, the centroid is multiplied by (t-1)/t with no compensating observation, so a rarely seen class's centroid decays toward its initialization (or toward zero) over 500k iterations. The paper does not specify n^{(l)}, the initialization of mu_k, or whether a per-class moving-average counter is used. Since the CE head and the CM loss both depend on these centroids, the mechanism behind the Table 1 gains is either mis-specified in Eq. (2) or inconsistent with the released code. Please correct the equation or describe the implemented update and its initialization precisely.
  2. [Section 4.1] The dataset split is internally inconsistent: 50,000 training + 5,000 validation + 10,000 test images sums to 65,000, whereas CIFAR-10 contains 60,000 images. This suggests a misreporting of the split or of the number of labeled/unlabeled samples. In addition, the training protocol states only that the batch size is 100, but never gives the labeled batch size n^{(l)} and unlabeled batch size n^{(u)} that appear in Eq. (2) and Eq. (3). The tuned values of beta and delta used to produce Table 1 are also not reported, despite the statement that they are tuned over the validation set. These omissions are directly relevant to evaluating the behavior of Eq. (2) and to reproducing the reported accuracies.
minor comments (5)
  1. [Section 4.1] The word "prepossessing" should be "preprocessing".
  2. [Section 4.2] The phrases "by 5.2% and 3.6%", "by 4.14%", and "by6.8%" should be expressed as percentage points (e.g., 5.20 and 3.61 percentage points) to avoid confusion with relative improvements.
  3. [Section 5.1] The caption of Figure 3 should state the number of labels and the configuration used; the reader must rely on the main text to learn that this is the CIFAR-10 600-label setting.
  4. [Section 5.3] Figure 5 would benefit from error bars or multiple-seed results, and the set of beta values tested should be listed, since the current figure reports a single trajectory without indicating run-to-run variability.
  5. [Section 4.1] The initial learning rate is not stated; the text says only that the learning rate is decayed by a factor of 0.1 after 400000 iterations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the SSL gains are independently measured against external baselines, and the self-cited clustering module is a described component rather than the evidence.

full rationale

SuperCM's contribution is an end-to-end training scheme where centroids are updated as class-wise moving averages of labeled features (Eq. 2) and the loss combines CE with the CM loss (Eqs. 1 and 3). The paper's central claim, improvement over CE and over Pseudo-Label/VAT baselines, is an empirical result reported on CIFAR-10 test sets with standard deviations over five runs. No parameter is fitted to the test set and then renamed as a prediction; beta and delta are tuned on the validation set. The only self-citation is [13] (Boubekki, Kampffmeyer, Brefeld, and Jenssen), which supplies the CM loss. That citation is not used as evidence for the SSL result: the paper states the loss itself and validates the extension through new experiments. Thus the derivation chain does not reduce to its inputs. A separate, non-circular concern is that Eq. (2) is ambiguous: if n(l) is the labeled batch size, the centroid update is frequency-scaled rather than a true class mean, and absent classes decay; this is a correctness or notation issue to be resolved by the code, not a circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or data-driven entities. The only tuned parameters are the loss weights beta and delta, whose values are not reported. The correctness of the approach rests on the prior CM loss from the authors' own [13] and on the clustering assumption for CIFAR-10 features.

free parameters (2)
  • beta (CM loss weight) = tuned on validation set, value not reported
    Weight for the CM loss in Eq. (3); Section 4.1 says hyper-parameters are tuned over validation data, but exact values are not given.
  • delta (base SSL loss weight) = tuned on validation set, value not reported
    Weight for the SSL base model loss when SuperCM is used as a regularizer; Section 4.1, values not reported.
assumptions (3)
  • domain assumption The CM loss in Eq. (1) is a valid differentiable approximation to the GMM Q-function and is correctly used as a clustering objective.
    SuperCM directly uses LCM from [13] without re-deriving or validating it; the correctness of this surrogate is assumed.
  • domain assumption The clustering assumption holds for CIFAR-10: each class forms a compact, roughly Gaussian cluster in the learned 128-dimensional feature space.
    The method explicitly enforces the clustering assumption (Section 1) and its success depends on unlabeled data from each class being separable into one GMM component.
  • ad hoc to paper The moving-average centroid update in Eq. (2) reliably estimates class centers even when a mini-batch lacks some classes or labels are very scarce.
    The paper introduces this update to avoid trivial solutions (Section 3.2) but provides no theoretical analysis of the decay toward zero for absent classes or of its behavior under extreme label scarcity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Supercm: Revisiting Clustering for Semi-Supervised Learning." pith.science (2026). https://pith.science/paper/VNAET2BQ

@misc{pith2026250623824,
  author       = {Pith},
  title        = {Pith review of: Supercm: Revisiting Clustering for Semi-Supervised Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VNAET2BQ}},
  note         = {Machine review of arXiv:2506.23824}
}
read the original abstract

The development of semi-supervised learning (SSL) has in recent years largely focused on the development of new consistency regularization or entropy minimization approaches, often resulting in models with complex training strategies to obtain the desired results. In this work, we instead propose a novel approach that explicitly incorporates the underlying clustering assumption in SSL through extending a recently proposed differentiable clustering module. Leveraging annotated data to guide the cluster centroids results in a simple end-to-end trainable deep SSL approach. We demonstrate that the proposed model improves the performance over the supervised-only baseline and show that our framework can be used in conjunction with other SSL methods to further boost their performance.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 26 canonical work pages

  1. [1]

    Supercm: Revisiting Clustering for Semi-Supervised Learning

    INTRODUCTION Traditional deep learning has achieved state-of-the-art per- formance on various tasks at the cost of large-scale super- vised training data. However, it is difficult to obtain such a dataset in many applications, due to an expensive and time- consuming annotation process [1]. Several approaches have tackled this dependency by exploiting info...

    work page Pith review arXiv 2025

  2. [2]

    For a more extensive survey the interested reader is referred to [3, 12]

    RELA TED WORKS To set the stage for SuperCM, we discuss relevant deep SSL and clustering approaches. For a more extensive survey the interested reader is referred to [3, 12]. 2.1. Semi-Supervised Learning We first discuss representative methods based on consis- tency regularization and entropy minimization for the SSL task. Consistency regularization meth...

    work page

  3. [3]

    Clustering Module As the key building block of our SSL approach, we first de- scribe the CM introduced in [13]

    METHOD 3.1. Clustering Module As the key building block of our SSL approach, we first de- scribe the CM introduced in [13]. The model aims to maxi- mize a differentiable, rephrased version of the Q-function of a Gaussian mixture model. The loss function of the CM can be stated as follows: LCM = 1 N NX i=1 ||xxxi − ¯xxxi||2 + NX i=1 KX k=1 γik(1 − γik)||µµ...

    work page

  4. [4]

    We follow the recommendations of [17] for data pre- possessing, model architecture, and training protocol

    EXPERIMENTS In this section, we compare the performance of the SuperCM as an SSL model, and as a regularizer for other SSL base- 1Code available at https://github.com/Durgesh93/SuperCM.git lines. We follow the recommendations of [17] for data pre- possessing, model architecture, and training protocol. 4.1. Experimental Setting Data We use CIFAR-10 [18] fo...

    work page

  5. [5]

    ANALYSIS In this section, we analyze different aspects of the SuperCM and discuss the results. 5.1. Feature Visualization We show in Figure 3, the UMAP[22] representations of the feature space learned by the models trained with CE and SuperCM for the CIFAR-10 600 labels configuration. It is evident that SuperCM yields more separated and compact classes, w...

    work page 2000

  6. [6]

    Our training strategy benefits from the built-in cluster- ing capability of the CM module and does not rely on com- plex training schemes

    CONCLUSION In this paper, we present a simple end-to-end framework for SSL. Our training strategy benefits from the built-in cluster- ing capability of the CM module and does not rely on com- plex training schemes. Facilitated by the differentiable CM, our method can be integrated into any gradient based SSL method as an unsupervised regularizer, paving t...

    work page

  7. [7]

    There are many consistent expla- nations of unlabeled data: Why you should average,

    Ben Athiwaratkun, Marc Finzi, Pavel Izmailov, and An- drew Gordon Wilson, “There are many consistent expla- nations of unlabeled data: Why you should average,” in International Conference on Learning Representations (ICLR), 2019

    work page 2019

  8. [8]

    Preparing medical imaging data for machine learning,

    Martin J. Willemink, Wojciech A. Koszek, Cailin Hard- ell, Jie Wu, Dominik Fleischmann, Hugh Harvey, Les R. Folio, Ronald M. Summers, Daniel L. Rubin, and Matthew P. Lungren, “Preparing medical imaging data for machine learning,” Radiology, vol. 295, no. 1, pp. 4–15, Apr. 2020

    work page 2020

Show all 29 references
  1. [9]

    Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien, Eds., Semi-Supervised Learning, The MIT Press, Sep. 2006

    2006

  2. [10]

    A survey on deep semi-supervised learning,

    Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu, “A survey on deep semi-supervised learning,” Comput- ing Research Repository (CoRR) , vol. abs/2103.00550, 2021

    2021 arXiv

  3. [11]

    Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,

    Antti Tarvainen and Harri Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” in Advances in Neural Information Processing Systems (NeurIPS), 2017, vol. 2017-December, pp. 1196–1205

    2017

  4. [12]

    Temporal ensembling for semi-supervised learning,

    Samuli Laine and Timo Aila, “Temporal ensembling for semi-supervised learning,” 5th International Con- ference on Learning Representations (ICLR), 2017

    2017

  5. [13]

    Virtual adversarial training: A reg- ularization method for supervised and semi-supervised learning,

    Takeru Miyato, Shin ichi Maeda, Masanori Koyama, and Shin Ishii, “Virtual adversarial training: A reg- ularization method for supervised and semi-supervised learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, pp. 1979–1993, 2019

    1979

  6. [14]

    Unsu- pervised deep embedding for clustering analysis,

    Junyuan Xie, Ross Girshick, and Ali Farhadi, “Unsu- pervised deep embedding for clustering analysis,” in Proceedings of the 33rd International Conference on In- ternational Conference on Machine Learning (ICML) , 2016, vol. 48, pp. 478–487

    2016

  7. [15]

    Semi-supervised learning by entropy minimization,

    Yves Grandvalet and Yoshua Bengio, “Semi-supervised learning by entropy minimization,” in Advances in Neu- ral Information Processing Systems (NeurIPS), 2005

    2005

  8. [16]

    Pseudo-label : The simple and effi- cient semi-supervised learning method for deep neural networks,

    Dong-Hyun Lee, “Pseudo-label : The simple and effi- cient semi-supervised learning method for deep neural networks,” ICML 2013 Workshop : Challenges in Rep- resentation Learning (WREPL), 07 2013

    2013

  9. [17]

    Meta pseudo labels,

    Hieu Pham, Qizhe Xie, Zihang Dai, and Quoc V . Le, “Meta pseudo labels,” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 11552–11563, 2021

    2021

  10. [18]

    S4l: Self-supervised semi-supervised learning,

    Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer, “S4l: Self-supervised semi-supervised learning,” IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1476–1485, 2019

    2019

  11. [19]

    A Comprehensive Survey on Deep Clus- tering: Taxonomy, Challenges, and Future Directions,

    Sheng Zhou, Hongjia Xu, Zhuonan Zheng, Jiawei Chen, Zhao Li, Jiajun Bu, Jia Wu, Xin Wang, Wenwu Zhu, and Martin Ester, “A Comprehensive Survey on Deep Clus- tering: Taxonomy, Challenges, and Future Directions,” Jun. 2022

    2022

  12. [20]

    Joint optimization of an autoencoder for clustering and embedding,

    Ahc `ene Boubekki, Michael C. Kampffmeyer, Ulf Brefeld, and Robert Jenssen, “Joint optimization of an autoencoder for clustering and embedding,” Machine Learning, vol. 110, pp. 1901–1937, 2021

    1901

  13. [21]

    Aver- aging weights leads to wider optima and better general- ization,

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson, “Aver- aging weights leads to wider optima and better general- ization,” in Thirty-Fourth Conference on Uncertainty in Artificial Intelligence (UAI). 2018, pp. 876–885, AUAI Press

    2018

  14. [22]

    Deep clustering for unsupervised learning of visual features,

    Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze, “Deep clustering for unsupervised learning of visual features,” in European Conference on Computer Vision (ECCV), 2018

    2018

  15. [23]

    Prototypical contrastive learning of unsupervised rep- resentations,

    Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi, “Prototypical contrastive learning of unsupervised rep- resentations,” in International Conference on Learning Representations (ICLR), 2021

    2021

  16. [24]

    Realistic evaluation of deep semi-supervised learning algorithms,

    Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, and Ian J. Goodfellow, “Realistic evaluation of deep semi-supervised learning algorithms,” Advances in Neural Information Processing Systems , vol. 2018- December, pp. 3235–3246, apr 2018

    2018

  17. [25]

    Learning multiple layers of features from tiny images,

    Alex Krizhevsky, “Learning multiple layers of features from tiny images,” 2009

    2009

  18. [26]

    Wide resid- ual networks,

    Sergey Zagoruyko and Nikos Komodakis, “Wide resid- ual networks,” in Proceedings of the British Machine Vision Conference (BMVC). Sep. 2016, pp. 87.1–87.12, BMV A Press

    2016

  19. [27]

    Adam: A method for stochastic optimization,

    Diederik P. Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,” in 3rd Interna- tional Conference on Learning Representations (ICLR), San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015

    2015

  20. [29]

    UMAP: uniform manifold approximation and projection for dimension reduction,

    Leland McInnes and John Healy, “UMAP: uniform manifold approximation and projection for dimension reduction,” Computing Research Repository (CoRR) , vol. abs/1802.03426, 2018

    2018 arXiv

  21. [100]

    The hyper-parameters β and δ are tuned over the validation dataset

    The learning rate is decayed once with a factor of 0.1 after 400000 iterations. The hyper-parameters β and δ are tuned over the validation dataset. Baseline There are two types of baselines for our experi- ments. For the SSL setting, the baseline is supervised-only training wi...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.