REVIEW 2 major objections 5 minor 17 references
SpectralGCD claims that training a classifier directly on CLIP image-concept similarities, selected by spectral filtering and aligned by forward/reverse distillation, matches or exceeds prior multimodal methods on six category-discovery ben
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 22:13 UTC pith:DH7R3NID
load-bearing objection Strong numbers and clean ablations, but the concept dictionary likely leaks the novel class names, so the discovery claim needs a stripped-dictionary test before it convinces. the 2 major comments →
SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
SpectralGCD's central claim is that class identity can be read off a cross-modal representation z(x;C), the vector of cosine similarities between an image and concept names from a large agnostic dictionary, so that p(y|x) is well approximated by p(y|z(x;C)). Because the dictionary is noisy, Spectral Filtering computes the covariance of softmaxed teacher similarities over the dataset, takes its top eigenvectors, and sums squared eigenvector components weighted by eigenvalues to score and retain only concepts whose co-activation carries signal. The retained concepts define the student's representation, and forward plus reverse distillation from the same frozen teacher keeps the student's simil
What carries the argument
The central object is the cross-modal representation z_{theta,phi}(x;C), a vector of CLIP cosine similarities between an image and every concept in a task-agnostic dictionary, treated as a mixture over semantics rather than as separate visual and textual features. Two mechanisms carry the argument. Spectral Filtering forms a cross-modal covariance matrix on softmax-normalized teacher similarities, keeps the eigenspace that explains a threshold of variance, and computes a concept importance score s_j = sum_i lambda_i v_{ij}^2 to select the dictionary subset that retains class-discriminative concepts. Forward and reverse knowledge distillation losses then transfer the teacher's concept-distrib
Load-bearing premise
The fixed task-agnostic concept dictionary plus the frozen teacher's similarity structure must actually capture class identity; if the dictionary lacks the discriminating concepts, or the leading covariance eigenvectors are driven by dataset-specific background bias rather than class semantics, the representation loses the information the classifier needs.
What would settle it
Take a dataset whose classes differ along concepts missing from a chosen dictionary while sharing strong visual backgrounds; if SpectralGCD's accuracy collapses to near the unimodal baseline, or if the top selected concepts are mostly background words common to all classes, the sufficiency claim is falsified. More directly, one could test conditional independence: whether a classifier on z(x;C) matches one on x as the dictionary is varied.
If this is right
- A classifier trained on filtered concept similarities should keep performing on novel classes, since it is not anchored to spurious visual cues that only separate known classes.
- Because the text encoder is frozen and teacher similarities are precomputed once, the training overhead stays close to unimodal levels even though the method is multimodal.
- The discovery pipeline can be re-run cheaply as new unlabeled data arrive, which the paper argues is the realistic deployment setting.
- On several benchmarks the distilled student surpasses the zero-shot accuracy of its own teacher, suggesting the method extracts more from the concept space than the teacher does at inference time.
- Using an unfiltered dictionary hurts performance on fine-grained datasets, so concept selection is a direct accuracy lever, not just a computational one.
Where Pith is reading between the lines
- If the sufficiency assumption p(y|x)=p(y|z(x;C)) holds approximately, then the bottleneck shifts from the classifier to the dictionary's concept coverage; a natural extension is to make the concept set image-adaptive rather than fixed.
- The same spectral-filtering recipe may transfer to other label-scarce settings, such as open-set recognition or continual discovery, whenever a frozen vision-language teacher and a large concept list are available.
- Covariance-based selection is an unsupervised proxy for class discrimination; a direct test would compare selected concepts against a supervised importance ranking and check whether the two agree on fine-grained datasets.
- The reported robustness to generic linguistic dictionaries suggests the method could scale to 100k+ concepts with low-rank covariance approximations, trading a small accuracy dip for much broader coverage.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SpectralGCD, a multimodal generalized category discovery (GCD) method that replaces separate visual and textual streams with a unified cross-modal representation: CLIP image–concept cosine similarities over a large concept dictionary. A spectral filtering stage selects task-relevant concepts using the eigendecomposition of the covariance of teacher softmaxed similarities; training then uses forward and reverse distillation from a frozen teacher plus the standard SimGCD contrastive/parametric losses. Experiments on six benchmarks (CUB, Stanford Cars, FGVC-Aircraft, CIFAR-10/100, ImageNet-100) report state-of-the-art or competitive accuracy at lower training cost than previous multimodal methods, with ablations over thresholds, dictionaries, teachers, student capacity, and Old/New imbalance.
Significance. If the reported results hold, the paper makes a useful practical contribution: it shows that a parametric classifier trained on CLIP cross-modal similarities can match or exceed unimodal and prior multimodal GCD methods at a fraction of the cost, and the public code plus extensive ablations are a strength. The spectral filtering method and distillation combination are well motivated and the efficiency analysis is informative. However, the central claim that the method performs 'category discovery' using only Old class names is threatened by evidence that the concept dictionary contains exact class names of novel categories; this must be resolved before the empirical claim can be interpreted as GCD rather than class-name retrieval.
major comments (2)
- [§5.1, Appendix I (Table 17)] The dictionary is described as 'task-agnostic' (Abstract, §4.1) and the GCD setup assumes knowledge of Old class names only (§3). However, the default Tags dictionary is 'a list of ~22K concepts arising from several benchmark datasets' (§5.1), and the WordNet run in Table 17 selects concepts such as 'Tern', 'Warbler', 'Woodpecker', 'Vesper Sparrow', and 'Indigo Bunting' — exact CUB class names. Since z(x;C) in Eq. (4) includes similarity to these concepts, a classifier trained on z can identify novel classes by retrieving their names from the dictionary. This converts the task into semi-supervised class-name retrieval and would invalidate the claim of discovering novel categories without their names. I request a systematic check of dictionary overlap with Old/New class names for Tags, OpenImages-v7, and WordNet, and an ablation in which all Old/New class names (and near-synonyms) are rem
- [§4.1] The key modeling assumption p(y|x)=p(y|z(x;C)) (Section 4.1) is asserted rather than tested. It is especially important because the dictionary is finite and Table 3 shows that the choice of dictionary changes All accuracy by as much as 3–10 points (e.g., Stanford Cars, OpenImages-v7 vs Tags). If the class-name-stripped experiment from the previous comment shows a large drop, it would indicate that the representation is not sufficient in the intended sense and that the reported gains rely on explicit class-name cues. I would like to see the sufficiency assumption examined by measuring performance against oracle concepts (class names) and against a concept set with all class-name entries removed.
minor comments (5)
- [Eq. (4) and throughout] Notation is inconsistent: zθ,ϕ(xi; C) is defined but later written as z(x;C), and the dictionary is sometimes C and sometimes ¯C. Please unify.
- [Table 1] Main-table results are reported as point estimates without standard deviations; deviations appear only in Appendix Table 7. Please include them in the main table or explicitly point the reader to the appendix.
- [Figure 4] The figure's time comparison would be clearer if the preparation phase for each method were labeled directly on the bars (Spectral Filtering, inversion network, image-to-text assignment), rather than only in the caption.
- [Eq. (10)] The term 'reverse distillation' is nonstandard: L_rd = -Σ σ(z_i) log σ(z*_i) is a cross-entropy of the teacher under the student, not the reverse KL divergence. Please clarify the relationship or rename it (e.g., 'negative cross-entropy').
- [Appendix H] In the 80% New imbalance setting, TextGCD outperforms SpectralGCD. This is a meaningful boundary on the method's SOTA claim and deserves a sentence in the main paper, not only in the appendix.
Circularity Check
Concept dictionary appears to contain benchmark class names, making novel-class 'discovery' partially reduce to CLIP class-name retrieval; otherwise no fitted tautology.
specific steps
-
other
[Section 5.1, Eq. (4), Eqs. (8)-(9), Appendix I Table 17]
"When not mentioned otherwise, we use the Tags concept dictionary from TextGCD as default textual information source C. ... the Tags used by TextGCD, a list of ~22K concepts arising from several benchmark datasets. ... Table 17: Top-ranked discriminative concepts ... 1 Tern 2 Warbler 3 Woodpecker 4 Vesper Sparrow 5 Indigo Bunting ..."
Eq. (4) defines the cross-modal representation as CLIP similarities to every dictionary concept, and Eqs. (8)-(9) retain only the concepts with highest importance. The paper's own Table 17 shows that on CUB/WordNet these top concepts are ground-truth bird class names (e.g., Vesper Sparrow, Indigo Bunting). If the 'task-agnostic' dictionary is not stripped of benchmark label names, then z(x;C) already contains direct CLIP similarities to the true novel-class names. Training the classifier on W^T z and reporting New-class accuracy then reduces, at least partially, to zero-shot class-name retrieval rather than unsupervised discovery from unlabeled data. The paper reports no class-name-stripped dictionary experiment, so the central GCD claim is partly built into the input representation.
full rationale
The training and evaluation loop is otherwise not circular: accuracy is measured against ground-truth labels on external benchmarks, and the losses in Eqs. (5), (10), and (11) do not encode the ground-truth New labels. The KD Spearman result in Table 2 is an expected consequence of minimizing forward/reverse KL divergence and is reported as alignment, not as an independent prediction of accuracy. No load-bearing self-citation chain appears: citations to Mistretta et al. merely motivate cross-modal alignment and reverse KD and do not supply a uniqueness theorem or forbid alternatives. The one serious issue is dictionary leakage. Because Eq. (4) feeds every dictionary concept's CLIP similarity into the classifier, and Spectral Filtering preferentially selects class-name concepts in Table 17, some of the reported New-class accuracy is attributable to the answer key being present in the input. This is a construction-level circularity for the discovery claim, though the method still requires clustering and parametric training, so it is partial rather than total.
Axiom & Free-Parameter Ledger
free parameters (4)
- beta_e =
0.95
- beta_c =
0.99
- lambda =
0.35
- CLIP logit temperature tau_t=tau_s =
0.01
axioms (4)
- domain assumption p(y|x)=p(y|z(x;C)) for the concept mixture z(x;C) (Section 4.1).
- domain assumption Softmax normalization of teacher similarities amplifies class-discriminative concepts and suppresses background concepts (Section 4.2).
- domain assumption Dominant eigenvectors of the cross-modal covariance matrix capture task-relevant concept co-variation while small eigenvalues are noise (Section 4.2).
- domain assumption The fixed dictionary C and frozen teacher are sufficiently broad and accurate for the task (Sections 4.1, 4.3, Limitations).
Cite this review
Pith. "Pith review of SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery." pith.science (2026). https://pith.science/paper/DH7R3NID
@misc{pith2026260217395,
author = {Pith},
title = {Pith review of: SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery},
year = {2026},
howpublished = {\url{https://pith.science/paper/DH7R3NID}},
note = {Machine review of arXiv:2602.17395}
}
read the original abstract
Generalized Category Discovery (GCD) aims to identify novel categories in unlabeled data while leveraging a small labeled subset of known classes. Training a parametric classifier solely on image features often leads to overfitting to old classes, and recent multimodal approaches improve performance by incorporating textual information. However, they treat modalities independently and incur high computational cost. We propose SpectralGCD, an efficient and effective multimodal approach to GCD that uses CLIP cross-modal image-concept similarities as a unified cross-modal representation. Each image is expressed as a mixture over semantic concepts from a large task-agnostic dictionary, which anchors learning to explicit semantics and reduces reliance on spurious visual cues. To maintain the semantic quality of representations learned by an efficient student, we introduce Spectral Filtering which exploits a cross-modal covariance matrix over the softmaxed similarities measured by a strong teacher model to automatically retain only relevant concepts from the dictionary. Forward and reverse knowledge distillation from the same teacher ensures that the cross-modal representations of the student remain both semantically sufficient and well-aligned. Across six benchmarks, SpectralGCD delivers accuracy comparable to or significantly superior to state-of-the-art methods at a fraction of the computational cost. The code is publicly available at: https://github.com/miccunifi/SpectralGCD.
Figures
Reference graph
Works this paper leans on
-
[1]
Emergence of invariance and disentanglement in deep rep- resentations.J
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep rep- resentations.J. Mach. Learn. Res., 19(1):1947–1980, January
1947
-
[5]
Fine-grained visual classification of aircraft.arXiv preprint arXiv:1306.5151,
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft.arXiv preprint arXiv:1306.5151,
-
[9]
ISBN 978-3-031-72897-6
Springer Nature Switzerland. ISBN 978-3-031-72897-6. Grzegorz Rype ´s´c, Daniel Marczak, Sebastian Cygert, Tomasz Trzci ´nski, and Bartłomiej Twar- dowski. Category adaptation meets projected distillation in generalized continual category discov- ery. In Aleˇs Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and G¨ul Varol (eds.),Co...
2024
-
[10]
Luyao Tang, Kunze Huang, Chaoqi Chen, and Cheng Chen. Generalized category discovery via token manifold capacity learning.arXiv preprint arXiv:2505.14044, 2025a. Luyao Tang, Kunze Huang, Chaoqi Chen, Yuxuan Yuan, Chenxin Li, Xiaotong Tu, Xinghao Ding, and Yue Huang. Dissecting generalized category discovery: Multiplex consensus under self- deconstruction....
-
[11]
The caltech-ucsd birds-200-2011 dataset
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset
2011
-
[13]
Textual knowledge matters: Cross-modality co-teaching for generalized visual class discovery
Haiyang Zheng, Nan Pu, Wenjing Li, Nicu Sebe, and Zhun Zhong. Textual knowledge matters: Cross-modality co-teaching for generalized visual class discovery. In Aleˇs Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and G¨ul Varol (eds.),Computer Vision – ECCV 2024, pp. 41–58, Cham,
2024
-
[14]
ISBN 978-3-031-72943-0
Springer Nature Switzerland. ISBN 978-3-031-72943-0. 14 Published as a conference paper at ICLR 2026 APPENDIXA IMPLEMENTATIONDETAILS We use the CLIP with image encoder ViT-B/16 trained by OpenAI (Radford et al., 2021), to ensure fairness with the rest of the multimodal methods in GCD. We train only the last transformer block of the visual encoder. The tex...
2026
-
[17]
20 Published as a conference paper at ICLR 2026 Table 17: Top-ranked discriminative concepts and common background concepts (with relative ranks) obtained using Spectral Filtering on the CUB dataset with a WordNet dictionary. Top Discriminative Concepts Common Background Concepts Rank Concept Rank Concept 1 Tern 354 Beak 2 Warbler 9,950 Wing 3 Woodpecker ...
2026
-
[2009]
Benoit Dufumier, Javiera Castillo Navarro, Devis Tuia, and Jean-Philippe Thiran
doi: 10.1109/CVPR.2009.5206848. Benoit Dufumier, Javiera Castillo Navarro, Devis Tuia, and Jean-Philippe Thiran. What to align in multimodal contrastive learning? InThe Thirteenth International Conference on Learning Representations. Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data filtering netwo...
arXiv 2009
-
[2011]
Get: Unlocking the multi-modal potential of clip for generalized category discovery
Enguang Wang, Zhimao Peng, Zhengyuan Xie, Fei Yang, Xialei Liu, and Ming-Ming Cheng. Get: Unlocking the multi-modal potential of clip for generalized category discovery. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20296– 20306, June 2025a. Guanghui Wang, Zhiyong Yang, Zitai Wang, Shi Wang, Qianqian Xu, a...
2026
-
[2013]
Marco Mistretta, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Andrew D. Bagdanov. Cross the gap: Exposing the intra-modal misalignment in CLIP via modality inversion. InThe Thirteenth International Conference on Learning Representations, 2025a. Marco Mistretta, Alberto Baldrati, Marco Bertini, and Andrew D. Bagdanov. Improving zero-shot general...
Pith/arXiv arXiv 2024
-
[2017]
Both contain a similar number of entries
which spans thousands of cat- egories. Both contain a similar number of entries. Here, we report TextGCD results using only these dictionaries, excluding the auxiliary LLM-generatedAttributesdictionary, to ensure that both meth- ods operate under the same textual constraints. Results on CUB are consistent with those observed on the other datasets:Tagsprov...
2026
-
[2020]
Concept bottleneck models
11 Published as a conference paper at ICLR 2026 Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Hal Daum´e III and Aarti Singh (eds.),Proceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Research, pp. 5338–5348....
2026
-
[2021]
Learn to categorize or categorize to learn? self- coding for generalized category discovery
12 Published as a conference paper at ICLR 2026 Sarah Rastegar, Hazel Doughty, and Cees Snoek. Learn to categorize or categorize to learn? self- coding for generalized category discovery. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.),Advances in Neural Information Processing Systems, volume 36, pp. 72794–72818. Curran Assoc...
2026
-
[2023]
Asano, Hazel Doughty, and Cees G
Sarah Rastegar, Mohammadreza Salehi, Yuki M. Asano, Hazel Doughty, and Cees G. M. Snoek. Selex: Self-expertise in fine-grained generalized category discovery. In Ale ˇs Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and G ¨ul Varol (eds.),Computer Vision – ECCV 2024, pp. 440–458, Cham,
2024
-
[2025]
Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automatically discovering and learning new visual categories with ranking statistics.arXiv preprint arXiv:2002.05714,
Pith/arXiv arXiv 2002
-
[4090]
In Table 8 we include additional dataset details, regarding the samples count and splits used in our evaluation
Reported performance is averaged on three different seeds (see Table 7 for standard deviations). In Table 8 we include additional dataset details, regarding the samples count and splits used in our evaluation. APPENDIXB BALANCE BETWEENOLD ANDNEWPERFORMANCE INGCD In this section, we discuss how unimodal approaches suffer from theOld/Newtradeoff more severe...
2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.