REVIEW 5 major objections 4 minor 25 references
Random crops beat SAM masks for label-free fine-grained region scoring, and a soft negative margin picks the most discriminative regions.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 05:09 UTC pith:QHCSZ7HK
load-bearing objection A useful empirical comparison of CLIP region-scoring strategies, but the random-crop-vs-SAM headline is confounded by context leakage and the paper needs repeated runs and full hyperparameters before the claims are verifiable. the 5 major comments →
CLIP-Guided Label-Free Discriminative Region Scoring for Fine-Grained Classification
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central discovery is a controlled comparison showing that, in a frozen-CLIP pipeline with only a linear classifier trained, the choice of local region matters more than the semantic quality of the segmentation: random crops yield higher accuracy than SAM masks on all five datasets, with the largest gap on CUB-200-2011 (0.656 for soft negative margin with global pseudo-labels vs 0.454 for SAM). The paper also finds that soft negative margin, which subtracts a temperature-weighted average of all non-target class similarities from the target similarity, is the most effective scoring strategy among the label-free variants, and that pseudo-labels derived from the global image embedding are mo
What carries the argument
The central mechanism is a three-stage 'score, select, aggregate' pipeline built on frozen CLIP embeddings. Candidate regions are generated either by SAM masks or random crops; each region, the global image, and class text prompts are embedded with CLIP and L2-normalized; then a scoring function (cosine similarity, hard negative margin, soft negative margin, or entropy confidence) assigns each region a discriminability score, optionally using pseudo-labels from the global or local prediction; the top-k regions are combined with softmax weights and concatenated with the global embedding for a linear classifier. The soft negative margin score, in particular, is what carries the argument: it is
Load-bearing premise
The claim that random crops beat SAM masks rests on the two region generators being matched on region count and area range, but those settings are not reported, so the ranking could reflect uneven resource or hyperparameter choices instead of the region type.
What would settle it
Run the same frozen-CLIP pipeline with explicit, identical values for the number of candidate regions and the area range for both SAM masks and random crops, and average accuracy over at least three seeds; if SAM-based selection reaches or exceeds random-crop accuracy, or if the gap disappears under matching, the paper's central ranking is unsupported.
If this is right
- Fine-grained classification can be improved without training region proposal networks or part detectors: frozen CLIP features plus simple scoring suffice.
- Soft negative margin scoring outperforms plain cosine similarity, so region selection should penalize alignment with visually similar classes.
- Pseudo-labels from CLIP's global prediction are reliable enough to replace ground-truth labels for region selection, with only a small accuracy drop.
- SAM's semantic masks are not the best region source for CLIP-based scoring; random crops provide better context, so practitioners can skip SAM and reduce preprocessing time.
- Top-k selection is robust, but the optimal amount of aggregation differs by region type: SAM benefits from all regions while random crops prefer a smaller subset.
Where Pith is reading between the lines
- Because random crops beat SAM across every dataset in the paper, a testable extension is whether a simple sliding-window or multi-scale crop strategy can close the remaining gap to fully supervised part-based methods.
- If pseudo-label scoring is so close to true-label scoring, the same scoring framework might transfer to open-set or out-of-distribution settings where labels are unavailable, using CLIP's zero-shot predictions as the pseudo-label source.
- The soft negative margin's temperature parameter is not reported; sweeping it could reveal whether the advantage comes from the margin form or from temperature calibration, and would make the recipe more reproducible.
- The paper's explanation for random crops' superiority—context preservation and noise isolation—could be tested directly by measuring region-level CLIP embedding quality against human-rated region usefulness.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a training-free framework for fine-grained classification. Local candidate regions are generated either from SAM masks or random crops; frozen CLIP embeddings of the regions, the full image, and class text are combined through several scoring strategies (cosine similarity, hard/soft negative margin, entropy confidence) with either true labels or two pseudo-label variants. The top-k regions are softmax-aggregated, concatenated with the global CLIP feature, and classified by a lightweight linear head. Experiments on CUB-200-2011, Oxford Flowers, Oxford-IIIT Pets, Stanford Cars, and FGVC-Aircraft report that random-crop regions consistently outperform SAM masks, that soft negative margin scoring is strongest among label-free scores, and that pseudo-label scoring closely approximates true-label scoring.
Significance. If the central empirical claims were robust, the paper would offer a simple, label-free recipe for fine-grained classification: random crops plus CLIP global pseudo-labels and soft negative margin scoring, without training complex localization modules. The manuscript is strong in scope: five datasets, a systematic comparison of region generators, several scoring strategies, and a clear controlled classification head. The pseudo-label variants and the comparison of global vs. local pseudo-labels are also useful. However, the paper currently lacks the evidence needed to verify the headline claims: key hyperparameters are omitted, no repeated-seed statistics are reported, and the random-crop/SAM comparison is confounded by information content. The contribution is therefore better described as an interesting hypothesis than a validated result.
major comments (5)
- [§3.2, Table 1] The central comparison between random crops and SAM masks is confounded by information content. In Table 1, the All-Region Average baseline already shows a gap of 17.9 points on CUB (random crop 0.6250 vs. SAM 0.4461) without any scoring. Scoring strategies add relatively small increments on top. Random rectangular crops, after resizing, preserve a full rectangular slice of context, whereas SAM masks isolate an irregular segment and remove surrounding context. Since the specific values of the 'same maximum number of regions and the same area range' mentioned in §3.2 are never reported, the reader cannot tell whether the two conditions are actually matched. This confound undermines the headline claim that random-crop-based pseudo-label scoring is superior due to region selection quality.
- [§3.4, Tables 1 and 3] Key hyperparameters are missing: the softmax temperature τ and the entropy penalty λ in §3.4 are defined but their values are never given, and the number of regions L and the area range in §3.2 are never specified. More importantly, no seeds or error bars are reported. This matters because Table 1 and Table 3 report different accuracies for the same nominal configuration: Random Crop / Global / Soft Neg. Margin / k=5 gives CUB 0.6559 in Table 1 but 0.6488 in Table 3; Stanford Cars gives 0.8183 vs. 0.8243; FGVC-Aircraft gives 0.5017 vs. 0.5050. These discrepancies are comparable to the differences between scoring methods, so the claimed ranking cannot be assessed without repeated trials and standard deviations.
- [§4.2, §4.3] The best scoring strategy and the best k are selected on the same test sets used to report the final accuracies. For instance, the conclusion that 'Soft Negative Margin scoring achieves the strongest performance' is based on the highest cell in Table 1, which is the same data used to make the comparison. This is a selection-on-test circularity. A fixed validation split, or repeated train/test partitioning with confidence intervals, is needed to support 'consistently outperforms' claims.
- [Abstract, §4.2, Table 2] The claim that pseudo-label scoring 'closely approximates' true-label performance is contradicted by the paper's own Table 2. For SAM masks and Soft Negative Margin on CUB, the true-label accuracy is 0.7002 and the reported deltas are ΔG = −0.2459 and ΔL = −0.2483, i.e., pseudo-label accuracy is about 0.454, a 24.6-point drop. Large drops also appear for FGVC-Aircraft under SAM. The approximation is close for random crops on some datasets, but the blanket statement in the abstract is not supported. The claim should be restricted to the conditions where it actually holds, or the discrepancy should be explained.
- [§4.3, Table 4] The text states that 'when using all regions, the accuracy of random cropping significantly decreased, while the accuracy of SAM masking increased.' Table 4 does not support the SAM half of this sentence. For SAM Soft Neg. Margin (Global), accuracy goes from 0.4534 at k=7 to 0.4410 at ALL; for Entropy Confidence (Global), from 0.4540 at k=7 to 0.4478 at ALL. The SAM ALL rows are generally lower than the best k rows, not higher. This factual discrepancy needs correction.
minor comments (4)
- [General] There are formatting and language issues: 'ofCLIP , SAM' in Section 2, inconsistent use of 'I' and 'we' ('In this section, I will briefly describe...'), and missing spaces around commas. The paper should be proofread.
- [Figure 1] Figure 1 is not referenced in the main text. Please add a reference and an explanatory caption that ties it to the pipeline in Section 3.1.
- [§5] The statement that SAM requires 'over 10 times more preprocessing time' is not supported by any measurement or protocol. If timing is claimed, report the setup and the measured numbers; otherwise remove or qualify.
- [Tables 3 and 4] The 'ALL' rows in Tables 3 and 4 should be clearly marked as using all available regions, and the text should describe the trend row-by-row rather than making a global statement that is contradicted by Table 4 (see major comment 5).
Circularity Check
No significant circularity: the paper is an empirical comparison with well-defined scoring formulas and no fitted parameter renamed as a prediction.
full rationale
This paper does not contain a derivation chain in which an output is equivalent to its input by construction. The scoring strategies in §3.4 (region-global similarity, region-text similarity, hard/soft negative margin, entropy confidence) are explicitly defined functions of CLIP embeddings and text embeddings, and the pseudo-label variants in Eqs. (4)–(5) are explicitly defined as CLIP zero-shot predictions, not as ground-truth labels smuggled into the label-free setting. The central claims — soft negative margin is strongest, random crops outperform SAM, pseudo-label scoring approximates true-label scoring — are empirical findings from Tables 1–4, not consequences of the definitions. No parameter is fitted to a subset of the test data and then reported as a prediction; the linear classifier is trained on the standard train split with a fixed protocol, and the region scoring hyperparameters (τ, λ, k) are evaluated rather than fitted. The fact that pseudo-labels come from the same CLIP model used for feature extraction is a design choice explicitly discussed in the paper, and it does not make the comparison circular: the paper does not claim to derive pseudo-label accuracy from the scoring equations, and the true-label comparison in Table 2 provides an external reference. Possible concerns about test-set selection of the best configuration, unreported values of L/τ/λ, and run-to-run variance reflected in Table 1 vs Table 3 discrepancies are reproducibility or experimental-design issues, not circularity. There is no load-bearing self-citation, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation. The derivation, such as it is, is self-contained: features are extracted from frozen CLIP, regions are generated by SAM or random crops, scores are computed by the given formulas, and accuracy is measured on standard benchmarks. Therefore the circularity score is 0.
Axiom & Free-Parameter Ledger
free parameters (5)
- softmax temperature τ =
not reported
- entropy penalty λ =
not reported
- top-k =
k=5 default; varied 2-7 and ALL
- number of regions L and area range =
not reported
- classifier optimization hyperparameters =
lr=1e-3, wd=1e-4, 30 epochs, batch 32
axioms (4)
- domain assumption CLIP embeddings in a shared text-image space reliably rank fine-grained visual similarity for the target domain.
- domain assumption SAM masks and random crops are comparable because they use the same maximum number of regions and the same area range.
- domain assumption A pseudo-label from top-1 CLIP prediction is reliable enough to guide region selection.
- standard math A linear classifier trained with cross-entropy on frozen CLIP features is a fair evaluation protocol.
read the original abstract
Recent vision models such as CLIP and SAM enable training-free segmentation and semantic encoding for fine-grained classification. A common approach is to compare the representations of segmented image regions with the text prompt embeddings of the corresponding labels. However, it remains unclear how different local regions and CLIP-based scoring strategies affect the selection of discriminative evidence, especially when ground-truth labels are unavailable. In this paper, we propose a unified CLIP-guided label-free region scoring framework for fine-grained classification. The framework evaluates cosine similarity-based, margin-based, and entropy-based scoring strategies using both SAM-generated masks and random crops, and introduces two label-free pseudo-label variants based on global image embeddings and local region embeddings. We conduct experiments on five fine-grained classification datasets to systematically compare different region generation methods and scoring strategies. The results show that Soft Negative Margin scoring achieves the strongest performance, and pseudo-label scoring closely approximates true-label performance. Although SAM produces semantically meaningful masks, random-crop-based pseudo-label scoring consistently outperforms SAM-based scoring across all datasets, suggesting that random crops preserve surrounding information and provide more stable semantic context when pseudo-labels are noisy. In addition, SAM masks benefit from aggregating embeddings from all regions, whereas random crops tend to perform better with a smaller top-k subset. These findings provide new insights for fine-grained classification.
Figures
Reference graph
Works this paper leans on
-
[1]
O’ Connor, and Suzanne Little
Sidra Aleem, Fangyijie Wang, Mayug Maniparambil, Eric Arazo, Julia Dietlmeier, Kathleen Curran, Noel E. O’ Connor, and Suzanne Little. Test-time adaptation with salip: A cascade of sam and clip for zero-shot medical image segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 5184–5193, June 2024
2024
-
[2]
Towards fine-grained adaptation of clip via a self-trained alignment score
Eman Ali, Sathira Silva, Chetan Arora, and Muhammad Haris Khan. Towards fine-grained adaptation of clip via a self-trained alignment score. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 5875–5885, March 2026
2026
-
[3]
Scrap-sam- clip: Assembling foundation models for typical shape recognition in scrap classification and rating.Sensors (Basel, Switzerland), 26, 2026
Guangda Bao, Wenzhi Xia, Haichuan Wang, Zhiyou Liao, Ting Wu, and Yun Zhou. Scrap-sam- clip: Assembling foundation models for typical shape recognition in scrap classification and rating.Sensors (Basel, Switzerland), 26, 2026. URL https://api.semanticscholar.org/ CorpusID:284895998
2026
-
[4]
Fine-grained visual classification with high-temperature refinement and background suppression, 2023
Po-Yung Chou, Yu-Yung Kao, and Cheng-Hung Lin. Fine-grained visual classification with high-temperature refinement and background suppression, 2023. URL https://arxiv.org/ abs/2303.06442
Pith/arXiv arXiv 2023
-
[5]
Fine-grained image classification via combining vision and language
Xiangteng He and Yuxin Peng. Fine-grained image classification via combining vision and language. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017
2017
-
[6]
Segment anything
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023
2023
-
[7]
3d object representations for fine- grained categorization
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. InProceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, June 2013
2013
-
[8]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks.Commun. ACM, 60(6):84–90, May 2017. ISSN 0001-0782. doi: 10.1145/3065386. URLhttps://doi.org/10.1145/3065386
doi:10.1145/3065386 2017
-
[9]
Visual-text cross alignment: Refining the similarity score in vision-language models, 2024
Jinhao Li, Haopeng Li, Sarah Erfani, Lei Feng, James Bailey, and Feng Liu. Visual-text cross alignment: Refining the similarity score in vision-language models, 2024. URL https: //arxiv.org/abs/2406.02915
Pith/arXiv arXiv 2024
-
[10]
Clipsam: Clip and sam collaboration for zero-shot anomaly segmentation, 2024
Shengze Li, Jianjian Cao, Peng Ye, Yuhan Ding, Chongjun Tu, and Tao Chen. Clipsam: Clip and sam collaboration for zero-shot anomaly segmentation, 2024. URL https://arxiv.org/ abs/2401.12665
Pith/arXiv arXiv 2024
-
[11]
Fully convolutional networks for semantic segmentation, 2015
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation, 2015. URLhttps://arxiv.org/abs/1411.4038
Pith/arXiv arXiv 2015
-
[12]
Fine- grained visual classification of aircraft, 2013
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft, 2013. URLhttps://arxiv.org/abs/1306.5151
Pith/arXiv arXiv 2013
-
[13]
Automated flower classification over a large number of classes
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. InIndian Conference on Computer Vision, Graphics and Image Processing, Dec 2008
2008
-
[14]
Valse: A task-independent benchmark for vision and language models centered on linguistic phenomena
Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gatt. Valse: A task-independent benchmark for vision and language models centered on linguistic phenomena. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page 8253–8280. Association for Comput...
-
[15]
Parkhi, Andrea Vedaldi, Andrew Zisserman, and C
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V . Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012
2012
-
[16]
Learning transferable visual models from natural language supervision, 2021
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https://arxiv.org/abs/2103.00020
Pith/arXiv arXiv 2021
-
[17]
Caltech-ucsd birds 200
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 09 2010
2010
-
[18]
Hierarchical part matching for fine-grained visual categorization
Lingxi Xie, Qi Tian, Richang Hong, Shuicheng Yan, and Bo Zhang. Hierarchical part matching for fine-grained visual categorization. InProceedings of the IEEE international conference on computer vision, pages 1641–1648, 2013
2013
-
[19]
Learning to navigate for fine-grained classification
Ze Yang, Tiange Luo, Dong Wang, Zhiqiang Hu, Jun Gao, and Liwei Wang. Learning to navigate for fine-grained classification. InProceedings of the European Conference on Computer Vision (ECCV), September 2018
2018
-
[20]
When and why vision-language models behave like bags-of-words, and what to do about it?, 2023
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it?, 2023. URLhttps://arxiv.org/abs/2210.01936
Pith/arXiv arXiv 2023
-
[21]
Part-based r-cnns for fine-grained category detection
Ning Zhang, Jeff Donahue, Ross Girshick, and Trevor Darrell. Part-based r-cnns for fine-grained category detection. InEuropean conference on computer vision, pages 834–849. Springer, 2014
2014
-
[22]
Picking deep filter responses for fine-grained image recognition
Xiaopeng Zhang, Hongkai Xiong, Wengang Zhou, Weiyao Lin, and Qi Tian. Picking deep filter responses for fine-grained image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1134–1142, 2016
2016
-
[23]
Diversified visual attention networks for fine-grained object classification.IEEE Transactions on Multimedia, 19(6): 1245–1256, 2017
Bo Zhao, Xiao Wu, Jiashi Feng, Qiang Peng, and Shuicheng Yan. Diversified visual attention networks for fine-grained object classification.IEEE Transactions on Multimedia, 19(6): 1245–1256, 2017
2017
-
[24]
Learning multi-attention convolutional neural network for fine-grained image recognition
Heliang Zheng, Jianlong Fu, Tao Mei, and Jiebo Luo. Learning multi-attention convolutional neural network for fine-grained image recognition. InProceedings of the IEEE international conference on computer vision, pages 5209–5217, 2017
2017
-
[25]
Pg-sam: A fine-grained prior-guided sam framework for prompt-free medical image segmentation
Yiheng Zhong, Zihong Luo, Chengzhi Liu, Feilong Tang, Yingzhen Hu, Zelin Peng, Ming Hu, Jionglong Su, Zongyuan Ge, and Imran Razzak. Pg-sam: A fine-grained prior-guided sam framework for prompt-free medical image segmentation. In2025 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 3369–3376, 2025. doi: 10.1109/ BIBM66473.2025...
arXiv 2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.