REVIEW 4 major objections 6 minor 34 references
Coarse2Fine: A Two-stage Training Method for Fine-grained Visual Classification
T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Coarse2Fine, a two-stage training method that feeds attention-highlighted images back through a second classifier, surpasses the previous state of the art on five fine-grained visual classification benchmarks.
desk verdict Plausible WS-DAN extension with a learnable up-sampler, but the reported SOTA gains are not pinned to the proposed mechanism; the paper needs an ablation and a fix for a dataset table error before I would trust the claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the differentiable feedback path from attention maps to raw image, implemented by a deconvolutional up-sampler inserted between a Bilinear Attention Pooling (BAP) block and a second classifier. BAP forms one feature vector per attention map by pooling the element-wise product of feature maps and attention maps; each vector is meant to correspond to an object part such as a beak or headlight. In Coarse2Fine, one attention map is up-sampled into a mask, element-wise multiplied with the input, and passed to the fine-grained network, so the fine-grained cross-entropy gradient reaches the attention weights through real pixel values. A center loss on the part-feature rows and orthogonal SVD-based initialization of the attention convolution are auxiliary mechanisms that encourage each attention map to specialize on a distinct part. The up-sampler is pre-trained on synthetic image and up-sampled pairs so it starts as a reasonable interpolation before end-to-end training.
What would settle it
Train the identical Coarse2Fine pipeline on CUB-200-2011 with a fixed bilinear up-sampler in place of the learned deconvolutional one, keeping all other settings the same; if top-1 accuracy stays at or above 89.5%, the claim that learning an inverse mapping to the input drives the improvement is falsified. A second check would feed the fine-grained network the original, unmasked image; matching accuracy would mean the highlighted-image path is not the source of the gain.
Extended reading notes
Core claim
The paper's central claim is that creating a differentiable path from the attended feature maps back to the input image improves fine-grained attention learning. Coarse2Fine cascades two classifiers that share a backbone: a coarse network produces bilinear attention maps, a deconvolutional up-sampler turns one randomly chosen attention map into a soft mask, and the mask is multiplied with the original image to feed a fine-grained classifier. Because the fine-grained loss is a function of the attention maps through this path, the attention maps are trained not only to separate classes at the feature level but also to isolate regions that let a second network make a fine distinction. The paper reports state-of-the-art top-1 accuracy on CUB-200-2011 (89.5%), FGVC-Aircraft (93.4%), Stanford Cars (94.7%), Stanford Dogs (93.0%), and iNaturalist 2017 (70.5%), and lower weakly supervised localization error than prior methods. It also argues that initializing the 1x1-convolution attention weights from the SVD of those weights, so attention maps start nearly orthogonal, contributes to the gain.
Load-bearing premise
The load-bearing premise is that multiplying an up-sampled attention map with the raw image yields a highlighted image whose fine-grained classification error trains the attention maps to find more discriminative parts; the paper asserts this correlation between feature-map activation and input location but does not isolate it with an ablation, so the accuracy gains could in principle come from the extra classifier, the up-sampler capacity, or the auxiliary losses rather than from the attention feedback itself.
Editorial extensions
If this is right
- On the five tested benchmarks, Coarse2Fine with orthogonal initialization reports top-1 accuracy above every listed prior model: 89.5% on CUB-200-2011, 93.4% on FGVC-Aircraft, 94.7% on Stanford Cars, 93.0% on Stanford Dogs, and 70.5% on iNaturalist 2017.
- The same attention maps improve weakly supervised object localization, with localization error falling to 17.2% on CUB-200-2011 and 17.9% on Stanford Dogs, below the prior WS-DAN numbers.
- The full method adds only 6144 parameters over the InceptionV3 backbone, so the attention feedback path is cheap in model size.
- Averaging the coarse and fine prediction heads gives the best accuracy, and the coarse head alone already beats a plain CNN, giving a practical inference trade-off.
- The attention module transfers to a face-attribute task: on CelebA, adding the attention layers lifts average accuracy over the selected six attributes by about 2.1 percentage points.
Reading between the lines
- Because the fine-grained network is trained only on highlighted images, the method can be viewed as a learned, task-driven augmentation; an extension would be to apply the same feedback loop to one-stage detectors or part-segmentation heads, where the mask supervision could come from dense labels instead of a second classifier.
- The random selection of one attention map per iteration resembles dropout over object parts, which suggests a testable variant: always sum all attention maps into the mask and measure whether specializing on a single part is what forces sharper attention.
- The deconvolutional up-sampler is pre-trained on synthetic interpolation pairs, so an alternative explanation of the gain is that a warm-started, higher-capacity up-sampler regularizes the attention maps; comparing the learned up-sampler against a fixed bilinear one within this exact pipeline would settle which part of the path matters.
- The improvement on the face-attribute task, where categories are not visually fine-grained in the usual sense, hints that the feedback path may help any task in which local appearance differences matter, not only species-level classification.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Coarse2Fine, a two-stage training method for fine-grained visual classification built on bilinear attention pooling. In the coarse stage, an InceptionV3 backbone produces attention maps and a coarse classification prediction. One randomly selected attention map is passed through a deconvolutional up-sampler and multiplied with the raw input to create a highlighted image; this image is classified by a second, fine-grained network. The fine-grained classification loss backpropagates through the up-sampler to the attention maps, creating a differentiable feedback path from the input space to the attended features. The method also includes orthogonal initialization of the 1x1 convolution that produces attention maps and a center loss applied to the pooled part features. The authors report state-of-the-art top-1 accuracy on CUB-200-2011, FGVC-Aircraft, Stanford Cars, Stanford Dogs, and iNaturalist 2017, improved weakly supervised object localization, and an attribute-classification experiment on CelebA.
Significance. If the central claim holds, the paper offers a simple, parameter-efficient way to improve attention learning in fine-grained recognition: a learnable deconvolutional up-sampler that lets fine-grained classification error supervise the attention maps directly. The reported gains over the strong WS-DAN baseline are modest but consistent across five datasets (0.1 to 1.6 accuracy points), and the orthogonal-initialization ablation (Table 8) is a useful, controlled comparison. However, the evidence for the proposed mechanism is incomplete: the experiments do not isolate the differentiable up-sampler from the many other changes relative to WS-DAN, and the reported iNaturalist dataset statistics are inconsistent with the public dataset. With the missing ablations and corrected dataset description, the paper could be a solid empirical contribution; as presented, the attribution of the gains to the Coarse2Fine mechanism is not established.
major comments (4)
- [Coarse2Fine Training; Tables 2-6] The central claim that the deconvolutional up-sampler creates a differentiable feedback path that improves attention learning is not tested by any ablation. Compared with WS-DAN, Coarse2Fine changes several factors at once: the learnable up-sampler, the random selection of one attention map per training step, the averaging of coarse and fine predictions at inference, the sharing of the backbone, the application of center loss to both feature sets, and orthogonal initialization. The only ablation provided (Table 8) addresses orthogonal initialization, not the feedback path. In particular, the paper asserts in the Approach section that a non-trainable up-sampler (e.g., bilinear) should cause accuracy degradation, citing Hu et al. 2019, but this is never tested within the proposed architecture. A reader cannot tell whether the reported improvements over WS-DAN come from the differentiable up-sampler, from two-head averaging, from random attention selection, or from orthogonal initialization. I request an ablation that replaces the learned deconvolutional up-sampler with a fixed bilinear or nearest-neighbor up-sampler, and an ablation that removes the fine-grained feedback path (for example, training the fine network on original or randomly masked images). Without these controls, the central claim is not supported.
- [Table 1] The iNaturalist 2017 row in Table 1 lists Training=12000 and Test=8580, which are exactly the Stanford Dogs numbers. The public iNaturalist 2017 dataset has a training set of roughly 579,000 images and a test set of roughly 96,000 images. This is not a minor typo, because it affects the interpretation of the 70.5% result in Table 6: if the model was trained on a 12,000-image subset, the comparison with WS-DAN (68.9%) may not be on the same training data. The authors must correct the dataset statistics and explicitly state which training and test splits were used for the iNaturalist experiments, including any random subsetting if that was done.
- [Tables 2-8] All accuracy numbers are reported as single runs without error bars or multiple-seed statistics. Several of the headline improvements over WS-DAN are very small (e.g., 0.1 points on CUB-200-2011 in Table 2 and 0.2 points on Stanford Cars in Table 4), and such differences are typically within run-to-run variance for CNN training. To support a state-of-the-art claim, the authors should provide the mean and standard deviation over at least three runs, or otherwise demonstrate that the differences are statistically meaningful. This concern applies equally to the weakly supervised localization results in Table 7.
- [Approach, Inference paragraph] The paper states that averaging the coarse and fine predictions gives the best accuracy, but no experiment or table is shown to support this choice. Since inference-time averaging is one of the differences from WS-DAN and directly affects the reported numbers, this is a load-bearing design decision that needs an ablation. Without it, the reader cannot separate the contribution of the two-head averaging from the contribution of the proposed feedback mechanism.
minor comments (6)
- [Discussion] The text refers to 'Figure 9' when describing the CelebA results, but the manuscript contains no Figure 9; the figure numbering is incomplete or the reference is wrong.
- [Coarse2Fine Training, 'only 6144 extra parameters'] The claim that the final model contains only 6144 extra parameters appears to count only the attention 1x1 convolution (8 x 768 = 6144), but the deconvolutional up-sampler also adds parameters. Please clarify the total parameter count attributable to the proposed components.
- [Setup, 'Pre-training of the deconvolutional network'] The pre-training procedure is underspecified: what images and resolutions were used for the 'synthetic dataset of pair of 60000 input and its up-sampled version,' and what exactly is the up-sampled target? It is also unclear whether the up-sampler remains trainable after pre-training. Please provide enough detail to reproduce this step.
- [Algorithm 1, line 12] The notation 'CoarseFeats+FineFeats' in the center-loss line is ambiguous: it could mean addition, concatenation, or separate losses on both feature sets. Please clarify how the center loss is applied to the two feature matrices.
- [Orthogonal initialization] The description of orthogonal initialization is mathematically unclear. If the attention weights are an 8x768 matrix X, the SVD gives X = U Sigma V^T with V of size 768x768, so 'taking V as the initial weights' cannot directly initialize a layer with 8 output channels. Please specify the exact initialization procedure (e.g., initializing with the first singular vectors or with a reshaped orthogonal matrix).
- [Object Localization Results] The paper says the localization evaluation follows Zhang et al. 2018, but it does not describe how the predicted bounding box is extracted from the attention maps (e.g., thresholding and connected components). Adding this detail would improve reproducibility.
Circularity Check
No significant circularity: the paper's claims are empirical benchmark accuracies with no derivation chain that reduces a predicted quantity to a fitted input.
full rationale
Coarse2Fine makes no derivation-like claims that could collapse by construction. Its central results are top-1 classification accuracies on five external fine-grained benchmarks and weakly supervised localization errors, all measured against published baselines such as WS-DAN. The method itself is a training procedure: attention maps are up-sampled by a deconvolutional network, multiplied with the raw image, and fed to a fine-grained classifier, with gradients flowing back to the attention maps. Nothing in the paper defines the reported accuracy in terms of a fitted parameter or a self-cited uniqueness theorem. Hyperparameters such as the number of attention maps (8), the center-loss weight (1.0), and the orthogonal initialization are chosen by hand or by standard practice, not fitted to reproduce the target accuracies. The only ablation reported (orthogonal initialization, Table 8) isolates a separate component and shows small gains of 0.2-0.8 points, which does not make the headline results circular. The assertion that a non-trainable bilinear up-sampler would degrade accuracy cites the external WS-DAN paper (Hu et al. 2019) and is not tested within this architecture; that is a legitimate ablation-omission or correctness concern, not a circularity concern. The apparent iNaturalist 2017 training-set size of 12,000 images in Table 1 is likely a typo and affects reproducibility, but it is not a reduction of a result to its input. No self-citations are load-bearing, no fitted input is renamed as a prediction, and no known result is merely renamed. The paper is self-contained as an empirical evaluation, so the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- λ (attention loss weight) =
1.0
- Number of attention maps M =
8
- Center loss momentum β =
not reported
- Orthogonal initialization matrix choice =
V from SVD of attention weights
assumptions (3)
- domain assumption Spatial locations of activated neurons in feature and attention maps are highly correlated with object parts in the raw input image.
- domain assumption ImageNet-pretrained InceptionV3 features at Mix6e provide a suitable representation for fine-grained part discovery.
- domain assumption Bilinear attention pooling, as formulated here (pooling over channels to produce a spatial row per part), yields discriminative part features under a softmax and center loss.
Cite this review
Pith. "Pith review of Coarse2Fine: A Two-stage Training Method for Fine-grained Visual Classification." pith.science (2026). https://pith.science/paper/ZPTA3ITF
@misc{pith2026190902680,
author = {Pith},
title = {Pith review of: Coarse2Fine: A Two-stage Training Method for Fine-grained Visual Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZPTA3ITF}},
note = {Machine review of arXiv:1909.02680}
}
read the original abstract
Small inter-class and large intra-class variations are the main challenges in fine-grained visual classification. Objects from different classes share visually similar structures and objects in the same class can have different poses and viewpoints. Therefore, the proper extraction of discriminative local features (e.g. bird's beak or car's headlight) is crucial. Most of the recent successes on this problem are based upon the attention models which can localize and attend the local discriminative objects parts. In this work, we propose a training method for visual attention networks, Coarse2Fine, which creates a differentiable path from the input space to the attended feature maps. Coarse2Fine learns an inverse mapping function from the attended feature maps to the informative regions in the raw image, which will guide the attention maps to better attend the fine-grained features. We show Coarse2Fine and orthogonal initialization of the attention weights can surpass the state-of-the-art accuracies on common fine-grained classification tasks.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
-
[2]
Dubey, A.; Gupta, O.; Guo, P.; Raskar, R.; Farrell, R.; and Naik, N. 2018. Pairwise confusion for fine-grained visual classification. In Ferrari, V.; Hebert, M.; Sminchisescu, C.; and Weiss, Y., eds., Computer Vision -- ECCV 2018 , 71--88. Cham: Springer International Publishing
work page 2018
-
[3]
Fu , J.; Zheng , H.; and Mei , T. 2017. Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 4476--4484
work page 2017
-
[4]
Girshick , R.; Donahue , J.; Darrell , T.; and Malik , J. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition , 580--587
work page 2014
-
[5]
He , K.; Zhang , X.; Ren , S.; and Sun , J. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 770--778
work page 2016
-
[6]
He, X., and Peng, Y. 2017. Weakly supervised learning of part selection model with spatial constraints for fine-grained image classification. In AAAI
work page 2017
-
[7]
V.; Mac Aodha , O.; Song, Y.; Shepard, A.; Adam, H.; Perona, P.; and Belongie, S
Horn, G. V.; Mac Aodha , O.; Song, Y.; Shepard, A.; Adam, H.; Perona, P.; and Belongie, S. J. 2017. The inaturalist challenge 2017 dataset. CoRR abs/1707.06642
arXiv 2017
-
[8]
Hu, T.; Qi, H.; Huang, C.; Huang, Q.; Lu, Y.; and Xu, J. 2018. Weakly supervised local attention network for fine-grained visual classification. CoRR abs/1808.02152
work page Pith review arXiv 2018
Show all 34 references
-
[9]
Hu, T., and Qi, H. 2019. See better before looking closer: Weakly supervised data augmentation network for fine-grained visual classification. CoRR abs/1901.09891
2019 arXiv
-
[10]
Jaderberg, M.; Simonyan, K.; Zisserman, A.; and kavukcuoglu, k. 2015. Spatial transformer networks. In Cortes, C.; Lawrence, N. D.; Lee, D. D.; Sugiyama, M.; and Garnett, R., eds., Advances in Neural Information Processing Systems 28 . Curran Associates, Inc. 2017--2025
2015
-
[11]
Khosla, A.; Jayadevaprakash, N.; Yao, B.; and Li, F.-F. 2012. Novel dataset for fine-grained image categorization : Stanford dogs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011
2012
-
[12]
Krause, J.; Stark, M.; Deng, J.; and Fei-Fei, L. 2013. 3d object representations for fine-grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops (ICCVW) , 554--561. Los Alamitos, CA, USA: IEEE Computer Society
2013
-
[13]
Krause , J.; Jin , H.; Yang , J.; and Fei-Fei , L. 2015. Fine-grained recognition without part annotations. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 5546--5555
2015
-
[14]
Li, P.; Xie, J.; Wang, Q.; and Gao, Z. 2018. Towards faster training of global covariance pooling networks by iterative matrix square root normalization. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
2018
-
[15]
Lin , D.; Shen , X.; Lu , C.; and Jia , J. 2015a. Deep lac: Deep localization, alignment and classification for fine-grained recognition. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 1666--1674
2015
-
[16]
Lin , T.; RoyChowdhury , A.; and Maji , S. 2015b. Bilinear cnn models for fine-grained visual recognition. In 2015 IEEE International Conference on Computer Vision (ICCV) , 1449--1457
2015
-
[17]
Liu, Z.; Luo, P.; Wang, X.; and Tang, X. 2015. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV)
2015
-
[18]
B.; and Vedaldi, A
Maji, S.; Rahtu, E.; Kannala, J.; Blaschko, M. B.; and Vedaldi, A. 2013. Fine-grained visual classification of aircraft. CoRR abs/1306.5151
2013 arXiv
-
[19]
Simon, M., and Rodner, E. 2015. Neural activation constellations: Unsupervised part model discovery with convolutional networks. In The IEEE International Conference on Computer Vision (ICCV)
2015
-
[20]
Sun, M.; Yuan, Y.; Zhou, F.; and Ding, E. 2018. Multi-attention multi-class constraint for fine-grained image recognition. In Ferrari, V.; Hebert, M.; Sminchisescu, C.; and Weiss, Y., eds., Computer Vision -- ECCV 2018 , 834--850. Cham: Springer International Publishing
2018
-
[21]
Szegedy, C.; Ioffe, S.; and Vanhoucke, V. 2016a. Inception-v4, inception-resnet and the impact of residual connections on learning. CoRR abs/1602.07261
-
[22]
Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; and Wojna, Z. 2016b. Rethinking the inception architecture for computer vision. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
-
[23]
Wah, C.; Branson, S.; Welinder, P.; Perona, P.; and Belongie, S. 2011. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001
2011
-
[24]
I.; and Davis, L
Wang, Y.; Morariu, V. I.; and Davis, L. S. 2016. Weakly-supervised discriminative patch learning via cnn for fine-grained recognition. ArXiv abs/1611.09932
2016 arXiv
-
[25]
Wang, F.; Jiang, M.; Qian, C.; Yang, S.; Li, C.; Zhang, H.; Wang, X.; and Tang, X. 2017. Residual attention network for image classification. CoRR abs/1704.06904
2017 arXiv
-
[26]
Wang, Q.; Teng, Z.; Xing, J.; Gao, J.; Hu, W.; and Maybank, S. J. 2018. Learning attentions: Residual attentional siamese network for high performance online visual tracking. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA...
2018
-
[27]
Wei, X.; Zhang, Y.; Gong, Y.; Zhang, J.; and Zheng, N. 2018. Grassmann pooling as compact homogeneous bilinear pooling for fine-grained visual classification. In Ferrari, V.; Hebert, M.; Sminchisescu, C.; and Weiss, Y., eds., Computer Vision -- ECCV 2018 , 365--380. Cham: Spri...
2018
-
[28]
Wen, Y.; Zhang, K.; Li, Z.; and Qiao, Y. 2016. A discriminative feature learning approach for deep face recognition. In ECCV
2016
-
[29]
Yang, Z.; Luo, T.; Wang, D.; Hu, Z.; Gao, J.; and Wang, L. 2018. Learning to navigate for fine-grained classification. In Ferrari, V.; Hebert, M.; Sminchisescu, C.; and Weiss, Y., eds., Computer Vision -- ECCV 2018 , 438--454. Cham: Springer International Publishing
2018
-
[30]
Zhang, N.; Donahue, J.; Girshick, R.; and Darrell, T. 2014. Part-based r-cnns for fine-grained category detection. In Fleet, D.; Pajdla, T.; Schiele, B.; and Tuytelaars, T., eds., Computer Vision -- ECCV 2014 , 834--849. Cham: Springer International Publishing
2014
-
[31]
Zhang, X.; Wei, Y.; Feng, J.; Yang, Y.; and Huang, T. S. 2018. Adversarial complementary learning for weakly supervised object localization. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
2018
-
[32]
Zheng , H.; Fu , J.; Mei , T.; and Luo , J. 2017. Learning multi-attention convolutional neural network for fine-grained image recognition. In 2017 IEEE International Conference on Computer Vision (ICCV) , 5219--5227
2017
-
[33]
Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; and Torralba, A. 2016. Learning deep features for discriminative localization. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
2016
-
[34]
Zhou, Z.-H. 2017. A brief introduction to weakly supervised learning . National Science Review 5(1):44--53
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.