REVIEW 3 major objections 5 minor 40 references
Detecting 11K Classes: Large Scale Object Detection without Fine-Grained Bounding Boxes
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A shared detector can learn fine-grained classes from image-level labels alone.
desk verdict The core semi-supervised detection recipe is real and the OpenImages/3K results are decent, but the 11K headline and 'nearly fully-supervised' claim are not backed by the reported numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is soft-attention based proposal re-ranking, defined by Eq. 7-8, coupled with a dual-level memory module with foreground attention pooling, Eq. 9-12. The re-ranking computes a coarse attention map from RPN proposals via softmax over classes and proposals, maps it to fine-grained labels through an encoding function $M(j)$ built from semantic or visual correlations, and multiplies it into the weakly supervised score map $S^w$ before pooling. The memory module applies a semi-supervised clustering-style loss at both the coarse proposal level and the fine image level, aggregating features by foreground attention pooling. A shared RPN, trained only on coarse boxes, generates proposals for both streams.
What would settle it
Train the same framework with coarse classes chosen to be visually distant from the fine-grained set, for example coarse classes from one domain and fine-grained classes from another, and measure fine-grained mAP; if it collapses while fully supervised fine-grained mAP stays high, the proposal-recall assumption is the load-bearing one. Concretely, compute the recall of the shared RPN on held-out fine-grained boxes and check whether fine-grained mAP correlates with that recall.
Extended reading notes
Core claim
The central claim is that correlations between coarse-grained and fine-grained classes are sufficient to transfer accurate localization knowledge. The paper's two-stream architecture shares a backbone and a region proposal network between a fully supervised detection stream, trained on coarse boxes, and a weakly supervised classification stream, trained on fine-grained image labels. A soft-attention re-ranking uses coarse proposal scores as an attention map, mapped through semantic or visual encodings, to re-rank fine-grained proposals, and a dual-level memory module regularizes both proposal-level and image-level distributions. Together these components lift fine-grained mAP from 34.0 to 51.9 on OpenImages, and yield a model that detects ImageNet's 11K classes with reasonable qualitative performance.
Load-bearing premise
The shared region proposal network, trained only on coarse-class bounding boxes, must propose the fine-grained objects with reliably high recall; if a fine-grained object never appears in the proposal list, neither attention re-ranking nor memory can recover it.
Editorial extensions
If this is right
- Fine-grained classes can be detected without fine-grained bounding boxes at accuracy close to fully supervised training, dramatically reducing annotation cost.
- The shared backbone benefits from diverse fine-grained images: the naive joint baseline already improves coarse-grained mAP by 7 points on OpenImages.
- The method outperforms pure weakly supervised detection by a large margin on OpenImages fine-grained classes, 51.9 versus 20.2 mAP.
- On ImageNet, the model detects 11K classes while keeping coarse-grained mAP above 49, outperforming YOLO-9000 and R-FCN-3000 on coarse classes.
- The framework is end-to-end trainable and can be transferred to any two-stage RPN-based detector.
Reading between the lines
- If the claim holds, annotation pipelines could shift from exhaustive box labeling to a coarse taxonomy plus image labels, potentially making web-scale detection economically feasible.
- A testable extension is replacing the fixed semantic or visual mapping with a learned correlation matrix updated during training, which might adapt to classes whose visual similarity differs from semantic distance.
- The method's success depends on proposal recall from the shared RPN; explicitly measuring fine-grained recall would reveal whether the approach extends to visually distant fine-grained classes.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a semi-supervised object detection framework for large-scale fine-grained detection. It assumes bounding-box annotations are available only for a set of coarse-grained classes while fine-grained classes have image-level labels only. The method combines a fully-supervised detection stream (R-FCN/SNIPER-style) with a weakly supervised fine-grained stream, connected by a shared backbone and RPN, a soft-attention proposal re-ranking mechanism built from coarse-to-fine class correlations, and a dual-level memory module. Experiments on OpenImages (34 coarse + 462 fine-grained classes) and ImageNet (200 coarse + 3K/11K fine-grained classes) report mAP@0.5 numbers, with OpenImages fine-grained mAP at 51.9 versus 59.1 for full supervision, ImageNet-3K fine-grained mAP at 35.1 versus 41.6, and a claim of near-parity in mAP@0.5:0.95 supported only by text statements. The 11K ImageNet model is evaluated only qualitatively.
Significance. The problem is practically important and the proposed architecture is a sensible way to combine coarse detection labels with fine-grained image-level labels. The OpenImages ablations are internally consistent and show that soft-attention (34.0 to 49.2 mAP-FG) and the dual-level memory (49.2 to 51.9) each contribute substantial gains, which is a meaningful technical result. If the parity claims were fully substantiated with quantitative results, the paper would be a strong contribution to large-scale detection. However, the headline claim of 'nearly fully-supervised accuracy' is overstated by the reported numbers (6-7 point gaps at mAP@0.5), and the flagship 11K ImageNet result is not quantitatively evaluated at all, so the significance as currently demonstrated is lower than the abstract suggests.
major comments (3)
- [Section 4.3, Table 4] The central claim of detecting 11K classes at nearly fully-supervised accuracy is unsupported by the evidence. In Table 4, Ours-11K reports mAP-FG as '-', and the only evidence for fine-grained 11K detection is the qualitative Figure 4. Since the title and abstract emphasize the 11K-class setting, the paper needs either a quantitative evaluation on a labeled subset of the 11K classes (even a small one) or a substantial softening of the claim. The only numeric fine-grained result, on 3K classes, shows a 6.5-point mAP@0.5 gap (35.1 vs 41.6), which is not 'nearly fully-supervised' in the usual sense.
- [Sections 4.2 and 4.3, after Tables 2 and 4] The mAP@0.5:0.95 parity claims are load-bearing for the 'as accurately as fully-supervised' statements but are not tabulated. The text says OpenImages results are 'both at around 36' and ImageNet results are 'our model at 25 and SNIPER-FG-Fully at 22,' yet no table reports these numbers or the evaluation protocol (which classes, which IoU range, how the fine-grained stream was evaluated on the 11K model). These numbers should be added to the tables with sufficient detail to be verifiable.
- [Sections 3.3.1 and 3.3.2] The shared RPN is trained only on coarse-grained bounding boxes, and the weakly supervised fine-grained stream reuses it to generate proposals; soft-attention re-ranking can only re-rank proposals that already exist. The paper never measures proposal recall on fine-grained classes. If a fine-grained object is visually distant from all coarse classes, the RPN may fail to propose it, and neither the attention nor the memory component can recover it. The paper should report proposal recall on a fine-grained set with box annotations (e.g., OpenImages FG validation or ImageNet-3K) and, ideally, break it down by class or by semantic/visual distance to the coarse classes.
minor comments (5)
- [Section 3.3.2, Eq. (7)] The notation is inconsistent: the text introduces 'the fine-grained attention map Ww(p)' but Eq. (7) defines 'Aw(p)'; please unify the symbol.
- [Section 3.3.2, Eq. (8)] The final weakly supervised loss uses a multi-label label vector y, but the classification loss Lcls is not formally defined; please state whether it is a sigmoid cross-entropy or a softmax-based loss and how it handles multiple positive labels.
- [Table 4] The YOLO-9000 and R-FCN-3000 comparisons are explicitly not trained on the same data as the proposed method; the sentence claiming the method 'outperforms YOLO-9000 and R-FCN-3000 by a large margin on coarse-grained classes' should be removed or clearly labeled as a non-apples-to-apples reference.
- [Section 4.3] The description 'randomly sample 1/8 images from all classes' does not specify the resulting training set size or whether the sampling is stratified per class; this matters for interpreting the 11K result and should be stated.
- [Section 3.3.3, Eq. (10)] The foreground attention pooling sums features only for proposals whose argmax score equals class c; if no proposal satisfies this for a class present in the image, the image-level feature is empty or zero, and the handling of this case is not described.
Circularity Check
No circularity found: fine-grained detection is evaluated on held-out bounding boxes, and the correlation, attention, and memory components are external inductive biases rather than fitted predictions.
full rationale
The central claim is that coarse-box plus image-label training detects fine-grained classes at near fully-supervised accuracy. The reported fine-grained numbers (51.9 mAP-FG on OpenImages, 35.1 on ImageNet-3K, Tables 2 and 4) are measured against held-out bounding-box annotations, so they are not fit constants or renamed training objectives. The visual correlation encoding of Section 3.2.2 is computed from features of a pretrained ResNet-101 over sampled class images, an external frozen source that does not use the detector's fine-grained outputs; the temperature and thresholds are hyper-parameters, not fits to the detection targets. The soft-attention re-ranking (Eqs. 5-8) only rescales proposal scores from the weakly supervised stream using coarse-stream scores; it cannot create a proposal that the shared RPN did not generate, and the fine-grained scores S_w are still learned from image-level labels. The dual-level memory loss (Eqs. 9-12) is a regularizer, not a predictor of box locations. No uniqueness theorem or load-bearing argument is imported from the authors' own prior work; self-citations [33], [34], [38] appear only in related-work descriptions of weakly supervised detection and do not support any step in the derivation. The missing quantitative mAP for the ImageNet-11K model and the lack of proposal-recall analysis are evidentiary gaps, which concern correctness rather than circularity.
Assumptions & free parameters
free parameters (4)
- lambda (weak-stream attention loss weight) =
0.1
- beta (soft-assignment temperature) =
not reported
- theta_i (hard-assignment class threshold) =
not reported
- Top-5 pooling count k =
5
assumptions (5)
- domain assumption Coarse-grained and fine-grained classes are connected by a reliable semantic or visual correlation structure (WordNet, Knowledge Graph, or feature-distance clusters).
- domain assumption The shared RPN, trained only on coarse-class boxes, generates proposals with sufficient recall for fine-grained objects.
- domain assumption ImageNet and OpenImages image-level labels are complete enough to supervise the fine-grained classification stream.
- domain assumption A frozen ResNet-101's averaged class features provide a stable basis for visual correlations.
- standard math Standard practice results (R-FCN, SNIPER, soft NMS) behave as described in their source papers.
Cite this review
Pith. "Pith review of Detecting 11K Classes: Large Scale Object Detection without Fine-Grained Bounding Boxes." pith.science (2026). https://pith.science/paper/U5BYL3MT
@misc{pith2026190805217,
author = {Pith},
title = {Pith review of: Detecting 11K Classes: Large Scale Object Detection without Fine-Grained Bounding Boxes},
year = {2026},
howpublished = {\url{https://pith.science/paper/U5BYL3MT}},
note = {Machine review of arXiv:1908.05217}
}
read the original abstract
Recent advances in deep learning greatly boost the performance of object detection. State-of-the-art methods such as Faster-RCNN, FPN and R-FCN have achieved high accuracy in challenging benchmark datasets. However, these methods require fully annotated object bounding boxes for training, which are incredibly hard to scale up due to the high annotation cost. Weakly-supervised methods, on the other hand, only require image-level labels for training, but the performance is far below their fully-supervised counterparts. In this paper, we propose a semi-supervised large scale fine-grained detection method, which only needs bounding box annotations of a smaller number of coarse-grained classes and image-level labels of large scale fine-grained classes, and can detect all classes at nearly fully-supervised accuracy. We achieve this by utilizing the correlations between coarse-grained and fine-grained classes with shared backbone, soft-attention based proposal re-ranking, and a dual-level memory module. Experiment results show that our methods can achieve close accuracy on object detection to state-of-the-art fully-supervised methods on two large scale datasets, ImageNet and OpenImages, with only a small fraction of fully annotated classes.
Figures
Reference graph
Works this paper leans on
-
[1]
Neural machine translation by jointly learning to align and translate
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473, 2014. 5
arXiv 2014
-
[2]
Weakly supervised deep detection networks
Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In CVPR, pages 2846–2854, 2016. 2, 5
work page 2016
-
[3]
Soft-nms - improving object detection with one line of code
Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis. Soft-nms - improving object detection with one line of code. In ICCV, pages 5562–5570, 2017. 7
work page 2017
-
[4]
Semi-supervised deep learning with memory
Yanbei Chen, Xiatian Zhu, and Shaogang Gong. Semi-supervised deep learning with memory. In ECCV, 2018. 6, 7
work page 2018
-
[5]
R-FCN: object detec- tion via region-based fully convolutional networks
Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-FCN: object detec- tion via region-based fully convolutional networks. In NIPS, pages 379–387, 2016. 1, 2, 4
work page 2016
-
[6]
Deformable convolutional networks
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, pages 764–773, 2017. 2
work page 2017
-
[7]
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255, 2009. 1, 6
work page 2009
-
[8]
Mark Everingham, Luc J. Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2):303–338, 2010. 1
work page 2010
Show all 40 references
-
[9]
NOTE-RCNN: noise tolerant ensemble RCNN for semi- supervised object detection
Jiyang Gao, Jiang Wang, Shengyang Dai, Li-Jia Li, and Ram Nevatia. NOTE-RCNN: noise tolerant ensemble RCNN for semi- supervised object detection. CoRR, abs/1812.00124, 2018. 3
2018 arXiv
-
[10]
Multi-evidence filtering and fusion for multi-label classification, object detection and semantic segmentation based on weakly supervised learning
Weifeng Ge, Sibei Yang, and Yizhou Yu. Multi-evidence filtering and fusion for multi-label classification, object detection and semantic segmentation based on weakly supervised learning. In CVPR, June
-
[11]
Girshick
Ross B. Girshick. Fast R-CNN. In ICCV, pages 1440–1448, 2015. 2
2015
-
[12]
Girshick
Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross B. Girshick. Mask R-CNN. In ICCV, pages 2980–2988, 2017. 1, 2
2017
-
[13]
Learn- ing to remember rare events
Lukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. Learn- ing to remember rare events. CoRR, abs/1703.03129, 2017. 6
2017 arXiv
-
[14]
Openimages: A public dataset for large-scale multi-label and multi-class image classification
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont-Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai...
-
[15]
Visual genome: Connecting language and vision using crowdsourced dense image annotations
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. 2016. 3
2016
-
[16]
Girshick, Kaiming He, Bharath Hariharan, and Serge J
Tsung-Yi Lin, Piotr Doll ´ar, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for ob- ject detection. In CVPR, pages 936–944, 2017. 1, 2
2017
-
[17]
Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Mi- crosoft COCO: common objects in context. In ECCV, pages 740– 755, 2014. 1
2014
-
[18]
Reed, Cheng-Yang Fu, and Alexander C
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg. SSD: single shot multibox detector. In ECCV, pages 21–37, 2016. 2
2016
-
[19]
Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston
Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. Key-value memory net- works for directly reading documents. InEMNLP, pages 1400–1409,
-
[20]
George A. Miller. Wordnet: A lexical database for english. Commun. ACM, 38(11):39–41, 1995. 3
1995
-
[21]
Papadopoulos, Jasper R
Dim P. Papadopoulos, Jasper R. R. Uijlings, Frank Keller, and Vitto- rio Ferrari. We don’t need no bounding-boxes: Training object class detectors using only human verification. In CVPR, pages 854–863,
-
[22]
Papadopoulos, Jasper R
Dim P. Papadopoulos, Jasper R. R. Uijlings, Frank Keller, and Vitto- rio Ferrari. Extreme clicking for efficient object annotation. InICCV, pages 4940–4949, 2017. 1
2017
-
[23]
Girshick, and Ali Farhadi
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, pages 779–788, 2016. 2
2016
-
[24]
YOLO9000: better, faster, stronger
Joseph Redmon and Ali Farhadi. YOLO9000: better, faster, stronger. In CVPR, pages 6517–6525, 2017. 2, 3, 8
2017
-
[25]
Yolov3: An incremental improve- ment
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improve- ment. CoRR, abs/1804.02767, 2018. 2
2018 arXiv
-
[26]
Girshick, and Jian Sun
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal net- works. IEEE T-PAMI, 39(6):1137–1149, 2017. 2
2017
-
[27]
Bharat Singh, Hengduo Li, Abhishek Sharma, and Larry S. Davis. R- FCN-3000 at 30fps: Decoupling detection and classification. 2018. 1, 2, 8
2018
-
[28]
SNIPER: Efficient multi-scale training
Bharat Singh, Mahyar Najibi, and Larry S Davis. SNIPER: Efficient multi-scale training. In NIPS, 2018. 1, 4, 6, 7, 8
2018
-
[29]
DOCK: detecting objects by transferring common- sense knowledge
Krishna Kumar Singh, Santosh Kumar Divvala, Ali Farhadi, and Yong Jae Lee. DOCK: detecting objects by transferring common- sense knowledge. In ECCV, pages 506–522, 2018. 7
2018
-
[30]
Multiple instance detection network with online instance classifier refinement
Peng Tang, Xinggang Wang, Xiang Bai, and Wenyu Liu. Multiple instance detection network with online instance classifier refinement. CoRR, abs/1704.00138, 2017. 2
2017 arXiv
-
[31]
Peng Tang, Xinggang Wang, Angtian Wang, Yongluan Yan, Wenyu Liu, Junzhou Huang, and Alan L. Yuille. Weakly supervised region proposal network and object detection. In ECCV, pages 370–386,
-
[32]
Gaizauskas, and Liming Chen
Yuxing Tang, Josiah Wang, Boyang Gao, Emmanuel Dellandr ´ea, Robert J. Gaizauskas, and Liming Chen. Large scale semi-supervised object detection using visual and semantic knowledge transfer. In CVPR, pages 2119–2128, 2016. 3, 7
2016
-
[33]
Exploiting web images for weakly supervised object detection
Qingyi Tao, Hao Yang, and Jianfei Cai. Exploiting web images for weakly supervised object detection. IEEE TMM, 2018. 2, 5
2018
-
[34]
Zero-annotation object de- tection with web knowledge transfer
Qingyi Tao, Hao Yang, and Jianfei Cai. Zero-annotation object de- tection with web knowledge transfer. In ECCV, 2018. 2, 5
2018
-
[35]
Jasper R. R. Uijlings, Stefan Popov, and Vittorio Ferrari. Revisit- ing knowledge transfer for training object class detectors. In CVPR, pages 1101–1110, 2018. 3, 7
2018
-
[36]
Jasper R. R. Uijlings, Koen E. A. van de Sande, Theo Gevers, and Arnold W. M. Smeulders. Selective search for object recognition. IJCV, 104(2):154–171, 2013. 2
2013
-
[37]
Zhe Wu, Navaneeth Bodla, Bharat Singh, Mahyar Najibi, Rama Chellappa, and Larry S. Davis. Soft sampling for robust object de- tection. CoRR, abs/1806.06986, 2018. 1
2018 arXiv
-
[38]
MIML-FCN+: multi-instance multi-label learning via fully convolu- tional networks with privileged information
Hao Yang, Joey Tianyi Zhou, Jianfei Cai, and Yew-Soon Ong. MIML-FCN+: multi-instance multi-label learning via fully convolu- tional networks with privileged information. In CVPR, pages 5996– 6004, 2017. 5
2017
-
[39]
W2f: A weakly-supervised to fully-supervised framework for object detection
Yongqiang Zhang, Yancheng Bai, Mingli Ding, Yongqiang Li, and Bernard Ghanem. W2f: A weakly-supervised to fully-supervised framework for object detection. In CVPR, June 2018. 2, 5
2018
-
[40]
Lawrence Zitnick and Piotr Doll ´ar
C. Lawrence Zitnick and Piotr Doll ´ar. Edge boxes: Locating object proposals from edges. In ECCV, pages 391–405, 2014. 2
2014
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.