REVIEW 4 major objections 6 minor 52 references
Rethinking Annotation for Object Detection: Is Annotating Small-size Instances Worth Its Cost?
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper argues that detectors can match fully annotated baselines on small objects using only test-time upscaling and resolution-matching augmentation.
desk verdict Genuinely new cost-benefit question, clean COCO experiment, honest appendix — but the aggregate parity hides per-class losses on small-only classes like traffic lights, so the 'small annotation isn't worth it' headline overreaches the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is test-time upscaling: the detector is applied twice to the same image, once at native resolution to catch medium and large objects and once enlarged by a factor α (around 2.0–2.2) to make small objects appear medium-sized, with detected boxes rescaled back and merged. Because upscaled small objects are blurrier than native medium objects, the training set is augmented by downscaling training images by 1/γ and upscaling them back by γ (γ = 3) — or by Gaussian blurring — so that the detector learns to map blurry, medium-sized appearances to the correct classes. A contrastive method, downscaling training images so medium objects become small synthetic instances, fails, isolating the domain gap in context and perspective as the limiting factor. Finally, knowledge distillation turns the two-pass teacher into a single-path student trained on pseudo labels for small instances plus the original medium and large annotations, restoring the practical computational cost of the baseline.
What would settle it
Compute per-class average precision for small instances on the COCO validation split for context-dependent classes whose training instances are predominantly small (for example, traffic light and remote); if the upscaling method falls clearly below the fully annotated baseline on those classes, the claimed parity does not hold for all objects.
Extended reading notes
Core claim
The central claim is that detectors can learn to detect small-size instances without any real small-size annotation, provided the input is upscaled at test time and the resolution gap is narrowed during training. In the COCO validation evaluation, the two-pass Up@Test-scale method reaches an average precision for small instances of 13.7 for Faster R-CNN against a baseline of 11.9, and 23.6 for FCOS against 22.3. Distilling the two-pass model into a single-path detector preserves the result, giving 11.0 for Faster R-CNN and 22.4 for FCOS, both comparable to their full-data baselines. The same experiments show that synthesizing small instances by downscaling training images does not match this performance, supporting the authors' conclusion that test-time upscaling, not small-instance annotation, is the effective ingredient.
Load-bearing premise
The load-bearing premise is that an upscaled small object, after the resolution-matching augmentation, looks enough like a native medium-sized object to the detector, so the remaining differences in context and perspective do not matter for the classes being detected.
Editorial extensions
If this is right
- If the claim holds, an annotation pipeline can omit or roughly label small instances and still obtain a detector whose small-object accuracy is comparable to one trained on complete data.
- The annotation budget saved on tiny objects could be redirected to harder medium and large annotations or to rare, context-dependent classes.
- Because the method is detector-agnostic, the same upscaling-plus-augmentation recipe may transfer to other anchor-based and anchor-free detectors.
- The failure of Down@Train predicts that synthetic small instances made by resizing will not substitute for real small annotations, so annotation decisions should account for context and perspective, not resolution alone.
Reading between the lines
- Editorial inference: the result raises the possibility of revising annotation guidelines so that crowd-like clusters of small instances are labeled with a single coarse box or an iscrowd flag, with test-time upscaling recovering the lost individual detections.
- Editorial inference: a testable extension is to apply the same recipe to datasets captured with different camera focal lengths or to video frames; if the context and perspective gap widens, parity with fully annotated baselines should degrade.
- Editorial inference: combining test-time upscaling with a super-resolution step could push small-object accuracy beyond the baseline, since the remaining gap the authors document for very small instances is partly a resolution effect.
- Editorial inference: the per-class analysis in the appendix suggests that classes whose training examples are mostly small, such as traffic lights and remotes, may still need annotations; a per-class cost-benefit rule could identify exactly which classes justify small-instance labeling.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper poses the cost-benefit question of whether annotating small-size object instances is worth its cost, and reformulates it as a falsifiable question: can a detector trained on data free of small-size instance annotations still detect small instances? The authors propose two methods: Up@Test, which upscales the input image at test time while training only on medium/large instances, optionally with blur or downscale-upscale augmentation to narrow the domain gap; and Down@Train, which downscales training images to create synthetic small instances. Experiments on COCO show that Up@Test with scale augmentation (γ=3) reaches small-instance AP slightly above the baseline trained on all annotations (Faster RCNN 13.7 vs 11.9; FCOS 23.6 vs 22.3), while Down@Train performs much worse. A distillation step yields a single-path student model with AP-small 11.0 vs 11.9 (Faster RCNN) and 22.4 vs 22.3 (FCOS), which the paper describes as comparable to the full-annotation baseline. The paper concludes that small-instance annotation may be unnecessary, and that this result should prompt rethinking of annotation practices.
Significance. The paper addresses a practically important and underexplored question: whether the high cost of annotating small objects is justified. The central claim, if it holds, would have direct implications for dataset construction and annotation cost reduction. The study is framed as a clear, testable hypothesis, and the proposed methods are simple, detector-agnostic, and include a distillation step that addresses the computational cost of running the detector twice. The paper is honest about the residual domain gap and shows failure cases. However, the evidence is limited to two detectors on a single dataset, and the headline conclusion relies on aggregate AP-small values that may conceal systematic per-class degradation, particularly for classes that only appear at small sizes in the training data. A per-class breakdown and a measure of variance across runs are needed to make the central claim robust.
major comments (4)
- [Sec. 5.4, observation (i) and Appendix A] The central claim that Up@Test achieves 'at least comparable performance' to Baseline is supported only at the aggregate AP-small level (Table 2: 13.7 vs 11.9 for Faster RCNN; 23.6 vs 22.3 for FCOS). Appendix A explicitly states that 'for objects that have only small-size instances in the training data (e.g., traffic light and remote), Baseline performs better.' This is not an edge case: traffic lights and remotes are exactly the kinds of small, context-dependent objects whose annotation the paper argues could be skipped. Without a per-class AP breakdown, the aggregate surplus could be driven by classes that have ample medium/large supervision while classes that predominantly appear small lose detection accuracy. Since the cost-benefit question is fundamentally per-class, the paper should report AP-small per class, or at least for classes with a high fraction of small instances, to support the headline conclusion.
- [Sec. 5.5, Table 3] The distilled single-path model for Faster RCNN achieves APS 11.0 vs Baseline 11.9, a deficit of 0.9, while for FCOS the values are 22.4 vs 22.3. The paper calls these 'comparable' but provides no error bars or multiple-seed results, so it is impossible to tell whether the Faster RCNN gap is within noise or a real degradation. The same issue applies to Table 2 (13.7 vs 11.9; 23.6 vs 22.3). The claim of 'at least comparable' would be much stronger with standard deviations over at least three runs.
- [Sec. 4.3 and Appendix A] The method rests on the assumption that the downscale-upscale augmentation (γ=3) or Gaussian blur closes the resolution part of the domain gap, and that the remaining context/perspective gap is small enough. Appendix A shows concrete failures on context-dependent cases (e.g., 'sports ball' on a cupcake, 'clock' in an eye) and on small-predominant classes. The paper acknowledges these failures but does not quantify how often they occur or how much they contribute to the aggregate result. A quantitative analysis of the residual domain gap, for instance by reporting AP-small separately for classes with a majority of small instances versus those with substantial medium/large supervision, would clarify the scope of the claim and the conditions under which the method can be expected to transfer to new datasets.
- [Sec. 5.1] All hyperparameters (α, γ, β, blur kernel and σ, confidence thresholds, mixing ratio) are selected based on COCO validation, and the final results are reported on that same validation split. This is effectively tuning on the test set, which can inflate the reported numbers and make the 'at least comparable' claim less reliable. The authors should either evaluate on a held-out test split (e.g., COCO test-dev) or, failing that, explicitly discuss the risk of overfitting to the validation set and report the sensitivity of the main results to the chosen hyperparameters.
minor comments (6)
- [Appendix A] The abbreviation 'DAT' is used in the opening paragraph of the appendix without ever being defined; it should be defined or replaced with 'Down@Train' for consistency.
- [Sec. 5.1] The sentence 'We do not employ any kind of multi-scale test or multi-scale training other than ours' is confusing because Up@Test itself is a form of multi-scale testing; please rephrase to clarify that no standard multi-scale augmentation or test-time augmentation is used beyond the proposed methods.
- [Sec. 5.1] The mixing ratio of 0.414 : 0.586 for blur/downscale augmentations is motivated only by the instance count ratio in COCO; the rationale for using this ratio for mixing training samples should be explained, since a ratio based on instance counts does not obviously translate to a ratio for image-level mixing.
- [Figure 3] The caption should state more explicitly that the 'human subject' is one of the authors and that the 500 images were chosen randomly, to avoid any impression of a formal human-study methodology.
- [Abstract and Sec. 5.5] The phrase 'at least comparable' is too strong given that the distilled Faster RCNN result (11.0 APS) is 0.9 below the baseline (11.9); consider softening to 'comparable within uncertainty' or reporting a confidence interval.
- [Figures 5-7] The legends and axis labels in Figures 5-7 are very small and hard to read; also, the figure captions should explicitly note that the broken lines are the Baseline values for each AP metric, not a single reference line.
Circularity Check
No circularity: the central claim is an empirical comparison against the external COCO validation benchmark, not a derivation from fitted inputs or self-citations.
full rationale
The paper makes no derived claim that is defined in terms of its own conclusion. The central observation (Sec. 5.4, observation i) is an empirical comparison against the external COCO validation set: Up@Test-scale is trained without small-instance annotations and evaluated on original small instances, with APS reported in Table 2 (13.7 vs 11.9 for Faster R-CNN; 23.6 vs 22.3 for FCOS). The domain-gap remedy (Gaussian blur or downscale-upscale by gamma) is a data augmentation applied to medium/large instances during training, not a fit to the small-instance APS being predicted. Hyperparameters (alpha, gamma, beta, confidence thresholds) are selected by validation performance, which is standard model selection rather than a fitted input renamed as a prediction. There are no self-citations, no uniqueness theorem imported from prior author work, and no ansatz smuggled in via citation. Appendix A's admission that classes appearing predominantly as small instances (e.g., traffic light and remote) are detected better by Baseline is a limitation of the central claim's scope, but it is an empirical boundary and does not make the derivation circular. The claim is directly falsifiable and self-contained against the COCO benchmark, so the circularity score is 0.
Assumptions & free parameters
free parameters (6)
- test-time upscaling factor alpha =
2.0 (Faster R-CNN), 2.2 (FCOS)
- training scale augmentation factor gamma =
3
- downscale factor beta for Down@Train =
1/3
- Gaussian blur kernel (k, sigma) =
(11, 2.0)
- distillation confidence threshold =
0.9 (Faster R-CNN), 0.7 (FCOS)
- augmentation mixing ratio =
0.414 : 0.586
assumptions (5)
- domain assumption COCO ground-truth annotations are correct
- domain assumption ImageNet pretraining transfers to COCO detection
- domain assumption Size scaling plus resolution matching suffices to bridge the train/test appearance gap for small instances
- domain assumption COCO small/medium/large area cutoffs define the problem
- domain assumption Validation-set hyperparameter selection gives unbiased performance estimates
Cite this review
Pith. "Pith review of Rethinking Annotation for Object Detection: Is Annotating Small-size Instances Worth Its Cost?." pith.science (2026). https://pith.science/paper/VWFA2BCT
@misc{pith2026241205611,
author = {Pith},
title = {Pith review of: Rethinking Annotation for Object Detection: Is Annotating Small-size Instances Worth Its Cost?},
year = {2026},
howpublished = {\url{https://pith.science/paper/VWFA2BCT}},
note = {Machine review of arXiv:2412.05611}
}
read the original abstract
Detecting objects occupying only small areas in an image is difficult, even for humans. Therefore, annotating small-size object instances is hard and thus costly. This study questions common sense by asking the following: is annotating small-size instances worth its cost? We restate it as the following verifiable question: can we detect small-size instances with a detector trained using training data free of small-size instances? We evaluate a method that upscales input images at test time and a method that downscales images at training time. The experiments conducted using the COCO dataset show the following. The first method, together with a remedy to narrow the domain gap between training and test inputs, achieves at least comparable performance to the baseline detector trained using complete training data. Although the method needs to apply the same detector twice to an input image with different scaling, we show that its distillation yields a single-path detector that performs equally well to the same baseline detector. These results point to the necessity of rethinking the annotation of training data for object detection.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
- [1]
-
[2]
J. Dai, Y . Li, K. He, and J. Sun. R-FCN: Object Detection via Region-based Fully Convolutional Networks. In Proc. NeurIPS. 2016. 2
work page 2016
-
[3]
K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian. CenterNet: Keypoint Triplets for Object Detection. In Proc. ICCV, 2019. 2
work page 2019
-
[4]
M. Everingham, L. V . Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The Pascal Visual Object Classes (VOC) Challenge. IJCV, 2010. 1, 3
work page 2010
-
[5]
C. Y . Fu, W. Liu, A. Ranga, A. Tyagi, and A. C. Berg. DSSD : Deconvolutional Single Shot Detector. arXiv:1701.06659,
-
[6]
J. Gao, J. Wang, S. Dai, L.-J. Li, and R. Nevatia. NOTE-RCNN: NOise Tolerant Ensemble RCNN for Semi- supervised Object Detection. In Proc. ICCV, 2019. 3
work page 2019
- [7]
-
[8]
B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama. Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels. In Proc. NeurIPS,
Show all 52 references
-
[9]
K. He, G. Gkioxari, P. Dollar, and R. Girshick. Mask R- CNN. In Proc. ICCV, 2017. 2
2017
-
[10]
K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In Proc. CVPR, 2016. 6
2016
-
[11]
Jeong, S
J. Jeong, S. Lee, J. Kim, and N. Kwak. Consistency-based Semi-supervised Learning for Object detection. In Proc. NeurIPS, 2019. 3
2019
-
[12]
Jiang, Z
L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei. Mentor- Net: Learning Data-Driven Curriculum for Very Deep Neu- ral Networks on Corrupted Labels. In Proc. ICML, 2018. 2
2018
-
[13]
T. Kong, F. Sun, H. Liu, Y . Jiang, and J. Shi. FoveaBox: Beyond Anchor-based Object Detector. arXiv:1904.03797,
1904 arXiv
-
[14]
Konyushkova, J
K. Konyushkova, J. Uijlings, C. H. Lampert, and V . Ferrari. Learning Intelligent Dialogs for Bounding Box Annotation. In Proc. CVPR, 2018. 1, 3
2018
-
[15]
Kosugi, T
S. Kosugi, T. Yamasaki, and K. Aizawa. Object-aware In- stance Labeling for Weakly Supervised Object Detection. In Proc. ICCV, 2019. 3
2019
-
[16]
Kuznetsova, H
A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, A. Kolesnikov, T. Duerig, and V . Ferrari. The Open Images Dataset V4: Unified Image Classification, Object Detection, and Visual Relationship Detection at Scale. IJCV, 2...
2020
-
[17]
Law and J
H. Law and J. Deng. CornerNet: Detecting Objects as Paired Keypoints. In Proc. ECCV, 2018. 2
2018
-
[18]
J. Li, C. Xiong, R. Socher, and S. Hoi. Towards Noise-resistant Object Detection with Noisy Annotations. arXiv:2003.01285, 2020. 2
2003 arXiv
-
[19]
T.-Y . Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Gir- shick, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick. Microsoft COCO: Common Objects in Context. In Proc. ECCV, 2014. 1, 3
2014
-
[20]
T. Y . Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature Pyramid Networks for Object Detection. In Proc. CVPR, 2017. 2, 3, 6
2017
-
[21]
T. Y . Lin, P. Goyal, R. Girshick, K. He, and P. Dollar. Focal Loss for Dense Object Detection. In Proc. ICCV, 2017. 2
2017
-
[22]
S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia. Path Aggregation Network for Instance Segmentation. In Proc. CVPR, 2018. 2
2018
-
[23]
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Fu, and A. C. Berg. SSD: Single Shot MultiBox Detector. In Proc. ECCV, 2016. 2
2016
-
[24]
X. Lu, B. Li, Y . Yue, Q. Li, and J. Yan. Grid R-CNN. In Proc. CVPR, 2019. 2
2019
-
[25]
Malach and S
E. Malach and S. Shalev-Shwartz. Decoupling ”When to Update” from ”How to Update”. In Proc. NeurIPS, 2017. 2
2017
-
[26]
H. Mao, X. Yang, and W. J. Dally. A Delay Metric for Video Object Detection: What Average Precision Fails to Tell. In Proc. ICCV, 2019. 3
2019
-
[27]
X. Ming, F. Wei, T. Zhang, D. Chen, and F. Wen. Group Sampling for Scale Invariant Face Detection. InProc. CVPR,
-
[28]
Oksuz, B
K. Oksuz, B. C. Cam, E. Akbas, and S. Kalkan. Localiza- tion Recall Precision (LRP): A New Performance Metric for Object Detection. In Proc. ECCV, 2018. 3
2018
-
[29]
D. P. Papadopoulos, J. R. R. Uijlings, F. Keller, and V . Fer- rari. Extreme Clicking for Efficient Object Annotation. In Proc. ICCV, 2017. 1, 3
2017
-
[30]
D. P. Papadopoulos, J. R. R. Uijlings, F. Keller, and V . Fer- rari. Training Object Class Detectors with Click Supervision. In Proc. CVPR, 2017. 1, 3
2017
-
[31]
Redmon and A
J. Redmon and A. Farhadi. YOLO9000: Better, Faster, Stronger. In Proc. CVPR, 2017. 2
2017
-
[32]
Redmon and A
J. Redmon and A. Farhadi. YOLOv3: An Incremental Im- provement. arXiv:1804.02767, 2018. 2
2018 arXiv
-
[33]
M. Ren, W. Zeng, B. Yang, and R. Urtasun. Learning to Reweight Examples for Robust Deep Learning. In Proc. ICML, 2018. 2
2018
-
[34]
S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards Real-time Object Detection with Region Proposal Networks. In Proc. NeurIPS, 2015. 2, 4, 6
2015
-
[35]
Rochan and Y
M. Rochan and Y . Wang. Weakly Supervised Localization of Novel Objects Using Appearance Transfer. In Proc. CVPR,
-
[36]
Samet, S
N. Samet, S. Hicsonmez, and E. Akbas. Reducing Label Noise in Anchor-free Object Detection. In Proc. BMVC,
-
[37]
Singh and L
B. Singh and L. S. Davis. An Analysis of Scale Invariance in Object Detection SNIP. In Proc. CVPR, 2018. 2
2018
-
[38]
Singh, M
B. Singh, M. Najibi, and Larry S Davis. SNIPER: Efficient Multi-scale Training. In Proc. NeurIPS. 2018. 2
2018
-
[39]
M. Tan, R. Pang, and Q. V . Le. EfficientDet: Scalable and Efficient Object Detection. In Proc. CVPR, 2020. 2, 3, 4
2020
-
[40]
Tanaka, D
D. Tanaka, D. Ikami, T. Yamasaki, and K. Aizawa. Joint Optimization Framework for Learning with Noisy Labels. In Proc. CVPR, 2018. 2
2018
-
[41]
Y . Tang, J. Wang, B. Gao, E. Dellandrea, R. Gaizauskas, and L. Chen. Large Scale Semi-supervised Object Detection Using Visual and Semantic Knowledge Transfer. In Proc. CVPR, 2016. 3
2016
-
[42]
Z. Tian, C. Shen, H. Chen, and T. He. FCOS: Fully Convo- lutional One-stage Object Detection. In Proc. ICCV, 2019. 2, 4, 6
2019
-
[43]
Touvron, A
H. Touvron, A. Vedaldi, M. Douze, and H. Jegou. Fixing the Train-test Resolution Discrepancy. In Proc. NeurIPS. 2019. 3
2019
-
[44]
Uijlings, S
J. Uijlings, S. Popov, and V . Ferrari. Revisiting Knowl- edge Transfer for Training Object Class Detectors. In Proc. CVPR, 2018. 3
2018
-
[45]
Z. Yang, S. Liu, H. Hu, L. Wang, and S. Lin. RepPoints: Point Set Representation for Object Detection. In Proc. ICCV, 2019. 2
2019
-
[46]
Yi and J
K. Yi and J. Wu. Probabilistic End-to-end Noise Correction for Learning with Noisy Labels. In Proc. CVPR, 2019. 2
2019
-
[47]
Zhang, L
S. Zhang, L. Wen, X. Bian, Z. Lei, and S. Z. Li. Single-shot Refinement Neural Network for Object Detection. In Proc. CVPR, 2018. 2
2018
-
[48]
Zhang, Y
X. Zhang, Y . Yang, and J. Feng. Learning to Localize Objects with Noisy Labeled Instances. In Proc. AAAI, 2019. 3
2019
-
[49]
Q. Zhao, T. Sheng, Y . Wang, Z. Tang, Y . Chen, L. Cai, and H. Ling. M2det: A Single-shot Object Detector Based on Multi-level Feature Pyramid Network. In Proc. AAAI, 2019. 2
2019
-
[50]
X. Zhou, D. Wang, and P. Kr ¨ahenb¨uhl. Objects as Points. arXiv:1904.07850, 2019. 2
1904 arXiv
-
[51]
X. Zhou, J. Zhuo, and P. Krahenbuhl. Bottom-up Object Detection by Grouping Extreme and Center Points. In Proc. CVPR, 2019. 2
2019
-
[52]
fire hydrant
C. Zhu, Y . He, and M. Savvides. Feature Selective Anchor- free Module for Single-shot Object Detection. In Proc. CVPR, 2019. 2 Appendix A. Examples of Detection Results As is reported in the main paper, Up@Test-scale is com- parable or slightly better than Baseline in terms o...
2019
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.