REVIEW 3 major objections 4 minor 1 cited by
Residual Objectness for Imbalance Reduction
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper proposes Residual Objectness, a fully learned cascade of objectness refinements that addresses foreground-background imbalance without sampling or reweighting, and reports consistent gains on COCO across three detector families.
desk verdict ResObj is a solid empirical component with ~1 AP gains on three detectors, but the paper doesn't isolate the residual mechanism from the extra objectness head, so the causal story is softer than the title suggests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is Residual Objectness: the objectness estimate is written as $o = o_0 + \sum_{t=1}^T r_t$, where $o_0$ comes from an initial objectness subnet and each $r_t$ from a separate residual subnet. The refinement at step $t$ applies only to anchors whose previous objectness score is above a minimum positive score, so easy negatives are progressively filtered out while positives continue to be improved. The residual subnets are isolated from the objectness subnet in backpropagation, and the final class score is computed as $P(\text{class}|\text{obj}) \times P(\text{obj})$. This sum-of-refinements formulation is what turns objectness estimation into a consecutive refinement procedure.
What would settle it
Train a T=1 ResObj detector exactly as in the paper, and a matched control that adds the same objectness head but combines its logits with the initial objectness by concatenation followed by a linear layer, rather than by residual addition, with the same loss and anchor selection. If the control's AP matches or exceeds the residual version, the residual refinement procedure is not the cause of the improvement.
Extended reading notes
Core claim
The paper's central discovery is that foreground-background imbalance can be transferred from the multi-class classification subnet to a binary objectness task, and then progressively resolved by refining the objectness score several times. A RetinaNet with a single objectness head and plain cross-entropy, no Focal Loss, reaches 34.2 AP versus 34.1 AP for the Focal Loss version, showing that the imbalance is implicitly alleviated by binary objectness prediction. Adding residual refinement, with the objectness written as $o = o_0 + \sum_{t=1}^T r_t$, raises the positive objectness scores over training and yields the reported gains: RetinaNet-ResObj 37.0 AP versus RetinaNet-FL 35.7 with ResNet-50, YOLOv3-ResObj 34.1 versus 33.0 at 608x608, and FasterRCNN-ResObj 38.4 versus 37.2 with ResNet-50. The claim is that this consecutive refinement, not any external sampling schedule, is what fixes the imbalance.
Load-bearing premise
The paper assumes that the observed AP gains come from the residual refinement formulation, rather than from simply adding extra supervised objectness heads; a control with the same number of added heads but without residual connections is missing, so the active ingredient is not isolated.
Editorial extensions
If this is right
- Detectors can drop Focal Loss and related hyper-parameters; the paper's RetinaNet-ResObj keeps the same anchor assignment but uses cross-entropy with an objectness head and residual refinement.
- The same residual objectness module can be plugged into a one-stage detector (YOLOv3) and a two-stage detector (Faster R-CNN's RPN), improving both by a similar margin.
- The cost of the added heads is small: the paper reports inference speed drops by roughly 0.1-0.2 FPS when moving from T=0 to T=2.
- Because the final class score multiplies the refined objectness, the mechanism directly sharpens detection confidence, which the paper illustrates with qualitative examples where object scores rise from T=0 to T=2.
- The reported gains are consistent across object scales (AP_S, AP_M, AP_L), not confined to large objects.
Reading between the lines
- The step ablation does not include a control with the same number of extra objectness heads but without residual connections, so whether the residual aggregation is the active ingredient, as opposed to extra supervised heads, remains untested.
- The same residual-refinement idea could be applied to any binary foreground/background head, including keypoint or center-point heads in anchor-free detectors, though the paper does not test those.
- Because the refinement is applied only to anchors whose previous objectness is above a minimum positive score, the procedure implicitly defines an online curriculum; one could compare its effective loss surface to Focal Loss and GHM analytically.
- The reported gains are relative (3-4%), i.e. absolute gains of about 1.0-1.3 AP, and the paper's longer-training ResNet-101 runs use scale jitter; whether the margin persists under identical training schedules is not isolated.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Residual Objectness (ResObj), a fully learning-based mechanism for foreground-background imbalance in object detection. The method first transfers the multiclass imbalance to a binary objectness head, then cascades multiple residual objectness subnets that iteratively refine the objectness score, with gradients of the residual subnets isolated from the objectness subnet during backpropagation. The authors replace Focal Loss in RetinaNet, the objectness module in YOLOv3, and random sampling in Faster R-CNN's RPN with the residual objectness mechanism, reporting relative AP improvements of 3.6%, 3.9%, and 3.2% on COCO test-dev, respectively. Ablations examine gradient flow, residual-head placement, and the number of residual steps T.
Significance. The empirical claim is practically valuable: if the residual-refinement mechanism is indeed the active ingredient, ResObj offers a simple, end-to-end trainable alternative to hand-designed sampling/reweighting schemes, with consistent gains across both one-stage and two-stage detectors and negligible inference overhead. The paper ships solid public-benchmark results (COCO test-dev) and a useful ablation set, including a T=0 control showing that the objectness module alone matches Focal Loss and separate ablations for gradient isolation and head placement. However, the central attribution of the gains to the residual formulation is not yet established, and the training procedure depends on an undefined threshold. These issues are fixable but require additional experiments and reporting.
major comments (3)
- [Section 4.2, Table 2(c)] The step ablation does not isolate the residual-refinement mechanism. Moving from T=0 (34.1 AP) to T=1 (35.2 AP) simultaneously adds an extra 3x3-convolution objectness head, an extra binary cross-entropy loss on anchors selected by the ominp threshold, and changes the inference score to sigmoid(o0+r1). Since no control with a second independent objectness head (e.g., a head trained with its own binary loss but combined multiplicatively or by summation without the residual framing) is reported, the observed +1.1 AP cannot be attributed to the residual formulation specifically. This is load-bearing because the paper's novelty and generality claims—'consecutive refinement' and 'progressively addressing the imbalance'—depend on the residual mechanism being the active ingredient, not merely on adding an extra supervised objectness head.
- [Algorithm 1, line 5; Section 4.2] The threshold ominp_{t-1} is undefined. Algorithm 1 uses it to decide which anchors contribute to the residual loss, and the paper claims the method 'rarely requires hyper-parameters tuning,' but no definition, estimation procedure, or reported values are given. Without this information, the training procedure is not reproducible, and it remains possible that a hand-tuned selection rule carries part of the observed effect. Please specify how ominp is computed (e.g., a percentile of positive objectness scores, a fixed value, or a moving statistic) and report its value or schedule for each detector.
- [Section 4.2, Table 2(c); Section 1] The claim that the method avoids hyper-parameter tuning is overstated. The number of residual steps T is an architecture hyper-parameter, the paper's own text says the T=2 result is achieved 'with simple hyper-parameters tuning,' and Table 1 shows that the RetinaNet-Obj baseline's AP varies from 33.1 to 34.1 depending on the inference and NMS thresholds. Please state exactly which hyper-parameters are tuned in the reported results and how sensitive the gains are to the inference threshold, since this directly qualifies the 'no hand-crafted sampling/reweighting' and 'fully learning-based' contributions.
minor comments (4)
- [Equation (1)] Equation (1) has an unmatched opening parenthesis after 'FocalLoss'; the summation expression is missing a closing parenthesis. This should be corrected for readability.
- [Section 4.3, Table 3] The text states that YOLOv3-ResObj has '1.1 AP (relative 3.9%) lower than us at both 320x320 and 608x608 scales,' but Table 3 shows 28.2 to 29.3 (relative 3.9%) at 320x320 and 33.0 to 34.1 (relative 3.3%) at 608x608. Please correct the inconsistent relative-gain statement.
- [Section 4.1 vs Table 3] The ablation tables use a 600-pixel image scale, while Table 3 reports results at 1333x800 for RetinaNet and Faster R-CNN. The text says 'we use a 600 pixel train and test image scale in all ablation studies,' but it would help to explicitly state the scale used for each table, especially because the vanilla RetinaNet baseline differs between Table 2 (34.2 AP) and Table 3 (35.7 AP).
- [Section 2] The related-work comparison states that existing cascaded architectures 'are not designed for one-stage detectors,' but C-RPN is discussed as an exception; the wording in the 'Comparison and Difference' paragraph ('most of them... are only applicable for the per-region stage, whereas the only exception C-RPN is designed for object tracking') is slightly confusing and should be clarified.
Circularity Check
No circularity: the AP improvements are measured on COCO, and the loss decomposition in Eqs. (4)-(7) is an analytic observation, not a prediction forced by construction.
full rationale
The paper's central contributions are empirical: Residual Objectness is defined as cascaded objectness modules with residual connections, and its effectiveness is demonstrated by COCO test-dev AP gains against published baselines (RetinaNet, YOLOv3, Faster R-CNN). The only analytical part, Eqs. (4)-(7), compares the negative-loss contribution of Focal Loss versus an objectness loss and shows the objectness formulation reduces the negative loss by roughly a factor of K; this is a loss-level observation and does not by itself entail any AP number. No parameter is fitted to a subset and then reported as a prediction: the ablations vary T and gradient flow, and the final APs are direct benchmark measurements. No load-bearing conclusion is imported from a self-citation; references to Focal Loss, Cascade R-CNN, etc. are standard external context. The main caveat, that the step ablation in Table 2(c) changes capacity, extra loss, and residual aggregation simultaneously, concerns causal attribution of the improvement, not circularity, since the improvement is still measured rather than derived from the method's own assumptions.
Assumptions & free parameters
free parameters (3)
- Number of residual steps T =
2
- Inference objectness threshold =
0.05
- NMS threshold =
0.50
assumptions (3)
- domain assumption Anchor assignment with IoU 0.5 for foreground and [0, 0.4) for background is used without reweighting, and all unassigned anchors incur only objectness loss.
- domain assumption The minimum positive score ominp in Algorithm 1 can be computed per batch and contains all positives plus hard negatives.
- ad hoc to paper Adding objectness subnets on top of the class head and isolating gradients from residual to objectness subnets is beneficial.
Cite this review
Pith. "Pith review of Residual Objectness for Imbalance Reduction." pith.science (2026). https://pith.science/paper/RM74O2TZ
@misc{pith2026190809075,
author = {Pith},
title = {Pith review of: Residual Objectness for Imbalance Reduction},
year = {2026},
howpublished = {\url{https://pith.science/paper/RM74O2TZ}},
note = {Machine review of arXiv:1908.09075}
}
read the original abstract
For a long time, object detectors have suffered from extreme imbalance between foregrounds and backgrounds. While several sampling/reweighting schemes have been explored to alleviate the imbalance, they are usually heuristic and demand laborious hyper-parameters tuning, which is hard to achieve the optimality. In this paper, we first reveal that such the imbalance could be addressed in a learning-based manner. Guided by this illuminating observation, we propose a novel Residual Objectness (ResObj) mechanism that addresses the imbalance by end-to-end optimization, while no further hand-crafted sampling/reweighting is required. Specifically, by applying multiple cascaded objectness-related modules with residual connections, we formulate an elegant consecutive refinement procedure for distinguishing the foregrounds from backgrounds, thereby progressively addressing the imbalance. Extensive experiments present the effectiveness of our method, as well as its compatibility and adaptivity for both region-based and one-stage detectors, namely, the RetinaNet-ResObj, YOLOv3-ResObj and FasterRCNN-ResObj achieve relative 3.6%, 3.9%, 3.2% Average Precision (AP) improvements compared with their vanilla models on COCO, respectively.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
Imbalance Problems in Object Detection: A Review
A taxonomy and critical review organizing eight object-detection imbalance problems under class, scale, spatial, and objective imbalance.
Reference graph
Works this paper leans on
-
[1]
R. B. Girshick, J. Donahue, T. Darrell, J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation, in: IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 580–587
work page 2014
-
[2]
R. B. Girshick, Fast R-CNN, in: IEEE International Conference on Com- puter Vision, 2015, pp. 1440–1448
work page 2015
-
[3]
S. Ren, K. He, R. B. Girshick, J. Sun, Faster R-CNN: towards real-time object detection with region proposal networks, IEEE Trans. Pattern Anal. Mach. Intell. 39 (6) (2017) 1137–1149
work page 2017
-
[4]
J. Dai, Y. Li, K. He, J. Sun, R-FCN: object detection via region-based fully convolutional networks, in: Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems, 2016, pp. 379–387
work page 2016
-
[5]
K. He, G. Gkioxari, P. Doll´ ar, R. B. Girshick, Mask R-CNN, in: IEEE International Conference on Computer Vision, 2017, pp. 2980–2988
work page 2017
-
[6]
Z. Cai, N. Vasconcelos, Cascade R-CNN: delving into high quality object detection, in: IEEE Conference on Computer Vision and Pattern Recogni- tion, 2018, pp. 6154–6162
work page 2018
-
[7]
J. Pang, K. Chen, J. Shi, H. Feng, W. Ouyang, D. Lin, Libra R-CNN: towards balanced learning for object detection, CoRR abs/1904.02701. arXiv:1904.02701. URL http://arxiv.org/abs/1904.02701
work page Pith review arXiv 1904
-
[8]
J. R. R. Uijlings, K. E. A. van de Sande, T. Gevers, A. W. M. Smeulders, Selective search for object recognition, International Journal of Computer Vision 104 (2) (2013) 154–171
work page 2013
Show all 45 references
-
[9]
C. L. Zitnick, P. Doll´ ar, Edge boxes: Locating object proposals from edges, in: 13th European Conference on Computer Vision, 2014, pp. 391–405. 20
2014
-
[10]
Redmon, S
J. Redmon, S. K. Divvala, R. B. Girshick, A. Farhadi, You only look once: Unified, real-time object detection, in: IEEE Conference on Computer Vi- sion and Pattern Recognition, 2016, pp. 779–788
2016
-
[11]
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. E. Reed, C. Fu, A. C. Berg, SSD: single shot multibox detector, in: 14th European Conference on Computer Vision, 2016, pp. 21–37
2016
-
[12]
Redmon, A
J. Redmon, A. Farhadi, YOLO9000: better, faster, stronger, in: IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6517– 6525
2017
-
[13]
T. Lin, P. Goyal, R. B. Girshick, K. He, P. Doll´ ar, Focal loss for dense object detection, in: IEEE International Conference on Computer Vision, 2017, pp. 2999–3007
2017
-
[14]
Zhang, L
S. Zhang, L. Wen, X. Bian, Z. Lei, S. Z. Li, Single-shot refinement neural network for object detection, in: IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 4203–4212
2018
-
[15]
Redmon, A
J. Redmon, A. Farhadi, Yolov3: An incremental improvement, CoRR abs/1804.02767. arXiv:1804.02767. URL http://arxiv.org/abs/1804.02767
-
[16]
Huang, Y
L. Huang, Y. Yang, Y. Deng, Y. Yu, Densebox: Unifying landmark local- ization with end to end object detection, CoRR abs/1509.04874. arXiv: 1509.04874. URL http://arxiv.org/abs/1509.04874
-
[17]
H. Law, J. Deng, Cornernet: Detecting objects as paired keypoints, in: 15th European Conference on Computer Vision, 2018, pp. 765–781
2018
-
[18]
X. Zhou, J. Zhuo, P. Kr¨ ahenb¨ uhl, Bottom-up object detection by grouping extreme and center points, CoRR abs/1901.08043. arXiv:1901.08043. URL http://arxiv.org/abs/1901.08043 21
1901 arXiv
-
[19]
J. Wang, K. Chen, S. Yang, C. C. Loy, D. Lin, Region proposal by guided anchoring, CoRR abs/1901.03278. arXiv:1901.03278. URL http://arxiv.org/abs/1901.03278
1901 arXiv
-
[20]
T. Kong, F. Sun, H. Liu, Y. Jiang, J. Shi, Foveabox: Beyond anchor-based object detector, CoRR abs/1904.03797. arXiv:1904.03797. URL http://arxiv.org/abs/1904.03797
1904 arXiv
-
[21]
X. Zhou, D. Wang, P. Kr¨ ahenb¨ uhl, Objects as points, CoRR abs/1904.07850. arXiv:1904.07850. URL http://arxiv.org/abs/1904.07850
1904 arXiv
-
[22]
B. Li, Y. Liu, X. Wang, Gradient harmonized single-stage detector, in: The Thirty-Third AAAI Conference on Artificial Intelligence, 2019, pp. 8577–8584
2019
-
[23]
Shrivastava, A
A. Shrivastava, A. Gupta, R. B. Girshick, Training region-based object detectors with online hard example mining, in: IEEE Conference on Com- puter Vision and Pattern Recognition, 2016, pp. 761–769
2016
-
[24]
T. Kong, F. Sun, A. Yao, H. Liu, M. Lu, Y. Chen, RON: reverse connection with objectness prior networks for object detection, in: IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 5244–5252
2017
-
[25]
C. Chi, S. Zhang, J. Xing, Z. Lei, S. Z. Li, X. Zou, Selective refinement network for high performance face detection, in: The Thirty-Third AAAI Conference on Artificial Intelligence, 2019, pp. 8231–8238
2019
-
[26]
J. Li, Y. Wang, C. Wang, Y. Tai, J. Qian, J. Yang, C. Wang, J. Li, F. Huang, DSFD: dual shot face detector, CoRR abs/1810.10220. arXiv: 1810.10220. URL http://arxiv.org/abs/1810.10220
-
[27]
P. A. Viola, M. J. Jones, Robust real-time face detection, International Journal of Computer Vision 57 (2) (2004) 137–154. 22
2004
-
[28]
P. F. Felzenszwalb, R. B. Girshick, D. A. McAllester, Cascade object de- tection with deformable part models, in: IEEE Conference on Computer Vision and Pattern Recognition, 2010, pp. 2241–2248
2010
-
[29]
Everingham, L
M. Everingham, L. J. V. Gool, C. K. I. Williams, J. M. Winn, A. Zisserman, The pascal visual object classes (VOC) challenge, International Journal of Computer Vision 88 (2) (2010) 303–338
2010
-
[30]
T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, C. L. Zitnick, Microsoft COCO: common objects in context, in: 13th Eu- ropean Conference on Computer Vision, 2014, pp. 740–755
2014
-
[31]
Alexe, T
B. Alexe, T. Deselaers, V. Ferrari, Measuring the objectness of image win- dows, IEEE Trans. Pattern Anal. Mach. Intell. 34 (11) (2012) 2189–2202
2012
-
[32]
Alexe, T
B. Alexe, T. Deselaers, V. Ferrari, What is an object?, in: IEEE Conference on Computer Vision and Pattern Recognition, 2010, pp. 73–80
2010
-
[33]
Cheng, Y
M. Cheng, Y. Liu, W. Lin, Z. Zhang, P. L. Rosin, P. H. S. Torr, BING: bina- rized normed gradients for objectness estimation at 300fps, Computational Visual Media 5 (1) (2019) 3–20
2019
-
[34]
J. Wang, X. Tao, M. Xu, Y. Duan, J. Lu, Hierarchical objectness network for region proposal generation and object detection, Pattern Recognition 83 (2018) 260–272
2018
-
[35]
B. Yang, J. Yan, Z. Lei, S. Z. Li, CRAFT objects from images, in: IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 6043– 6051
2016
-
[36]
Najibi, M
M. Najibi, M. Rastegari, L. S. Davis, G-CNN: an iterative grid based object detector, in: IEEE Conference on Computer Vision and Pattern Recogni- tion, 2016, pp. 2369–2377
2016
-
[37]
Ghodrati, A
A. Ghodrati, A. Diba, M. Pedersoli, T. Tuytelaars, L. V. Gool, Deeppro- posals: Hunting objects and actions by cascading deep convolutional layers, International Journal of Computer Vision 124 (2) (2017) 115–131. 23
2017
-
[38]
Cheng, Y
K. Cheng, Y. Chen, W. Fang, Improved object detection with iterative localization refinement in convolutional neural networks, IEEE Trans. Cir- cuits Syst. Video Techn. 28 (9) (2018) 2261–2275
2018
-
[39]
H. Fan, H. Ling, Siamese cascaded region proposal networks for real-time visual tracking, CoRR abs/1812.06148. arXiv:1812.06148. URL http://arxiv.org/abs/1812.06148
-
[40]
K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recogni- tion, in: IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778
2016
-
[41]
T. Lin, P. Doll´ ar, R. B. Girshick, K. He, B. Hariharan, S. J. Belongie, Feature pyramid networks for object detection, in: IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 936–944
2017
-
[42]
Girshick, I
R. Girshick, I. Radosavovic, G. Gkioxari, P. Doll´ ar, K. He, Detectron, https://github.com/facebookresearch/detectron (2018)
2018
-
[43]
Redmon, Darknet: Open source neural networks in c, http://pjreddie
J. Redmon, Darknet: Open source neural networks in c, http://pjreddie. com/darknet/ (2013–2016)
2013
-
[44]
Russakovsky, J
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, F. Li, Imagenet large scale visual recognition challenge, International Journal of Computer Vision 115 (3) (2015) 211–252
2015
-
[45]
Francisco, G
M. Francisco, G. Ross, maskrcnn-benchmark: Fast, modular ref- erence implementation of Instance Segmentation and Object Detec- tion algorithms in PyTorch, https://github.com/facebookresearch/ maskrcnn-benchmark, accessed: [July 21, 2019] (2018). 24
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.