REVIEW 3 major objections 5 minor 32 references
IoU-balanced Loss Functions for Single-stage Object Detection
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper claims that reweighting the classification and localization losses of single-stage detectors with the IoU of each predicted box improves localization accuracy substantially—AP by 1.0–1.7% and AP75 by 1.0–2.4% on COCO…
desk verdict IoU-weighted losses are a plausible, simple +1-2 AP gain for single-stage detectors, but the classification loss as written is an algebraic no-op unless the weights are detached—needs a clear revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a per-example weight that depends on the IoU between the currently predicted box and its ground-truth box, applied to both loss terms. For classification, the weight is iou^η normalized so the total positive-example loss stays constant; for localization, the weight is w_loc·iou^λ applied to each smooth L1 term, with the gradient detached so the weight acts as a fixed training signal rather than a differentiable function of the prediction. The paper derives closed-form gradient upper bounds using the Bounded IoU approximation, which treats IoU as a simplified function of coordinate offsets, and uses them to show why the scheme raises inlier gradients and suppresses outlier gradients.
What would settle it
Train a single-stage detector with the same two loss weights but allow gradients to flow through the IoU weight, or freeze the weights at their first-iteration values; if the AP gains disappear or training diverges, the detachment and the moving weight are load-bearing. Alternatively, log per-example gradient norms during training: the paper's claim implies high-IoU examples should contribute more gradient and low-IoU examples less than with plain smooth L1.
Extended reading notes
Core claim
The central claim is that localization accuracy in single-stage detectors is limited less by the network design than by the loss functions that train it. Standard cross-entropy treats every positive anchor equally, so the classifier learns high scores even for poorly localized boxes; standard smooth L1 lets outliers dominate the regression gradient. The paper's IoU-balanced classification loss multiplies the cross-entropy term of each positive example by a normalized weight proportional to the regressed IoU raised to a power η, and the IoU-balanced localization loss multiplies the smooth L1 terms by a weight w_loc·IoU^λ that up-weights inliers and down-weights outliers. The gradient is not propagated through the IoU weight. With the best settings (η=1.5, λ=1.5) the combined losses improve AP by 1.0–1.7% and AP75 by 1.0–2.4% on COCO test-dev across SSD, RetinaNet, and FoveaBox, and improve AP80/90 by 1.6–3.9% on PASCAL VOC, with no architectural change and no inference cost.
Load-bearing premise
The approach relies on the assumption that weighting each example by how well the model currently localizes it—while preventing the gradient from adjusting that weight—will continue improving the model instead of destabilizing training.
Editorial extensions
If this is right
- On COCO test-dev, applying IoU-balanced losses to SSD300/512, RetinaNet, and FoveaBox improves AP by 1.0–1.7% and AP75 by 1.0–2.4%.
- The improvements concentrate at high IoU thresholds: AP80 and AP90 rise by 1.6–2.6% on COCO and 1.6–3.9% on PASCAL VOC, showing the losses mainly buy tighter localization.
- The losses generalize across datasets (COCO, PASCAL VOC, Cityscapes) and across backbones (ResNet-18 to ResNeXt-101) with consistent AP gains of about 1–1.5%.
- The correlation between classification scores and localization improves: the paper reports higher average classification scores for high-IoU detections and lower scores for low-IoU detections after training with the IoU-balanced classification loss.
- Two-stage detectors benefit less (about 0.4–0.6% AP), consistent with the claim that the mismatch is smaller when proposals are already well localized.
Reading between the lines
- Because the reweighting is a drop-in training-time change, the same recipe could apply to other dense prediction tasks where confidence and spatial accuracy are jointly trained, such as keypoint detection or instance segmentation, provided an IoU-like quality measure can be computed per positive example.
- The gradient-detachment choice is the part most worth probing: allowing gradients to flow through the IoU weight would turn the loss into a feedback loop, and the paper gives no stability analysis. A controlled experiment varying only the detach operation would isolate whether detachment is load-bearing.
- The use of Bounded IoU rather than exact IoU in the gradient derivation suggests the exact gradient may behave differently near the δ boundary; measuring the true gradient norms during training would show whether the claimed inlier/outlier rebalancing holds in practice.
- Since the gains are largest at high IoU thresholds, the losses may be a cheaper alternative to cascade or refinement modules when the goal is strict localization, for example in scene text or aerial imagery.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes two loss modifications for single-stage object detectors. The IoU-balanced classification loss (Eqs. 3-4) is intended to reweight positive classification examples by the current predicted IoU, and the IoU-balanced localization loss (Eqs. 5-6) reweights smooth L1 by a power of IoU while stopping gradient flow through the weight. Experiments with SSD, RetinaNet, and FoveaBox on COCO, PASCAL VOC, and Cityscapes report consistent AP gains of about 1.0-1.7 points on COCO test-dev and larger gains at high IoU thresholds, with unchanged inference speed.
Significance. If the mechanism is correct, the contribution is practically valuable because it improves localization without architectural changes or inference cost. The paper's strengths are the breadth of the empirical study (three datasets, multiple detectors and backbones, a two-stage comparison), the generalization check using hyperparameters fixed from COCO, and the explicit gradient analysis for the localization loss. However, the classification loss as written is a mathematical no-op, so the central claim currently rests on an underspecified or incorrect formula.
major comments (3)
- [Sec. 3.2, Eqs. (3)-(4)] As written, the IoU-balanced classification loss is algebraically identical to the standard cross-entropy loss on the positive examples. Substituting Eq. (4) into Eq. (3), the positive-term sum Σ_{i∈Pos} w_i CE_i equals [Σ_j CE_j / Σ_j iou_j^η CE_j] · Σ_i iou_i^η CE_i = Σ_j CE_j, so the forward loss does not depend on IoU at all. If gradients are allowed to flow through w_i, the gradients with respect to the classification logits are also those of standard CE. If w_i is intended to be detached (as is explicitly stated for the localization loss in Sec. 3.3), the forward value is still standard CE and only the gradient is reweighted; this is not stated for the classification loss. The paper must specify the exact implementation, including any stop-gradient, and correct the normalization (for example, using w_i = iou_i^η / mean_j(iou_j^η)) so that the loss is not a mathematical no-op. Without this correction, the +0.7 AP attributed to the IoU-balanced classification loss in Table 2 is unexplained.
- [Sec. 3.3, Eqs. (5)-(14)] The localization loss uses the model's own current IoU as a per-example weight, with gradients detached from the weight, creating a feedback loop whose dynamics are not analyzed. The gradient formulas in Eqs. (13)-(14) are derived from the Bounded IoU approximation with the additional assumptions δ=0.111 and w_t=w_s, so they describe an upper-bound surrogate rather than the exact IoU used in training. The paper should either provide evidence that the exact-IoU gradient behaves in the same way (for example, a plot of the exact gradient against d for representative boxes) or state clearly that the analysis is only heuristic. This matters because the claimed improvement mechanism is precisely that the weight up-weights inliers and down-weights outliers.
- [Sec. 4, Tables 1-7] All reported results are single training runs without error bars or significance tests. The headline improvements are 0.7-1.7 AP, a range where typical seed-to-seed variation for these detectors is non-negligible, and the hyperparameters η and λ are selected on COCO val (Table 3) before reporting test-dev numbers. I recommend reporting results over at least three seeds for the main comparisons, or otherwise providing evidence that the gains exceed run-to-run noise.
minor comments (5)
- [Sec. 1 and Sec. 3.3] There are several language slips: 'We constrains' should be 'We constrain', and 'it's variants' should be 'its variants'.
- [Sec. 3.2, Eq. (4)] The symbol N is used both for the number of positive examples and as the upper limit of summation; please define N explicitly as the number of positive examples.
- [Sec. 3.3, Eqs. (6)-(7)] Equation (7) redefines w_i after Eq. (6) has already defined it; please clarify that Eq. (7) is an alternative normalization strategy and is not used in the main experiments.
- [Figure 2] The caption and axis labels in Figure 2 are small, and the curves for different λ values are hard to distinguish in print; please enlarge the figure or use distinct line styles.
- [References] Reference [23] contains a typo: 'Beyound' should be 'Beyond'.
Circularity Check
IoU-balanced classification loss is algebraically identical to standard CE by Eq. 4's normalization, so its reported gains are not explained by the loss as written.
-
self definitional
[Section 3.2, Equations (3)-(4)]
"Lcls = N∑ i∈Pos wi(ioui)∗ CE(pi, ˆpi) + M∑ i∈Neg CE(pi, ˆpi) (3) wi(ioui) =iouη i N∑ i=1 CE(pi, ˆpi) N∑ i=1 iouη i CE(pi, ˆpi) (4) ... the normalization strategy as Equ.4 shows is adopted to keep the sum of classification loss for positive examples unchanged compared with the standard cross-entropy loss during training."
Substituting Eq. 4 into the positive-example term of Eq. 3 gives: Σ_i [iou_i^η CE_i (Σ_j CE_j) / (Σ_j iou_j^η CE_j)] = (Σ_j CE_j)(Σ_i iou_i^η CE_i)/(Σ_j iou_j^η CE_j) = Σ_j CE_j. Thus the IoU-balanced positive classification loss is exactly the standard cross-entropy loss as a function of CE and IoU, with no dependence on the IoU weights after the cancellation. If gradients flow through w_i, the total loss and its gradients are identical to standard CE, so the +0.7 AP attributed to IoU-Cls in Table 2 cannot arise. If the weights are meant to be detached, the forward value is still standard CE, and the stop-gradient is stated only for the localization loss in Sec. 3.3, not for the classification loss.
full rationale
The paper's main empirical claims on COCO, PASCAL VOC, and Cityscapes are externally measured and are not themselves derived from the loss formulas, so the experimental comparison is not circular in the usual sense. However, the IoU-balanced classification loss is self-definitional: its prescribed normalization, Eq. 4, cancels the IoU weights exactly, making the positive classification loss algebraically equal to the standard cross-entropy loss it is meant to replace. This is a direct, exhibitable reduction from the paper's own equations. The localization loss is different: Eq. 6 with a detached weight in Sec. 3.3 is a genuine gradient reweighting, and the Bounded IoU approximation is an external modeling choice, so that component is a real modification. There is no load-bearing self-citation: reference [20] is related work by overlap authors but is not used to justify the proposed losses. Because one of the two central components reduces by construction to a no-op while the other remains independent, the appropriate score is partial circularity, 6, rather than a full 8-10.
Assumptions & free parameters
free parameters (3)
- η (IoU-balanced classification loss exponent) =
1.5
- λ (IoU-balanced localization loss exponent) =
1.5
- w_loc (localization loss scale) =
3.049 for λ=1.5
assumptions (3)
- domain assumption The IoU between a predicted box and its ground truth can be bounded by the simplified Bounded IoU expressions (Eq. 9-12) taken from Fitness NMS [18].
- ad hoc to paper Gradient detachment of the IoU weight is a safe optimization choice.
- domain assumption The MMDetection implementations of RetinaNet, SSD, FoveaBox, and Faster R-CNN are valid baselines and the default hyperparameters are optimal for the baselines.
Cite this review
Pith. "Pith review of IoU-balanced Loss Functions for Single-stage Object Detection." pith.science (2026). https://pith.science/paper/43IFRRSG
@misc{pith2026190805641,
author = {Pith},
title = {Pith review of: IoU-balanced Loss Functions for Single-stage Object Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/43IFRRSG}},
note = {Machine review of arXiv:1908.05641}
}
abstract
Single-stage object detectors have been widely applied in computer vision applications due to their high efficiency. However, we find that the loss functions adopted by single-stage object detectors hurt the localization accuracy seriously. Firstly, the standard cross-entropy loss for classification is independent of the localization task and drives all the positive examples to learn as high classification scores as possible regardless of localization accuracy during training. As a result, there will be many detections that have high classification scores but low IoU or detections that have low classification scores but high IoU. Secondly, for the standard smooth L1 loss, the gradient is dominated by the outliers that have poor localization accuracy during training. The above two problems will decrease the localization accuracy of single-stage detectors. In this work, IoU-balanced loss functions that consist of IoU-balanced classification loss and IoU-balanced localization loss are proposed to solve the above problems. The IoU-balanced classification loss pays more attention to positive examples with high IoU and can enhance the correlation between classification and localization tasks. The IoU-balanced localization loss decreases the gradient of examples with low IoU and increases the gradient of examples with high IoU, which can improve the localization accuracy of models. Extensive experiments on challenging public datasets such as MS COCO, PASCAL VOC and Cityscapes demonstrate that both IoU-balanced losses can bring substantial improvement for the popular single-stage detectors, especially for the localization accuracy. On COCO test-dev, the proposed methods can substantially improve AP by $1.0\%\sim1.7\%$ and AP75 by $1.0\%\sim2.4\%$. On PASCAL VOC, it can also substantially improve AP by $1.3\%\sim1.5\%$ and AP80, AP90 by $1.6\%\sim3.9\%$.
Figures
Reference graph
Works this paper leans on
-
[1]
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, A. C. Berg, Ssd: Single shot multibox detector, in: European conference on computer vision, Springer, 2016, pp. 21–37
work page 2016
- [2]
-
[3]
T.-Y. Lin, P. Goyal, R. Girshick, K. He, P. Doll´ ar, Focal loss for dense object detection, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2980–2988
work page 2017
- [4]
- [5]
-
[6]
B. Li, Y. Liu, X. Wang, Gradient harmonized single-stage detector, in: Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 33, 2019, pp. 8577–8584
work page 2019
-
[7]
S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: Towards real-time object detection with region proposal networks, in: Advances in neural information processing systems, 2015, pp. 91–99. 19
work page 2015
-
[8]
Z. Cai, N. Vasconcelos, Cascade r-cnn: Delving into high quality object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 6154–6162
work page 2018
Show all 32 references
-
[9]
K. He, G. Gkioxari, P. Doll´ ar, R. Girshick, Mask r-cnn, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961– 2969
2017
-
[10]
T.-Y. Lin, P. Doll´ ar, R. Girshick, K. He, B. Hariharan, S. Belongie, Feature pyramid networks for object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2117–2125
2017
-
[11]
J. Dai, Y. Li, K. He, J. Sun, R-fcn: Object detection via region-based fully convolutional networks, in: Advances in neural information pro- cessing systems, 2016, pp. 379–387
2016
-
[12]
Girshick, Fast r-cnn, in: Proceedings of the IEEE international conference on computer vision, 2015, pp
R. Girshick, Fast r-cnn, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 1440–1448
2015
-
[13]
Girshick, J
R. Girshick, J. Donahue, T. Darrell, J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation, in: Proceed- ings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 580–587
2014
-
[14]
Bodla, B
N. Bodla, B. Singh, R. Chellappa, L. S. Davis, Soft-nms–improving object detection with one line of code, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 5561–5569
2017
-
[15]
Gidaris, N
S. Gidaris, N. Komodakis, Object detection via a multi-region and semantic segmentation-aware cnn model, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 1134–1142
2015
-
[16]
Shrivastava, A
A. Shrivastava, A. Gupta, R. Girshick, Training region-based object detectors with online hard example mining, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 761–769
2016
-
[17]
J. Pang, K. Chen, J. Shi, H. Feng, W. Ouyang, D. Lin, Libra r-cnn: Towards balanced learning for object detection, in: Proceedings of the 20 IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 821–830
2019
-
[18]
Tychsen-Smith, L
L. Tychsen-Smith, L. Petersson, Improving object localization with fit- ness nms and bounded iou loss, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 6877–6885
2018
-
[19]
Jiang, R
B. Jiang, R. Luo, J. Mao, T. Xiao, Y. Jiang, Acquisition of localiza- tion confidence for accurate object detection, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 784–799
2018
-
[20]
S. Wu, X. Li, X. Wang, Iou-aware single-stage object detector for accu- rate localization, Image and Vision Computing (2019)
2019
-
[21]
Huang, L
Z. Huang, L. Huang, Y. Gong, C. Huang, X. Wang, Mask scoring r- cnn, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 6409–6418
2019
-
[22]
K. He, X. Zhang, S. Ren, J. Sun, Spatial pyramid pooling in deep convolutional networks for visual recognition, IEEE transactions on pattern analysis and machine intelligence 37 (2015) 1904–1916
2015
-
[23]
T. Kong, F. Sun, H. Liu, Y. Jiang, L. Li, J. Shi, Foveabox: Beyound anchor-based object detection, IEEE Transactions on Image Processing 29 (2020) 7389–7398
2020
-
[24]
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, C. L. Zitnick, Microsoft coco: Common objects in context, in: European conference on computer vision, Springer, 2014, pp. 740– 755
2014
-
[25]
Everingham, L
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, A. Zisserman, The pascal visual object classes (voc) challenge, International journal of computer vision 88 (2010) 303–338
2010
-
[26]
Cordts, M
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benen- son, U. Franke, S. Roth, B. Schiele, The cityscapes dataset for semantic urban scene understanding, in: Proc. of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2016. 21
2016
-
[27]
K. Chen, J. Wang, J. Pang, Y. Cao, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Xu, et al., Mmdetection: Open mmlab detection toolbox and benchmark, arXiv preprint arXiv:1906.07155 (2019)
2019 arXiv
-
[28]
Goyal, P
P. Goyal, P. Doll´ ar, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, K. He, Accurate, large minibatch sgd: Training imagenet in 1 hour, arXiv preprint arXiv:1706.02677 (2017)
2017 arXiv
-
[29]
Redmon, A
J. Redmon, A. Farhadi, Yolo9000: better, faster, stronger, in: Proceed- ings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7263–7271
2017
-
[30]
Redmon, A
J. Redmon, A. Farhadi, Yolov3: An incremental improvement, arXiv preprint arXiv:1804.02767 (2018)
2018 arXiv
-
[31]
J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, Y. Wei, Deformable convolutional networks, in: Proceedings of the IEEE international con- ference on computer vision, 2017, pp. 764–773
2017
-
[32]
Girshick, I
R. Girshick, I. Radosavovic, G. Gkioxari, P. Doll´ ar, K. He, Detectron, https://github.com/facebookresearch/detectron, 2018. 22
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.