Pith. sign in

REVIEW 3 major objections 5 minor 32 references

IoU-balanced Loss Functions for Single-stage Object Detection

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read This paper claims that reweighting the classification and localization losses of single-stage detectors with the IoU of each predicted box improves localization accuracy substantially—AP by 1.0–1.7% and AP75 by 1.0–2.4% on COCO…

desk verdict IoU-weighted losses are a plausible, simple +1-2 AP gain for single-stage detectors, but the classification loss as written is an algebraic no-op unless the weights are detached—needs a clear revision. read the letter →

arxiv 1908.05641 v2 pith:43IFRRSG submitted 2019-08-15 cs.CV

classification cs.CV
keywords IoU-balancedclassificationlosslocalizationobjectdetectionaccurateclassimbalanceexampleminingsmoothL1single-stagedetectors
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the two standard loss terms in single-stage object detectors—cross-entropy for classification and smooth L1 for bounding-box regression—are themselves a source of localization error, and that fixing the loss functions alone can recover much of the accuracy that more complex architectures are built to gain. To that end it proposes IoU-balanced losses: a classification loss that up-weights positive examples whose predicted boxes have high intersection-over-union with ground truth, and a localization loss that does the same for the regression gradients while suppressing the gradients of low-IoU outliers. On COCO test-dev the losses raise AP by 1.0–1.7 points and AP75 by 1.0–2.4 points across SSD, RetinaNet, and FoveaBox, with no change to architecture or inference time. The paper also shows the gains transfer to PASCAL VOC and Cityscapes, with the largest gains at high IoU thresholds (AP80/AP90), exactly where localization quality matters most.

What carries the argument

The central mechanism is a per-example weight that depends on the IoU between the currently predicted box and its ground-truth box, applied to both loss terms. For classification, the weight is iou^η normalized so the total positive-example loss stays constant; for localization, the weight is w_loc·iou^λ applied to each smooth L1 term, with the gradient detached so the weight acts as a fixed training signal rather than a differentiable function of the prediction. The paper derives closed-form gradient upper bounds using the Bounded IoU approximation, which treats IoU as a simplified function of coordinate offsets, and uses them to show why the scheme raises inlier gradients and suppresses outlier gradients.

What would settle it

Train a single-stage detector with the same two loss weights but allow gradients to flow through the IoU weight, or freeze the weights at their first-iteration values; if the AP gains disappear or training diverges, the detachment and the moving weight are load-bearing. Alternatively, log per-example gradient norms during training: the paper's claim implies high-IoU examples should contribute more gradient and low-IoU examples less than with plain smooth L1.

Watch

Extended reading notes

Core claim

The central claim is that localization accuracy in single-stage detectors is limited less by the network design than by the loss functions that train it. Standard cross-entropy treats every positive anchor equally, so the classifier learns high scores even for poorly localized boxes; standard smooth L1 lets outliers dominate the regression gradient. The paper's IoU-balanced classification loss multiplies the cross-entropy term of each positive example by a normalized weight proportional to the regressed IoU raised to a power η, and the IoU-balanced localization loss multiplies the smooth L1 terms by a weight w_loc·IoU^λ that up-weights inliers and down-weights outliers. The gradient is not propagated through the IoU weight. With the best settings (η=1.5, λ=1.5) the combined losses improve AP by 1.0–1.7% and AP75 by 1.0–2.4% on COCO test-dev across SSD, RetinaNet, and FoveaBox, and improve AP80/90 by 1.6–3.9% on PASCAL VOC, with no architectural change and no inference cost.

Load-bearing premise

The approach relies on the assumption that weighting each example by how well the model currently localizes it—while preventing the gradient from adjusting that weight—will continue improving the model instead of destabilizing training.

Editorial extensions

If this is right

  • On COCO test-dev, applying IoU-balanced losses to SSD300/512, RetinaNet, and FoveaBox improves AP by 1.0–1.7% and AP75 by 1.0–2.4%.
  • The improvements concentrate at high IoU thresholds: AP80 and AP90 rise by 1.6–2.6% on COCO and 1.6–3.9% on PASCAL VOC, showing the losses mainly buy tighter localization.
  • The losses generalize across datasets (COCO, PASCAL VOC, Cityscapes) and across backbones (ResNet-18 to ResNeXt-101) with consistent AP gains of about 1–1.5%.
  • The correlation between classification scores and localization improves: the paper reports higher average classification scores for high-IoU detections and lower scores for low-IoU detections after training with the IoU-balanced classification loss.
  • Two-stage detectors benefit less (about 0.4–0.6% AP), consistent with the claim that the mismatch is smaller when proposals are already well localized.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the reweighting is a drop-in training-time change, the same recipe could apply to other dense prediction tasks where confidence and spatial accuracy are jointly trained, such as keypoint detection or instance segmentation, provided an IoU-like quality measure can be computed per positive example.
  • The gradient-detachment choice is the part most worth probing: allowing gradients to flow through the IoU weight would turn the loss into a feedback loop, and the paper gives no stability analysis. A controlled experiment varying only the detach operation would isolate whether detachment is load-bearing.
  • The use of Bounded IoU rather than exact IoU in the gradient derivation suggests the exact gradient may behave differently near the δ boundary; measuring the true gradient norms during training would show whether the claimed inlier/outlier rebalancing holds in practice.
  • Since the gains are largest at high IoU thresholds, the losses may be a cheaper alternative to cascade or refinement modules when the goal is strict localization, for example in scene text or aerial imagery.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes two loss modifications for single-stage object detectors. The IoU-balanced classification loss (Eqs. 3-4) is intended to reweight positive classification examples by the current predicted IoU, and the IoU-balanced localization loss (Eqs. 5-6) reweights smooth L1 by a power of IoU while stopping gradient flow through the weight. Experiments with SSD, RetinaNet, and FoveaBox on COCO, PASCAL VOC, and Cityscapes report consistent AP gains of about 1.0-1.7 points on COCO test-dev and larger gains at high IoU thresholds, with unchanged inference speed.

Significance. If the mechanism is correct, the contribution is practically valuable because it improves localization without architectural changes or inference cost. The paper's strengths are the breadth of the empirical study (three datasets, multiple detectors and backbones, a two-stage comparison), the generalization check using hyperparameters fixed from COCO, and the explicit gradient analysis for the localization loss. However, the classification loss as written is a mathematical no-op, so the central claim currently rests on an underspecified or incorrect formula.

major comments (3)
  1. [Sec. 3.2, Eqs. (3)-(4)] As written, the IoU-balanced classification loss is algebraically identical to the standard cross-entropy loss on the positive examples. Substituting Eq. (4) into Eq. (3), the positive-term sum Σ_{i∈Pos} w_i CE_i equals [Σ_j CE_j / Σ_j iou_j^η CE_j] · Σ_i iou_i^η CE_i = Σ_j CE_j, so the forward loss does not depend on IoU at all. If gradients are allowed to flow through w_i, the gradients with respect to the classification logits are also those of standard CE. If w_i is intended to be detached (as is explicitly stated for the localization loss in Sec. 3.3), the forward value is still standard CE and only the gradient is reweighted; this is not stated for the classification loss. The paper must specify the exact implementation, including any stop-gradient, and correct the normalization (for example, using w_i = iou_i^η / mean_j(iou_j^η)) so that the loss is not a mathematical no-op. Without this correction, the +0.7 AP attributed to the IoU-balanced classification loss in Table 2 is unexplained.
  2. [Sec. 3.3, Eqs. (5)-(14)] The localization loss uses the model's own current IoU as a per-example weight, with gradients detached from the weight, creating a feedback loop whose dynamics are not analyzed. The gradient formulas in Eqs. (13)-(14) are derived from the Bounded IoU approximation with the additional assumptions δ=0.111 and w_t=w_s, so they describe an upper-bound surrogate rather than the exact IoU used in training. The paper should either provide evidence that the exact-IoU gradient behaves in the same way (for example, a plot of the exact gradient against d for representative boxes) or state clearly that the analysis is only heuristic. This matters because the claimed improvement mechanism is precisely that the weight up-weights inliers and down-weights outliers.
  3. [Sec. 4, Tables 1-7] All reported results are single training runs without error bars or significance tests. The headline improvements are 0.7-1.7 AP, a range where typical seed-to-seed variation for these detectors is non-negligible, and the hyperparameters η and λ are selected on COCO val (Table 3) before reporting test-dev numbers. I recommend reporting results over at least three seeds for the main comparisons, or otherwise providing evidence that the gains exceed run-to-run noise.
minor comments (5)
  1. [Sec. 1 and Sec. 3.3] There are several language slips: 'We constrains' should be 'We constrain', and 'it's variants' should be 'its variants'.
  2. [Sec. 3.2, Eq. (4)] The symbol N is used both for the number of positive examples and as the upper limit of summation; please define N explicitly as the number of positive examples.
  3. [Sec. 3.3, Eqs. (6)-(7)] Equation (7) redefines w_i after Eq. (6) has already defined it; please clarify that Eq. (7) is an alternative normalization strategy and is not used in the main experiments.
  4. [Figure 2] The caption and axis labels in Figure 2 are small, and the curves for different λ values are hard to distinguish in print; please enlarge the figure or use distinct line styles.
  5. [References] Reference [23] contains a typo: 'Beyound' should be 'Beyond'.

Circularity Check

1 steps flagged · score 6.0 of 10

IoU-balanced classification loss is algebraically identical to standard CE by Eq. 4's normalization, so its reported gains are not explained by the loss as written.

  1. self definitional [Section 3.2, Equations (3)-(4)]
    "Lcls = N∑ i∈Pos wi(ioui)∗ CE(pi, ˆpi) + M∑ i∈Neg CE(pi, ˆpi) (3) wi(ioui) =iouη i N∑ i=1 CE(pi, ˆpi) N∑ i=1 iouη i CE(pi, ˆpi) (4) ... the normalization strategy as Equ.4 shows is adopted to keep the sum of classification loss for positive examples unchanged compared with the standard cross-entropy loss during training."

    Substituting Eq. 4 into the positive-example term of Eq. 3 gives: Σ_i [iou_i^η CE_i (Σ_j CE_j) / (Σ_j iou_j^η CE_j)] = (Σ_j CE_j)(Σ_i iou_i^η CE_i)/(Σ_j iou_j^η CE_j) = Σ_j CE_j. Thus the IoU-balanced positive classification loss is exactly the standard cross-entropy loss as a function of CE and IoU, with no dependence on the IoU weights after the cancellation. If gradients flow through w_i, the total loss and its gradients are identical to standard CE, so the +0.7 AP attributed to IoU-Cls in Table 2 cannot arise. If the weights are meant to be detached, the forward value is still standard CE, and the stop-gradient is stated only for the localization loss in Sec. 3.3, not for the classification loss.

full rationale

The paper's main empirical claims on COCO, PASCAL VOC, and Cityscapes are externally measured and are not themselves derived from the loss formulas, so the experimental comparison is not circular in the usual sense. However, the IoU-balanced classification loss is self-definitional: its prescribed normalization, Eq. 4, cancels the IoU weights exactly, making the positive classification loss algebraically equal to the standard cross-entropy loss it is meant to replace. This is a direct, exhibitable reduction from the paper's own equations. The localization loss is different: Eq. 6 with a detached weight in Sec. 3.3 is a genuine gradient reweighting, and the Bounded IoU approximation is an external modeling choice, so that component is a real modification. There is no load-bearing self-citation: reference [20] is related work by overlap authors but is not used to justify the proposed losses. Because one of the two central components reduces by construction to a no-op while the other remains independent, the appropriate score is partial circularity, 6, rather than a full 8-10.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim depends on two design hyperparameters (η, λ), a manually tuned loss scale w_loc, the Bounded IoU approximation for gradient analysis, and gradient detachment. No new physical or conceptual entities are introduced.

free parameters (3)
  • η (IoU-balanced classification loss exponent) = 1.5
    Searched over {0, 1.0, 1.4, 1.5, 1.6} on COCO val (Table 3); controls how strongly high-IoU positives are up-weighted.
  • λ (IoU-balanced localization loss exponent) = 1.5
    Searched over {0, 0.5, 1.0, 1.5, 1.8} on COCO val (Table 3); controls the trade-off between inlier and outlier gradients.
  • w_loc (localization loss scale) = 3.049 for λ=1.5
    Manually adjusted for each λ to keep the sum of localization loss unchanged; it scales the gradient magnitude and affects training dynamics.
assumptions (3)
  • domain assumption The IoU between a predicted box and its ground truth can be bounded by the simplified Bounded IoU expressions (Eq. 9-12) taken from Fitness NMS [18].
    Used in Sec. 3.3 to derive the gradient norms in Figure 2; the actual training loss uses the exact IoU, so the analysis is illustrative, not exact.
  • ad hoc to paper Gradient detachment of the IoU weight is a safe optimization choice.
    Stated in Sec. 3.3; without this, the gradient would include ∂w/∂l, changing the optimization objective. The paper offers no stability analysis.
  • domain assumption The MMDetection implementations of RetinaNet, SSD, FoveaBox, and Faster R-CNN are valid baselines and the default hyperparameters are optimal for the baselines.
    All comparisons use MMDetection reimplementations; the improvements are relative to these baselines, so the result depends on these implementations being faithful.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IoU-balanced Loss Functions for Single-stage Object Detection." pith.science (2026). https://pith.science/paper/43IFRRSG

@misc{pith2026190805641,
  author       = {Pith},
  title        = {Pith review of: IoU-balanced Loss Functions for Single-stage Object Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/43IFRRSG}},
  note         = {Machine review of arXiv:1908.05641}
}
abstract

Single-stage object detectors have been widely applied in computer vision applications due to their high efficiency. However, we find that the loss functions adopted by single-stage object detectors hurt the localization accuracy seriously. Firstly, the standard cross-entropy loss for classification is independent of the localization task and drives all the positive examples to learn as high classification scores as possible regardless of localization accuracy during training. As a result, there will be many detections that have high classification scores but low IoU or detections that have low classification scores but high IoU. Secondly, for the standard smooth L1 loss, the gradient is dominated by the outliers that have poor localization accuracy during training. The above two problems will decrease the localization accuracy of single-stage detectors. In this work, IoU-balanced loss functions that consist of IoU-balanced classification loss and IoU-balanced localization loss are proposed to solve the above problems. The IoU-balanced classification loss pays more attention to positive examples with high IoU and can enhance the correlation between classification and localization tasks. The IoU-balanced localization loss decreases the gradient of examples with low IoU and increases the gradient of examples with high IoU, which can improve the localization accuracy of models. Extensive experiments on challenging public datasets such as MS COCO, PASCAL VOC and Cityscapes demonstrate that both IoU-balanced losses can bring substantial improvement for the popular single-stage detectors, especially for the localization accuracy. On COCO test-dev, the proposed methods can substantially improve AP by $1.0\%\sim1.7\%$ and AP75 by $1.0\%\sim2.4\%$. On PASCAL VOC, it can also substantially improve AP by $1.3\%\sim1.5\%$ and AP80, AP90 by $1.6\%\sim3.9\%$.

Figures

Figures reproduced from arXiv: 1908.05641 by the authors.

Figure 1
Figure 1. Speed (ms) versus accuracy (AP) on COCO test-dev. IoU-balanced losses can [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The gradient norm of the standard smooth L1 loss ( [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. (a) Average classification scores of detections within different IoU range. IoU [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Visualization of detection results from the baseline (a,c) and IoU-balanced [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 24 canonical work pages

  1. [1]

    W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, A. C. Berg, Ssd: Single shot multibox detector, in: European conference on computer vision, Springer, 2016, pp. 21–37

  2. [2]

    Redmon, S

    J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You only look once: Unified, real-time object detection, in: Proceedings of the IEEE confer- ence on computer vision and pattern recognition, 2016, pp. 779–788

  3. [3]

    T.-Y. Lin, P. Goyal, R. Girshick, K. He, P. Doll´ ar, Focal loss for dense object detection, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2980–2988

  4. [4]

    Zhang, L

    S. Zhang, L. Wen, X. Bian, Z. Lei, S. Z. Li, Single-shot refinement neural network for object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 4203–4212

  5. [5]

    Zhang, S

    Z. Zhang, S. Qiao, C. Xie, W. Shen, B. Wang, A. L. Yuille, Single- shot object detection with enriched semantics, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 5813–5821

  6. [6]

    B. Li, Y. Liu, X. Wang, Gradient harmonized single-stage detector, in: Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 33, 2019, pp. 8577–8584

  7. [7]

    S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: Towards real-time object detection with region proposal networks, in: Advances in neural information processing systems, 2015, pp. 91–99. 19

  8. [8]

    Z. Cai, N. Vasconcelos, Cascade r-cnn: Delving into high quality object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 6154–6162

Show all 32 references
  1. [9]

    K. He, G. Gkioxari, P. Doll´ ar, R. Girshick, Mask r-cnn, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961– 2969

  2. [10]

    T.-Y. Lin, P. Doll´ ar, R. Girshick, K. He, B. Hariharan, S. Belongie, Feature pyramid networks for object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2117–2125

  3. [11]

    J. Dai, Y. Li, K. He, J. Sun, R-fcn: Object detection via region-based fully convolutional networks, in: Advances in neural information pro- cessing systems, 2016, pp. 379–387

  4. [12]

    Girshick, Fast r-cnn, in: Proceedings of the IEEE international conference on computer vision, 2015, pp

    R. Girshick, Fast r-cnn, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 1440–1448

  5. [13]

    Girshick, J

    R. Girshick, J. Donahue, T. Darrell, J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation, in: Proceed- ings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 580–587

  6. [14]

    Bodla, B

    N. Bodla, B. Singh, R. Chellappa, L. S. Davis, Soft-nms–improving object detection with one line of code, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 5561–5569

  7. [15]

    Gidaris, N

    S. Gidaris, N. Komodakis, Object detection via a multi-region and semantic segmentation-aware cnn model, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 1134–1142

  8. [16]

    Shrivastava, A

    A. Shrivastava, A. Gupta, R. Girshick, Training region-based object detectors with online hard example mining, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 761–769

  9. [17]

    J. Pang, K. Chen, J. Shi, H. Feng, W. Ouyang, D. Lin, Libra r-cnn: Towards balanced learning for object detection, in: Proceedings of the 20 IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 821–830

  10. [18]

    Tychsen-Smith, L

    L. Tychsen-Smith, L. Petersson, Improving object localization with fit- ness nms and bounded iou loss, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 6877–6885

  11. [19]

    Jiang, R

    B. Jiang, R. Luo, J. Mao, T. Xiao, Y. Jiang, Acquisition of localiza- tion confidence for accurate object detection, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 784–799

  12. [20]

    S. Wu, X. Li, X. Wang, Iou-aware single-stage object detector for accu- rate localization, Image and Vision Computing (2019)

  13. [21]

    Huang, L

    Z. Huang, L. Huang, Y. Gong, C. Huang, X. Wang, Mask scoring r- cnn, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 6409–6418

  14. [22]

    K. He, X. Zhang, S. Ren, J. Sun, Spatial pyramid pooling in deep convolutional networks for visual recognition, IEEE transactions on pattern analysis and machine intelligence 37 (2015) 1904–1916

  15. [23]

    T. Kong, F. Sun, H. Liu, Y. Jiang, L. Li, J. Shi, Foveabox: Beyound anchor-based object detection, IEEE Transactions on Image Processing 29 (2020) 7389–7398

  16. [24]

    T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, C. L. Zitnick, Microsoft coco: Common objects in context, in: European conference on computer vision, Springer, 2014, pp. 740– 755

  17. [25]

    Everingham, L

    M. Everingham, L. Van Gool, C. K. Williams, J. Winn, A. Zisserman, The pascal visual object classes (voc) challenge, International journal of computer vision 88 (2010) 303–338

  18. [26]

    Cordts, M

    M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benen- son, U. Franke, S. Roth, B. Schiele, The cityscapes dataset for semantic urban scene understanding, in: Proc. of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2016. 21

  19. [27]

    K. Chen, J. Wang, J. Pang, Y. Cao, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Xu, et al., Mmdetection: Open mmlab detection toolbox and benchmark, arXiv preprint arXiv:1906.07155 (2019)

  20. [28]

    Goyal, P

    P. Goyal, P. Doll´ ar, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, K. He, Accurate, large minibatch sgd: Training imagenet in 1 hour, arXiv preprint arXiv:1706.02677 (2017)

  21. [29]

    Redmon, A

    J. Redmon, A. Farhadi, Yolo9000: better, faster, stronger, in: Proceed- ings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7263–7271

  22. [30]

    Redmon, A

    J. Redmon, A. Farhadi, Yolov3: An incremental improvement, arXiv preprint arXiv:1804.02767 (2018)

  23. [31]

    J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, Y. Wei, Deformable convolutional networks, in: Proceedings of the IEEE international con- ference on computer vision, 2017, pp. 764–773

  24. [32]

    Girshick, I

    R. Girshick, I. Radosavovic, G. Gkioxari, P. Doll´ ar, K. He, Detectron, https://github.com/facebookresearch/detectron, 2018. 22

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.