REVIEW 3 major objections 7 minor 35 references
Object detection confidence should be reported per coordinate, not per box.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 14:45 UTC pith:S7QM7GFM
load-bearing objection A coherent new calibration dimension for detectors, with an evaluation target that is self-defined but not fatal; deserves peer review. the 3 major comments →
Rethinking Detection Calibration: A Coordinate and Direction Perspective
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper claims that coordinate-level calibration is a distinct axis of detector reliability that box-level expected calibration error cannot capture. It proposes CAR, a geometric ratio per coordinate defined as inter_w/(dist_xl + inter_w) for x-coordinates and analogously for y-coordinates, and asserts a perfect-calibration condition E[bar p_t] = hat p_t for each coordinate. The confidence re-encoder maps logits and bounding box features into four calibrated scores, minimizing a negative log-likelihood loss against CAR. A separate directional estimator predicts the sign of each coordinate's deviation, and its output is aggregated with the coordinate confidences to approximate box IoU, whic
What carries the argument
The load-bearing object is the Coordinate-wise Alignment Ratio (CAR), a deterministic geometric ratio that assigns each of the four bounding box coordinates a scalar in [0,1] based on axis-wise intersection length and absolute coordinate difference. CAR acts as the ground-truth accuracy target for the confidence re-encoder, and the paper's central calibration equation equates the expected CAR over bins to the predicted coordinate-wise confidence. A directional displacement estimator (DDE) adds the sign of deviation so that coordinate confidences are augmented with directional information, enabling box-level IoU approximation.
Load-bearing premise
The central assumption is that CAR is the true coordinate-wise probability of correctness; CAR is a deterministic geometric ratio based on overlap lengths, not a statistical frequency over matched ground-truth labels, so every coordinate-level calibration claim is measured against this self-defined quantity.
What would settle it
Take a set of detections grouped by a single ReDC coordinate confidence value, say y2, and compute the empirical fraction of those coordinates satisfying an independent correctness criterion such as |hat y2 - y2| < epsilon pixels; if that fraction consistently differs from the stated confidence while C-ECE computed with CAR is near zero, then the coordinate-level scores are calibrated to CAR but not to task-relevant accuracy. A simpler directional check: for boxes with a predicted direction 'too high', the sign of the actual y2 error should match at the same rate as the estimated direction pro
If this is right
- Deterministic object detectors can produce four independent, direction-aware confidence scores for every box without architectural changes, since ReDC only consumes existing logits and box features.
- Downstream planning systems could use the per-coordinate directional signals to distinguish, for example, a bottom edge that is too high from one that is too low, even when box-level IoU is identical.
- Box-level IoU calibration can be recovered from coordinate-level outputs, so ReDC's representation is a strict superset of prior box-level calibration methods.
- Domain-shift evaluation suggests the method retains coordinate-wise calibration better than box-level calibrators when the detector is applied to corrupted or foggy images.
- The new evaluation metrics, C-ECE and Da-CE, give a standard way to audit fine-grained localization reliability that was previously invisible to box-level ECE.
Where Pith is reading between the lines
- If coordinate-wise calibration is adopted as a standard audit, it could reveal systematic per-edge biases of detectors (e.g., a consistent upward shift in the bottom edge) that box-level ECE will always miss, enabling targeted correction.
- The binary direction prediction could be extended to estimate the magnitude of each coordinate's error, turning the directional signal into a full confidence interval over each edge.
- The CAR geometry is asymmetric and depends on the coordinate's role; a testable extension is whether a detector trained with a coordinate-wise calibration objective would also improve its box-level AP, not just its reported confidence.
- A practical check for deployment: use ReDC's per-coordinate confidences to rank candidate boxes by worst-edge reliability, and see whether this ranking improves recall under a strict localization tolerance compared to ranking by box confidence.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ReDC, a post-hoc calibration framework for deterministic object detectors that outputs per-coordinate confidence scores for the four bounding-box coordinates and predicts the direction of each coordinate's deviation from ground truth. The method defines a coordinate-wise alignment ratio (CAR) as a continuous empirical-accuracy target, trains a confidence re-encoder with a negative-log-likelihood loss to match CAR, and trains a directional displacement estimator with binary cross-entropy. The paper introduces two metrics, C-ECE and Da-CE, and reports experiments on COCO, Cityscapes, COCO-C, and Foggy Cityscapes, as well as across one-stage, two-stage, and transformer detectors. The paper claims that ReDC achieves better coordinate-level calibration than prior box-level methods while maintaining competitive box-level calibration.
Significance. The paper addresses a real gap: existing detection calibration methods summarize localization quality into a single box-level score, whereas coordinate-level and directional errors matter in downstream tasks such as autonomous driving. The post-hoc, model-agnostic design is a strength, and the experimental coverage (in-domain, domain-shift, multiple architectures, and randomized ablations) is more extensive than is typical for such a paper. If the coordinate-level confidence scores were independently validated, the contribution would be significant. However, the primary evaluation metric C-ECE measures consistency with the paper's own CAR target, which is a deterministic geometric ratio rather than a frequency of coordinate correctness. Figure 6 and the GP-Normal correlation provide some independent evidence, but the reported correlations are weak. Thus the central claim is plausible but not yet fully established.
major comments (3)
- [§4.1, Eq. (7), Eq. (8), Eq. (14); Tables 1–4] CAR is used both as the training target (L_cal in Eq. 10) and the evaluation target (C-ECE in Eq. 14). CAR is a deterministic geometric ratio, not an empirical frequency over matched labels, so a low C-ECE shows only that the confidence re-encoder fits this self-defined target. The paper does not provide an independent check that a predicted coordinate confidence of, say, 0.9 matches the empirical frequency that the coordinate is within a task-relevant tolerance (e.g., a few pixels). The y2 correlation analysis and GP-Normal comparison are partial steps, but the reported mean correlation (0.1771) is low. Please add a tolerance-based reliability analysis or an equivalent independent calibration evaluation.
- [§4.2, Eq. (17)] Da-CE sets false-positive errors to zero and computes the metric only over true positives. In detection calibration, false positives are a primary source of overconfidence, and a direction-aware calibration metric that discards them can overstate coordinate-level reliability. Because the direction of a false positive is undefined under the current matching setup, this is not an internal inconsistency, but the manuscript should report the false-positive rate and analyze coordinate confidence on false positives separately, or propose a metric that does not silently zero these errors.
- [§4.1, 'Furthermore...'] The paper claims that box-level IoU can be derived from CAR and directional information, but the derivation is not in the main text and is deferred to the supplementary material. This aggregation is load-bearing for the claim that ReDC 'covers the representational scope of prior calibration approaches' and for the box-level results in Tables 1–4. Please state the aggregation formula and, if an approximation is involved, quantify its error against true IoU (e.g., mean absolute error or a reliability diagram) before relying on it for box-level calibration.
minor comments (7)
- [Notation, Eqs. (8), (11), (15), (17)] The set notation 't∈ {x1.y1, x2, y2}' appears repeatedly and should be 't∈ {x1, y1, x2, y2}'.
- [Eq. (9)] The notation φCR_t(f_i) is ambiguous. Please specify the architecture and output dimensions of the confidence re-encoder, and clarify how the four coordinate-wise scalars are generated from the feature and logit.
- [§4.1, Eq. (13)] The class-specific threshold τ^c_t is said to be 'determined based on the direction accuracy during training', but the selection criterion is not specified. Report how τ is chosen and its sensitivity.
- [Table 5] The DDE ablation improves Cy1-ECE by about 1.0 point but changes the other three coordinates by less than 0.3 points. The claim that DDE improves coordinate-level calibration should be softened or supported by paired significance tests.
- [§5.2, 'Comparison with GP-Normal'] The correlation comparison (0.1771 vs. 0.0263) needs details: number of samples, matching procedure, and confidence intervals. A mean correlation of 0.1771 is low and should be interpreted with care.
- [Tables 1–4 and datasets] The text refers to 'COCO minitest', which is not a standard COCO split. Clarify the exact split used and whether it is custom; also state the number of bins J used for C-ECE and Da-CE.
- [Fig. 6] The axis labels and figure legends are difficult to read in the provided version. Please enlarge fonts and add units for the Euclidean distance.
Circularity Check
C-ECE evaluates agreement with the paper's own CAR target: Eq. 10 trains p_hat to match p_bar and Eq. 14 measures that same match, so low C-ECE is a goodness-of-fit check; independent evidence (Fig. 6, GP-Normal comparison) only partially mitigates.
specific steps
-
self definitional
[Sec. 4.1, Eq. 8, Eq. 10; Sec. 4.2, Eq. 14]
"Based on CAR, we require the following definition to be satisfied at the coordinate-wise level... E_{ˆb∈B(ˆp_t)}[¯p_t] = ˆp_t ... (8) ... ¯p_t denotes the coordinate-wise true probability associated with ˆb as defined in Eq. 7. ... L_cal := E[−(¯p log(ˆp)+(1−¯p) log(1−ˆp))] (10) ... C_t-ECE = 1/C Σ_c Σ_j |B^c_j|/|B^c| |¯p_t(B^c_j)−ˆp_t(B^c_j)| (14)"
The coordinate-wise 'true probability' is not an independently defined frequency over matched labels; it is the deterministic geometric ratio CAR (Eq. 7). The confidence re-encoder is trained with the NLL loss (Eq. 10) to make p_hat approximate p_bar, and C-ECE (Eq. 14) is the binned absolute residual between those same two quantities. Thus a low C-ECE demonstrates that the re-encoder fits the authors' self-defined target, not that the confidence scores are reliable against an external coordinate-error notion. The minival/minitest split shows generalization of the fit to CAR, not the semantic validity of CAR as ground truth. Fig. 6 and the GP-Normal correlation comparison provide partial independent grounding, but the headline coordinate-level calibration claim is measured against its own
full rationale
ReDC's central derivation is not internally inconsistent and is not a direct train/test leak: CR is trained on minival and evaluated on minitest in Tables 1–4, and the box-level IoU approximation is calibrated to actual IoU using isotonic regression and Platt scaling. The partial circularity is semantic rather than statistical: the 'empirical coordinate-wise accuracy' used for both training (Eq. 10) and the new C-ECE metric (Eq. 14) is CAR, a deterministic geometric ratio defined by the authors in Eq. 7, not a frequency over matched labels. C-ECE is effectively the binned L1 version of the same NLL training objective, so low C-ECE shows only that CR is a good regressor to CAR. This would be a severe issue if CAR were the only evidence, but the paper does include some independent checks: Fig. 6 shows ReDC's y2 confidence negatively correlates with pixel distance error, and the GP-Normal comparison measures correlation with coordinate-wise error. These give the central reliability claim external content beyond self-fit. One further scope limitation is acknowledged in Eq. 17: Da-CE sets false-positive errors to zero and uses only true positive samples, so direction-aware calibration is demonstrated only on matched detections, not on the false-positive cases most relevant to autonomous-driving safety. No load-bearing self-citation chain was found; self-citations (e.g., refs. 2 and 11) appear only in related-work contexts. Overall, the coordinate-wise evaluation is partially circular because the metric and the training target are the same self-defined quantity, but the presence of independent correlation checks and held-out evaluation keeps the score moderate.
Axiom & Free-Parameter Ledger
free parameters (3)
- Class-specific direction threshold tau^c_t =
Not reported (selected per class/coordinate during training)
- IoU calibrators (isotonic regression / Platt scaling) =
Not listed
- Per-coordinate bias beta_t in confidence re-encoder =
Learned
axioms (4)
- ad hoc to paper CAR (Eq. 7) is a valid measure of coordinate-wise empirical accuracy / true probability of coordinate correctness.
- domain assumption The sign of the displacement (+1/-1) is sufficient directional information for coordinate-wise calibration.
- ad hoc to paper Da-CE can set false-positive errors to zero without biasing calibration assessment.
- ad hoc to paper IoU can be derived from CAR and directional information.
invented entities (1)
-
Coordinate-wise Alignment Ratio (CAR)
no independent evidence
read the original abstract
Deep learning based object detectors require trustworthiness beyond competitive detection performance, but deep neural networks are prone to overconfident predictions, assigning high confidence scores to predictions that are likely to be inaccurate. To improve the alignment between confidence scores and prediction accuracy, existing methods calibrate confidence scores based on box-level localization, such as precision or intersection over union with the ground truth bounding box. However, box-level localization reflects only a measure of agreement between the predicted box and the ground truth, resulting in calibrated confidence scores for box-level accuracy failing to capture the localization accuracy of coordinates of box. To tackle this issue, we propose a novel post-hoc calibration framework, rethinking detection calibration (ReDC), which provides reliable coordinate-level confidence scores, including directional information. The proposed framework defines coordinate-wise alignment and deviation direction between predictions and ground truth. Based on the alignment measure, confidence re-encoding produces reliable coordinate-level confidence scores, while directional displacement estimation predicts coordinate-wise deviation directions. Extensive experiments under in-domain and out-domain scenarios demonstrate that the proposed approach expresses the coordinate-wise localization of detected objects more precisely than existing methods. Furthermore, our method covers the representational scope of prior calibration approaches by aggregating coordinate-level confidence scores into box-level localization.
Figures
Reference graph
Works this paper leans on
-
[1]
In: Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Cai, Z., Vasconcelos, N.: Cascade r-cnn: Delving into high quality object detection. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
2018
-
[2]
In: The IEEE International Conference on Computer Vision (ICCV) (2019)
Choi, J., Chun, D., Kim, H., Lee, H.J.: Gaussian yolov3: An accurate and fast object detector using localization uncertainty for autonomous driving. In: The IEEE International Conference on Computer Vision (ICCV) (2019)
2019
-
[3]
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding.In:ConferenceonComputerVisionandPatternRecognition(CVPR) (2016)
2016
-
[4]
IEEE Transactions on Intelligent Transportation Systems23(8), 9961–9980 (2021)
Feng, D., Harakeh, A., Waslander, S.L., Dietmayer, K.: A review and comparative study on probabilistic object detection in autonomous driving. IEEE Transactions on Intelligent Transportation Systems23(8), 9961–9980 (2021)
2021
-
[5]
In: International Conference on Machine Learning (ICML) (2017)
Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: International Conference on Machine Learning (ICML) (2017)
2017
-
[6]
In: 2020 IEEE International Conference on Robotics and Automation (ICRA)
Harakeh, A., Smart, M., Waslander, S.L.: Bayesod: A bayesian approach for uncer- tainty estimation in deep object detectors. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). pp. 87–93. IEEE (2020)
2020
-
[7]
In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
2016
-
[8]
In: Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
He, Y., Zhu, C., Wang, J., Savvides, M., Zhang, X.: Bounding box regression with uncertainty for accurate object detection. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
2019
-
[9]
In: International Conference on Learning Represen- tations (ICLR) (2019)
Hendrycks, D., Dietterich, T.: Benchmarking neural network robustness to common corruptions and perturbations. In: International Conference on Learning Represen- tations (ICLR) (2019)
2019
-
[10]
In: Machine learning for health workshop
Jaeger, P.F., Kohl, S.A., Bickelhaupt, S., Isensee, F., Kuder, T.A., Schlemmer, H.P., Maier-Hein, K.H.: Retina u-net: Embarrassingly simple exploitation of segmentation supervision for medical object detection. In: Machine learning for health workshop. pp. 171–183. PMLR (2020)
2020
-
[11]
In: International Conference on Machine Learning (ICML) (2023)
Jung, S., Seo, S., Jeong, Y., Choi, J.: Scaling of class-wise training losses for post-hoc calibration. In: International Conference on Machine Learning (ICML) (2023)
2023
-
[12]
In: The European Conference on Computer Vision (ECCV) (2024)
Kuzucu, S., Oksuz, K., Sadeghi, J., Dokania, P.K.: On calibration of object detectors: Pitfalls, evaluation and baselines. In: The European Conference on Computer Vision (ECCV) (2024)
2024
-
[13]
In: Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2020)
Küppers, F., Kronenberger, J., Shantia, A., Haselhoff, A.: Multivariate confidence calibration for object detection. In: Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2020)
2020
-
[14]
In: European Conference on Com- puter Vision Workshops (ECCVW) (2022)
Küppers, F., Schneider, J., Haselhoff, A.: Parametric and multivariate uncertainty calibration for regression and object detection. In: European Conference on Com- puter Vision Workshops (ECCVW) (2022)
2022
-
[15]
In: European Conference on Computer Vision Workshops (ECCVW) (2022)
Lee, Y., Hwang, J.W., Kim, H.I., Yun, K., Kown, Y., Hwang, S.J.: Localization uncertainty estimation for anchor-free object detection. In: European Conference on Computer Vision Workshops (ECCVW) (2022)
2022
-
[16]
In: European conference on computer vision (ECCV) (2014)
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision (ECCV) (2014)
2014
-
[17]
In: Conference on Computer Vision and Pattern Recognition (CVPR) (2018) Rethinking Detection Calibration: A Coordinate and Direction Perspective 17
Liu, Y., Jourabloo, A., Liu, X.: Learning deep models for face anti-spoofing: Bi- nary or auxiliary supervision. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2018) Rethinking Detection Calibration: A Coordinate and Direction Perspective 17
2018
-
[18]
Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
Munir, M.A., Khan, M.H., Khan, S., Khan, F.: Bridging precision and confidence: A train-time loss for calibrating object detection. Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
2023
-
[19]
Advances in Neural Information Processing Systems (NeurIPS) (2022)
Munir, M.A., Khan, M.H., Sarfraz, M., Ali, M.: Towards improving calibration in object detection under domain shift. Advances in Neural Information Processing Systems (NeurIPS) (2022)
2022
-
[20]
Advances in Neural Information Processing Systems (NeurIPS (2023)
Munir, M.A., Khan, S., Khan, M.H., Ali, M., Khan, F.: Cal-detr: Calibrated detec- tion transformer. Advances in Neural Information Processing Systems (NeurIPS (2023)
2023
-
[21]
In: European Conference on Computer Vision (ECCV) (2018)
Oksuz, K., Cam, B., Akbas, E., Kalkan, S.: Localization recall precision (lrp): A new performance metric for object detection. In: European Conference on Computer Vision (ECCV) (2018)
2018
-
[22]
In: Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
Oksuz, K., Joy, T., Dokania, P.K.: Towards building self-aware object detectors via reliable uncertainty quantification and calibration. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
2023
-
[23]
In: Advances in Neural Information Processing Systems (NeurIPS) (2021)
Pan, T.Y., Zhang, C., Li, Y., Hu, H., Xuan, D., Changpinyo, S., Gong, B., Chao, W.L.: On model calibration for long-tailed object detection and instance segmenta- tion. In: Advances in Neural Information Processing Systems (NeurIPS) (2021)
2021
-
[24]
In: Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
Pathiraja, B., Gunawardhana, M., Khan, M.H.: Multiclass confidence and local- ization calibration for object detection. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
2023
-
[25]
Advances in large margin classifiers10(3), 61–74 (1999)
Platt, J., et al.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers10(3), 61–74 (1999)
1999
-
[26]
International Journal of Computer Vision126(9), 973–992 (2018)
Sakaridis, C., Dai, D., Van Gool, L.: Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision126(9), 973–992 (2018)
2018
-
[27]
In: International Conference on Machine Learning (ICML) (2019)
Song, H., Diethe, T., Kull, M., Flach, P.: Distribution calibration for regression. In: International Conference on Machine Learning (ICML) (2019)
2019
-
[28]
In: In European Conference on Computer Vision (ECCV) (2022)
Tomani, C., Cremers, D., Buettner, F.: Parameterized temperature scaling for boosting the expressive power in post-hoc uncertainty calibration. In: In European Conference on Computer Vision (ECCV) (2022)
2022
-
[29]
In: Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2017)
Wu, B., Iandola, F., Jin, P.H., Keutzer, K.: Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving. In: Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2017)
2017
-
[30]
In: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining
Zadrozny, B., Elkan, C.: Transforming classifier scores into accurate multiclass probability estimates. In: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 694–699 (2002)
2002
-
[31]
In: International Conference on Learning Representations (ICLR) (2023)
Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L., Shum, H.Y.: DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In: International Conference on Learning Representations (ICLR) (2023)
2023
-
[32]
In: Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
Zhang, H., Wang, Y., Dayoub, F., Sünderhauf, N.: Varifocalnet: An iou-aware dense object detector. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
2021
-
[33]
In: International Conference on Machine Learning (ICML) (2020)
Zhang, J., Kailkhura, B., Han, T.: Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In: International Conference on Machine Learning (ICML) (2020)
2020
-
[34]
IEEE transactions on medical imaging39(7), 2531–2540 (2020) 18 Lee et al
Zhang, L., Wang, X., Yang, D., Sanford, T., Harmon, S., Turkbey, B., Wood, B.J., Roth, H., Myronenko, A., Xu, D., et al.: Generalizing deep learning for medical image segmentation to unseen domains via deep stacked transformation. IEEE transactions on medical imaging39(7), 2531–2540 (2020) 18 Lee et al
2020
-
[35]
In: International Conference on Learning Representations (ICLR) (2021)
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable detr: Deformable trans- formers for end-to-end object detection. In: International Conference on Learning Representations (ICLR) (2021)
2021
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.