Pith. sign in

REVIEW 2 major objections 2 minor 24 references

DnD compares two object detectors by taking set intersections and differences on their shared ground-truth matches.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-27 22:15 UTC pith:FF7DNG3N

load-bearing objection DnD is a straightforward set-difference procedure on top of standard matching that gives a direct view of shared and unique detections between two models. the 2 major comments →

arxiv 2606.07503 v1 pith:FF7DNG3N submitted 2026-06-05 cs.CV

Differences in Detection: Explainability Where it Matters

Johannes Theodoridis , Johannes Maucher , Andreas Schilling This is my paper
classification cs.CV
keywords object detectionmodel comparisonexplainabilitymean average precisionerror analysisconfusion matrixground truth matchingTIDE
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Differences in Detection (DnD) to compare two object detection models more directly than separate summary scores allow. Starting from the same matching algorithm used for standard metrics, it identifies the ground-truth instances both models detect correctly, the instances only one model detects, and the instances both models miss. This produces explicit shared and unique success and failure sets instead of independent mAP or TIDE numbers. When the same error typing is applied, the differences appear inside a confusion matrix that isolates where the models diverge in their mistakes. The resulting subsets can also steer explainability methods toward examples that actually affect the reported performance numbers.

Core claim

DnD calculates the intersection of ground truth labels recognized by both models, followed by the corresponding difference sets and the complement set of ground truth labels missed by both models, providing a more direct and intuitive comparison than independent summary statistics.

What carries the argument

Differences in Detection (DnD), which applies set intersection, difference, and complement operations to the ground-truth matches obtained from the same matching algorithm run on each model.

Load-bearing premise

That performing set differences on the matched ground-truth instances produces insights that are meaningfully more actionable or less misleading than comparing separate mAP and TIDE scores.

What would settle it

A controlled user study in which practitioners using only DnD outputs select or debug models faster or more accurately than practitioners given only mAP and TIDE scores.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Reveals which detections are shared successes, unique to one model, or shared failures.
  • Allows detection errors to be analyzed in a standard confusion matrix once error types are attached.
  • Supplies structured subsets of examples that explainability methods can target because they affect the metrics.
  • Complements rather than replaces mAP and TIDE by adding instance-level overlap information.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same set-construction logic could be applied to more than two models by computing multi-way intersections and symmetric differences.
  • The resulting subsets could serve as a natural training or validation split when the goal is to improve one model relative to another.
  • Because the method re-uses the existing matching step, it adds negligible computational cost while changing only the post-processing view of the results.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper proposes Differences in Detection (DnD), a method to compare two object detection models by applying the same matching algorithm and then computing the intersection of ground-truth labels detected by both, the symmetric difference sets for each model, and the complement of labels missed by both. This set-algebraic view is positioned as complementary to mAP and TIDE, with natural extension to error-type confusion matrices, and is suggested as a way to select metric-relevant examples for explainability techniques such as ODAM.

Significance. If the claimed intuitiveness and actionability hold, DnD would supply a lightweight, structured reorganization of existing matching outputs that could help practitioners isolate shared versus model-specific errors without replacing summary metrics. The approach is parameter-free and directly leverages standard IoU-based assignment, which is a strength, but its practical utility remains untested.

major comments (2)
  1. [Abstract] Abstract: the central claim that the resulting comparison 'is more direct and intuitive than a comparison of independent summary statistics' and 'reveals individual and shared mistakes' is asserted without any quantitative evaluation, user study, ablation, or concrete example showing that DnD changes conclusions drawn from mAP/TIDE; this is load-bearing for the paper's positioning as a useful complement.
  2. [Method / Error-type analysis] Method description (inferred from abstract and § on error types): while the set operations are a direct reorganization of standard matching outputs, the manuscript supplies no demonstration that the intersection/difference sets yield more actionable insights for guiding ODAM or other explainability methods than simply inspecting per-model TIDE error breakdowns.
minor comments (2)
  1. The GitHub link is provided, supporting reproducibility; however, the manuscript does not indicate whether the released code includes the exact matching routine used in the reported comparisons.
  2. Notation for the intersection, difference, and complement sets is described procedurally but not formalized with equations or pseudocode, which would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive report. The two major comments correctly identify that the manuscript's positioning relies on claims of intuitiveness and actionability that are not yet supported by concrete examples or demonstrations. We address each point below and will revise the manuscript to incorporate the requested evidence.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that the resulting comparison 'is more direct and intuitive than a comparison of independent summary statistics' and 'reveals individual and shared mistakes' is asserted without any quantitative evaluation, user study, ablation, or concrete example showing that DnD changes conclusions drawn from mAP/TIDE; this is load-bearing for the paper's positioning as a useful complement.

    Authors: We agree that the abstract's claims would be strengthened by at least one concrete example. The current manuscript presents DnD as a direct reorganization of standard matching outputs but does not show an instance where the intersection/symmetric-difference sets lead to different practitioner conclusions than separate mAP and TIDE scores. We will add such an example (e.g., two detectors with similar mAP but visibly different shared vs. model-specific errors) in a revised experiments or applications section and will adjust the abstract wording to reflect the addition. revision: yes

  2. Referee: [Method / Error-type analysis] Method description (inferred from abstract and § on error types): while the set operations are a direct reorganization of standard matching outputs, the manuscript supplies no demonstration that the intersection/difference sets yield more actionable insights for guiding ODAM or other explainability methods than simply inspecting per-model TIDE error breakdowns.

    Authors: We accept this observation. Although the set-algebraic view is parameter-free and directly reuses IoU matching, the manuscript contains no side-by-side illustration showing that the DnD subsets produce more targeted example selection for ODAM than TIDE error-type breakdowns alone. We will add a short illustrative case study in the revision that applies both approaches to the same pair of detectors and highlights any differences in the resulting explainability focus. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper defines DnD as a direct procedural computation of set intersections, differences, and complements on ground-truth matches obtained from any standard matching algorithm (e.g., IoU-based assignment). No equations, fitted parameters, or derived quantities are introduced; the method is explicitly complementary to mAP and TIDE rather than a replacement metric. No self-citations, uniqueness theorems, or ansatzes are invoked as load-bearing steps. The derivation is therefore self-contained and does not reduce to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms, or invented entities are introduced or required by the abstract description.

pith-pipeline@v0.9.1-grok · 5701 in / 1024 out tokens · 14348 ms · 2026-06-27T22:15:07.531883+00:00 · methodology

0 comments
read the original abstract

We propose Differences in Detection (DnD), an intuitive method to compare two object detection models. Based on the same matching algorithm, it complements the standard metrics of mean Average Precision ($mAP$) and TIDE error analysis with the ability to compare two models directly. More specifically, we calculate the intersection of ground truth labels that are recognized by both models, followed by the corresponding difference sets and the complement set of ground truth labels that are missed by both models. The resulting comparison is more direct and intuitive than a comparison of independent summary statistics. It reveals individual and shared mistakes and becomes particularly interesting when combined with error types. In this case, the differences in detection errors can be analyzed naturally in a standard confusion matrix. While valuable in itself, we believe that one of the best applications of DnD is to guide explainability methods such as ODAM towards metric-relevant examples, grounded in structured subsets. The code for our method is available here: https://github.com/JohannesTheo/differences-in-detection

Figures

Figures reproduced from arXiv: 2606.07503 by Andreas Schilling, Johannes Maucher, Johannes Theodoridis.

Figure 1
Figure 1. Figure 1: For the predictions of two models we calculate the inter [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Differences in Detection for two Mask R-CNN models [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of summary metrics and TIDE errors for [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Example from the set of individual errors of ViTDet. [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 7
Figure 7. Figure 7: Example from the set of individual errors on COCO-C. [PITH_FULL_IMAGE:figures/full_fig_p004_7.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of summary metrics and TIDE errors for [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Differences in Detection for Mask R-CNN with FAN-S [PITH_FULL_IMAGE:figures/full_fig_p004_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 2 canonical work pages

  1. [1]

    TIDE: A General Toolbox for Identifying Object Detection Errors

    Daniel Bolya, Sean Foley, James Hays, and Judy Hoffman. TIDE: A General Toolbox for Identifying Object Detection Errors. InComputer Vision – ECCV 2020: 16th Euro- pean Conference, Glasgow, UK, August 23–28, 2020, Pro- ceedings, Part III, pages 558–573, Berlin, Heidelberg, 2020. Springer-Verlag. 1, 2

    2020

  2. [2]

    Empirical Upper Bound in Object Detection and More, 2019

    Ali Borji and Seyed Mehdi Iranmanesh. Empirical Upper Bound in Object Detection and More, 2019. arXiv:1911.12451 [cs]. 1

    work page arXiv 2019

  3. [3]

    Grad-CAM++: General- ized Gradient-Based Visual Explanations for Deep Convolu- tional Networks

    Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-CAM++: General- ized Gradient-Based Visual Explanations for Deep Convolu- tional Networks. In2018 IEEE Winter Conference on Appli- cations of Computer Vision (WACV), pages 839–847, 2018. 2

    2018

  4. [4]

    Berg, and Alexander Kirillov

    Bowen Cheng, Ross Girshick, Piotr Dollar, Alexander C. Berg, and Alexander Kirillov. Boundary IoU: Improving Object-Centric Image Segmentation Evaluation. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15334–15342, 2021. 2

    2021

  5. [5]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. InInternational Conference on Learning Representa- tions, 2021. 3

    2021

  6. [6]

    Cubuk, Quoc V

    Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D. Cubuk, Quoc V . Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 2918–2928, 2021. 3

    2021

  7. [7]

    Mask R-CNN

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask R-CNN. In2017 IEEE International Confer- ence on Computer Vision (ICCV), pages 2980–2988, 2017. 3

    2017

  8. [8]

    Benchmarking Neu- ral Network Robustness to Common Corruptions and Pertur- bations

    Dan Hendrycks and Thomas Dietterich. Benchmarking Neu- ral Network Robustness to Common Corruptions and Pertur- bations. InInternational Conference on Learning Represen- tations, 2019. 4

    2019

  9. [9]

    Exploring Plain Vision Transformer Backbones for Object Detection

    Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring Plain Vision Transformer Backbones for Object Detection. InComputer Vision – ECCV 2022, pages 280– 296, Cham, 2022. Springer Nature Switzerland. 3

    2022

  10. [10]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. InComputer Vision – ECCV 2014, pages 740–755, Cham,

    2014

  11. [11]

    Springer International Publishing. 1, 2

  12. [12]

    Michaelis, B

    C. Michaelis, B. Mitzkus, R. Geirhos, E. Rusak, O. Bring- mann, A. S. Ecker, M. Bethge, and W. Brendel. Bench- marking robustness in object detection: Autonomous driving when winter is coming. InMachine learning for autonomous driving workshop, NeurIPS 2019, 2019. 4

    2019

  13. [13]

    Faster-COCO-Eval: Faster interpretation of the original COCOEval, 2024

    MiXaiLL76. Faster-COCO-Eval: Faster interpretation of the original COCOEval, 2024. 1

    2024

  14. [14]

    Morariu, Ashutosh Mehra, Vicente Ordonez, and Kate Saenko

    Vitali Petsiuk, Rajiv Jain, Varun Manjunatha, Vlad I. Morariu, Ashutosh Mehra, Vicente Ordonez, and Kate Saenko. Black-box Explanation of Object Detectors via Saliency Maps. In2021 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 11438– 11447, 2021. 2

    2021

  15. [15]

    Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Ba- tra

    Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Ba- tra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In2017 IEEE Interna- tional Conference on Computer Vision (ICCV), pages 618– 626, 2017. 2

    2017

  16. [16]

    Benchmarking Object De- tectors with COCO: A New Path Forward

    Shweta Singh, Aayan Yadav, Jitesh Jain, Humphrey Shi, Justin Johnson, and Karan Desai. Benchmarking Object De- tectors with COCO: A New Path Forward. InComputer Vi- sion – ECCV 2024, pages 279–295, Cham, 2025. Springer Nature Switzerland. 2

    2024

  17. [17]

    Trapped in Texture Bias? A Large Scale Comparison of Deep Instance Segmentation

    Johannes Theodoridis, Jessica Hofmann, Johannes Maucher, and Andreas Schilling. Trapped in Texture Bias? A Large Scale Comparison of Deep Instance Segmentation. InCom- puter Vision – ECCV 2022, pages 609–627, Cham, 2022. Springer Nature Switzerland. 4

    2022

  18. [18]

    ConvNeXt V2: Co-Designing and Scaling ConvNets With Masked Autoencoders

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xin- lei Chen, Zhuang Liu, In So Kweon, and Saining Xie. ConvNeXt V2: Co-Designing and Scaling ConvNets With Masked Autoencoders. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16133–16142, 2023. 3

    2023

  19. [19]

    Explaining Object Detection Through Difference Map

    Shujun Xia, Chenyang Zhao, and Antoni Chan. Explaining Object Detection Through Difference Map. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion (ICCV) Workshops, pages 713–722, 2025. 2

    2025

  20. [20]

    Spatial Sensitive Grad-CAM++: Improved Visual Explanation for Object Detectors via Weighted Combination of Gradient Map

    Toshinori Yamauchi. Spatial Sensitive Grad-CAM++: Improved Visual Explanation for Object Detectors via Weighted Combination of Gradient Map. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 8164–8168, 2024

    2024

  21. [21]

    Spatial Sensi- tive GRAD-CAM: Visual Explanations for Object Detection by Incorporating Spatial Sensitivity

    Toshinori Yamauchi and Masayoshi Ishikawa. Spatial Sensi- tive GRAD-CAM: Visual Explanations for Object Detection by Incorporating Spatial Sensitivity. In2022 IEEE Interna- tional Conference on Image Processing (ICIP), pages 256– 260, 2022

    2022

  22. [22]

    Chenyang ZHAO and Antoni B. Chan. ODAM: Gradient- based Instance-Specific Visual Explanations for Object De- tection. InThe Eleventh International Conference on Learn- ing Representations, 2023. 2

    2023

  23. [23]

    Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Ani- mashree Anandkumar, Jiashi Feng, and Jose M. Alvarez. Un- derstanding The Robustness in Vision Transformers. InPro- ceedings of the 39th International Conference on Machine Learning, pages 27378–27394. PMLR, 2022. 4

    2022

  24. [24]

    Benchmarking a Benchmark: How Reliable is MS-COCO? InICCV 2023 DataComp Workshop, 2023

    Eric Zimmermann, Justin Szeto, Jerome Pasquero, and Fred- eric Ratle. Benchmarking a Benchmark: How Reliable is MS-COCO? InICCV 2023 DataComp Workshop, 2023. eprint: 2311.02709. 2 5

    work page arXiv 2023