{"id":"46851a0a-3307-4ebd-9bc8-17a04f5e8cbc","arxiv_id":"2504.19249","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A structured evaluation of three object-detection XAI methods shows that faithfulness, localization, and speed are in tension, with D-CLOSE most faithful, G-CAME fastest and most localized, and D-RISE in between.","lead":"This paper proposes ODExAI, a framework for evaluating explainable AI methods in object detection, and demonstrates it by comparing three XAI methods on two detectors and two datasets. It finds clear trade-offs: some methods are more faithful to the model, while others are faster or better at localizing objects.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 5.2 concedes the framework's metrics are unnormalized and 'can be misleading'; Table 1's raw PG/OA/EBPG/Sparsity/Time values are on incompatible scales, so the claimed trade-offs and 'reliably measures' conclusion may be scale artifacts.","rationale":"The reader's weakest assumption centers on missing variance and metric validity. I agree with the conditional verdict but want to sharpen the most load-bearing issue. The paper's own Section 5.2 explicitly concedes that the framework lacks a proper normalization scheme and that this 'can be misleading' when metrics differ in scale or unit. That concession directly undercuts Section 4.3's claim that ODExAI 'reliably measures' explanation quality and informs method selection. The Table 1 metrics are on incompatible scales, and Figure 5 plots them together, so the headline trade-offs might be an artifact of raw-scale differences. This is a correctness risk about the central claim, not a disagreement with consensus. The same limitation motivates the absence of a random baseline: without knowing the null value of each metric, high scores such as OA=0.881 or PG=96.13% are not calibrated. A skill-score normalization test would settle whether the rankings survive. Because the framework is explicitly a demonstrative toy experiment and the authors already identify the fix, my stress-test does not change the reader's conditional verdict; it strengthens the conditions under which the paper should be accepted.","tokens_in":10688,"tokens_out":7374,"duration_ms":72966,"concrete_test":"Implement the skill score suggested in Section 5.2: for each metric m, define S=(m_raw - m_random)/(m_ideal - m_random), with m_random obtained from a uniform/random saliency map and m_ideal as the metric's oracle value; then recompute Table 1 for all 12 cells. Check whether, after this normalization, G-CAME still dominates localization and D-CLOSE still dominates faithfulness, and whether D-CLOSE vs D-RISE OA on PASCAL VOC/YOLOX remains distinguishable across five random seeds. If rankings change materially, the current conclusions are artifacts of raw scale and the 'reliably measures' claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 4.3 that ODExAI 'reliably measures explanation quality and informs method selection' requires the quantitative comparisons in Table 1 to be meaningful across dimensions. They are not, in their current form, for reasons the authors themselves state in Section 5.2: the framework 'lacks a proper normalization scheme, which can be misleading when metrics differ in scale or unit' and mixes higher-better and lower-better metrics. Table 1 mixes OA (a difference of two AUC values, roughly in [-1,1]), PG and EBPG (percentages), Sparsity (a ratio Smax/Smean after min-max normalization, with unbounded upper range), and runtime in seconds whose magnitude depends on implementation and hardware. The spider plot in Figure 5 plots these raw values together, so scale differences, not method quality, can dominate the shape. The skill-score alternative cited as reference [5] is absent, so there is no random baseline; a method can appear 'good' simply because the metric's null value is not near zero. This is not a cosmetic limitation: without normalization or a reference baseline, the trade-off story (D-CLOSE faithful but slow, G-CAME localized but less faithful) may be an artifact of how raw numbers are presented. Additionally, the numbers come from a single run, with no seed variance reported for the stochastic D-CLOSE and D-RISE methods, though the normalization issue is the more fundamental threat to the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ODExAI, a framework for evaluating explainable AI (XAI) methods for object detection along three dimensions: localization accuracy (PG, EBPG), faithfulness to the model (Deletion/Insertion, OA), and computational complexity (Sparsity, runtime). The framework is demonstrated in a 'toy experiment' comparing D-CLOSE, G-CAME, and D-RISE on YOLOX and Faster R-CNN using MS-COCO and PASCAL VOC, with results reported in Table 1 and visualized as spider plots in Figure 5. The central empirical claim, stated in Section 4.3, is that ODExAI 'reliably measures explanation quality and informs method selection,' leading to conclusions that D-CLOSE is most faithful but slow, G-CAME is best localized and fastest but less faithful, and D-RISE is moderate across all dimensions.","tokens_in":10970,"tokens_out":4160,"duration_ms":41784,"significance":"If the central claim were fully supported, ODExAI would be a useful standardized benchmark for comparing XAI methods in object detection, a domain where no such consensus framework exists. The paper's strengths are the use of standard, correctly cited metrics (PG, EBPG, Deletion/Insertion, OA, Sparsity) and the public release of code at the provided GitHub URL. However, as it stands, the paper delivers a demonstration rather than a validated evaluation standard: the empirical evidence consists of single point estimates from a self-described toy experiment, with no error bars, sample sizes, statistical tests, or seed variance reported, and Section 5.2 concedes that the framework lacks a proper normalization scheme, which can be misleading when metrics differ in scale or unit. These issues directly affect the load-bearing claim that ODExAI reliably measures explanation quality, so the contribution is currently a framework proposal with illustrative results rather than a fully supported empirical benchmark.","major_comments":[{"comment":"The central claim that ODExAI 'reliably measures explanation quality and informs method selection' is not supported by the reported evidence. Each cell of Table 1 is a single point estimate with no variance, sample size, or statistical test; D-CLOSE and D-RISE are stochastic methods using 2000 random perturbation masks, so seed variance should be reported to establish that the observed differences are not noise. The paper itself labels the study a 'toy experiment' in the contribution list, which further weakens the basis for the reliability claim.","section":"Section 4.3, Table 1"},{"comment":"The framework's own stated limitation undermines the trade-off analysis. Section 5.2 admits that the framework 'lacks a proper normalization scheme, which can be misleading when metrics differ in scale or unit' and that metrics mix higher-better and lower-better directions. Since Table 1 and Figure 5 plot raw OA (an AUC difference), PG/EBPG percentages, Sparsity (a ratio with unbounded upper range), and runtime in seconds on the same spider plot, the visual shape and the resulting trade-off narrative (e.g., D-CLOSE faithful but slow, G-CAME localized but less faithful) may be artifacts of scale rather than of method quality. The skill-score baseline cited as reference [5] is mentioned only as future work, so no random or reference baseline is used to put the raw values in context.","section":"Section 5.2, Figure 5"},{"comment":"The localization metrics PG and EBPG assume that the ground-truth bounding box is the correct region of interest for an explanation, but this assumption is not validated. Object-detection explanations may reasonably include contextual or surrounding pixels that contribute to the model's decision, so restricting localization to the bounding box could systematically disadvantage methods that capture context. Since localization is one of the three headline dimensions, the choice should be justified empirically, for example by a sensitivity analysis using different ROI definitions, or explicitly discussed as a limitation with a concrete test.","section":"Section 3.2"}],"minor_comments":[{"comment":"The PG row contains concatenated values without delimiters, such as '94.3186.55' and '96.1384.12', making the table difficult to read; the values should be separated clearly.","section":"Table 1"},{"comment":"DETR is cited as reference [57], but [57] is the FCOS paper; DETR corresponds to the Carion et al. reference currently numbered [9], and Deformable DETR should have its own distinct citation.","section":"Section 2.1"},{"comment":"References [4] and [5] are identical (both Bommer et al., 'Finding the right XAI method'), which creates a duplicate entry; the duplicate should be merged and the in-text citation for the skill score corrected accordingly.","section":"References"},{"comment":"The proposed solution of 'assign weights' to handle higher-better versus lower-better metrics is only a sentence; the paper should specify how the weights would be determined so that they do not introduce a new arbitrary degree of freedom.","section":"Section 5.2"},{"comment":"The spider plot axes are not labeled with units or any normalization information; without such labels, the visualization cannot be interpreted across metrics with different physical dimensions.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is better framed as a framework proposal and demonstration than as a fully validated evaluation benchmark; the authors' own Section 5.2 limitations directly undercut the strong wording of the Section 4.3 claim. The reproducibility elements (public code, standard metrics) are positive, but the empirical basis needs substantial strengthening before publication. Also note the duplicate references [4]/[5] and the need to reconcile the 'toy experiment' phrasing with the generality of the conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a sensible integration of existing XAI-for-detection metrics into one framework, with a public web tool and a small benchmark. The paper's own Section 5.2 undercuts the 'reliably measures' conclusion in Section 4.3, and the toy experiment lacks error bars, so treat the trade-off story as illustrative rather than proven.\n\nWhat's actually new: the framework itself is a combination, not a new theory. The authors are honest about that. The useful part is having the metrics (PG, EBPG, Del/Ins, OA, Sparsity, Time) organized under three dimensions, with equations correctly cited, and an actual tool and code release. The benchmark comparing D-CLOSE, G-CAME, D-RISE across two detectors and two datasets is new as a concrete data point. Credit also for explicitly calling the experiment a toy and for flagging normalization as a known issue.\n\nSoft spots: the main one is that Table 1 and the spider plot mix values on incompatible scales (percentages, AUC differences, a ratio, seconds). The stress-test note is right: the shapes in Figure 5 can be driven by units, not by method quality. The authors admit this in 5.2 but still state in 4.3 that ODExAI 'reliably measures explanation quality.' That's a mismatch. Also, every cell is a single point estimate; D-CLOSE and D-RISE are stochastic, and there's no seed variance, no sample size, no statistical test, no random baseline. The skill-score fix they mention is the right direction, but it's not implemented. Minor: reference [4] and [5] are the same paper, and runtime comparisons depend on hardware.\n\nWho this is for: people working on XAI for object detection who want a quick way to structure an evaluation. It's more of a workshop/technical report level contribution than a definitive benchmark. I'd engage with it as a referee: the framework is worth building on, but the claims need to be scaled back to match the evidence, and the paper needs at least per-seed variance and a normalized/baseline version of the metrics.","headline":"A useful integration of existing XAI-for-detection metrics, shipped as a tool and a small benchmark, but the empirical support for the 'reliably measures' claim is thin and partly undercut by the paper's own limitations section.","tokens_in":11509,"tokens_out":1990,"would_cite":false,"duration_ms":20752,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes ODExAI, a three-axis benchmark for explainable AI in object detection, and uses it to show that D-CLOSE is the most faithful explainer, G-CAME the best localized and fastest, and D-RISE moderate on every axis.","keywords":["explainable AI","object detection","evaluation framework","saliency maps","faithfulness","localization accuracy","computational complexity","XAI benchmarking"],"falsifier":"Recompute the same benchmark with multiple random seeds and per-image distributions: if the gaps reported in the paper, such as D-CLOSE's OA of 0.881 versus G-CAME's 0.644 on MS-COCO/YOLOX or G-CAME's PG of 94.31% versus D-CLOSE's 87.86%, overlap under resampling, the claimed trade-offs would not be established.","tokens_in":10482,"feed_emoji":"🎯","tokens_out":11944,"duration_ms":106560,"temperature":0.7,"pith_summary":"ODExAI is an evaluation framework that scores explainable AI (XAI) methods for object detectors on three dimensions: localization accuracy, faithfulness to the model's behavior, and computational complexity. The paper demonstrates it in a toy experiment with three explanation methods—D-CLOSE, G-CAME, and D-RISE—across two detectors (YOLOX and Faster R-CNN) and two datasets (MS-COCO and PASCAL VOC). The central empirical finding is that no method wins on all axes: D-CLOSE is the most faithful and produces sparse maps but needs over a minute per image, G-CAME localizes best and runs in under a second but is less faithful, and D-RISE is consistently moderate. Because the ranking changes by dimension, the paper argues that task-specific evaluation, rather than a single headline score, is required before choosing an explanation method.","feed_headline":"D-CLOSE is faithful, G-CAME is fast: XAI benchmark exposes trade-offs","feed_subtitle":"Three measurable dimensions—localization, faithfulness, complexity—separate explanation methods on MS-COCO and PASCAL VOC.","key_machinery":"The central object is the metric grid: five quantitative scores grouped into three dimensions—Pointing Game and Energy-based Pointing Game for localization, Deletion, Insertion and Over-All for faithfulness, and Sparsity plus runtime for complexity. Each saliency map is scored on every axis, producing a table that exposes the trade-offs. The load-bearing identities are the formulas themselves: $\\mathrm{PG}$ counts whether the maximum-saliency point lands inside the annotated bounding box, $\\mathrm{EBPG}$ is the fraction of saliency energy inside the box, $\\mathrm{OA} = \\mathrm{AUC}(\\text{Insertion}) - \\mathrm{AUC}(\\text{Deletion})$ combines the two perturbation curves, and Sparsity is the ratio $S_{\\max}/S_{\\mathrm{mean}}$ of the normalized map. The framework's argument runs by assembling these scores into a single comparison table and reading method selection off the resulting trade-offs.","core_discovery":"On the paper's own terms, ODExAI reliably measures explanation quality and informs method selection for specific detection tasks. Concretely, the benchmark shows a structured trade-off: D-CLOSE achieves the highest faithfulness, with Over-All (OA) values such as 0.881 on MS-COCO with YOLOX and 0.863 with Faster R-CNN, at the cost of runtimes around 70 seconds; G-CAME achieves the best localization, with Pointing Game scores up to 96.13% and Energy-based Pointing Game scores up to 70.11%, and the lowest runtime (0.54s), but lower faithfulness (OA 0.549 in the same Faster R-CNN setting) and more diffuse saliency maps; D-RISE stays moderate across all dimensions without leading on the main higher-is-better metrics. The paper interprets these results as evidence that region-based methods are better suited to auditing and debugging, while CAM-based methods fit real-time, localization-focused deployments.","pith_inferences":["Editorial inference: re-running the same evaluation with segmentation masks or human-annotated regions of interest instead of bounding boxes could shift the localization rankings, especially G-CAME's lead, and this is directly testable with the paper's released benchmark.","Editorial inference: because each cell in the benchmark is a single number, resampling across images and random seeds and reporting confidence intervals would show which of the advertised gaps are stable; I would expect some gaps to shrink substantially.","Editorial inference: applying ODExAI to transformer-based detectors, where attention maps play a different role than convolutional feature maps, would test whether the low faithfulness of CAM-based methods is a general property or an artifact of the two detectors benchmarked here."],"forward_implications":["If the central claim is right, a real-time object detection pipeline should prefer a CAM-based explainer like G-CAME when localization is the priority, accepting lower faithfulness.","For auditing or debugging, where the explanation must track the model's actual reasoning, a region-based method like D-CLOSE is the defensible choice despite its high runtime.","Reporting a single evaluation metric, such as faithfulness alone, would mislead: the three dimensions must be reported together to see the trade-off profile.","ODExAI provides a repeatable protocol: any new explainer can be inserted and scored on the same five metrics across the same detectors and datasets, making comparisons standardizable.","A method that is moderate on every axis, such as D-RISE, remains useful as a fallback when the target dimension is unspecified or when explanation diversity is valued."],"supporting_citations":[{"why":"defines D-CLOSE, the region-based method whose faithfulness performance anchors the benchmark","marker":"[58]"},{"why":"defines G-CAME, the CAM-based method that anchors the localization and runtime side of the trade-off","marker":"[40]"},{"why":"defines D-RISE, the moderate method used as the third benchmarked explainer","marker":"[45]"},{"why":"supplies the Pointing Game metric used to measure localization accuracy","marker":"[64]"},{"why":"cited as the source of the Energy-based Pointing Game metric used for localization","marker":"[59]"},{"why":"supplies the Deletion and Insertion perturbation metrics used for faithfulness","marker":"[44]"},{"why":"supplies the Over-All metric, defined as insertion AUC minus deletion AUC","marker":"[65]"},{"why":"supplies the Sparsity metric used for the complexity dimension","marker":"[16]"},{"why":"provides the MS-COCO dataset used in the benchmark","marker":"[26]"},{"why":"provides the PASCAL VOC dataset used in the benchmark","marker":"[13]"}],"fun_headline_variants":["Explainability trade-offs quantified: D-CLOSE faithful, G-CAME fast","ODExAI benchmark: faithfulness vs runtime trade-off for XAI","Region-based XAI: faithful but slow; CAM-based: fast but less faithful","New XAI evaluation framework: choose methods by task, not habit","ODExAI: Object detection explainability benchmark reveals trade-offs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the assumption that the five selected metrics, each computed once per setting, accurately capture whether an explanation is localized, faithful, and simple, and that the bounding box is the correct spatial target for explanation quality.","fun_headline_variants_meta":{"raw":{"variants":["Explainability trade-offs quantified: D-CLOSE faithful, G-CAME fast","ODExAI benchmark: faithfulness vs runtime trade-off for XAI","Region-based XAI: faithful but slow; CAM-based: fast but less faithful","New XAI evaluation framework: choose methods by task, not habit","ODExAI: Object detection explainability benchmark reveals trade-offs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000671,"raw_usage":{"total_tokens":3096,"prompt_tokens":1022,"completion_tokens":2074,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":1978}},"tokens_in":638,"tokens_out":2074,"duration_ms":17675,"temperature":1.0,"reasoning_tokens":1978,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:56:37.732088+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the same benchmark with multiple random seeds and per-image distributions: if the gaps reported in the paper, such as D-CLOSE's OA of 0.881 versus G-CAME's 0.644 on MS-COCO/YOLOX or G-CAME's PG of 94.31% versus D-CLOSE's 87.86%, overlap under resampling, the claimed trade-offs would not be established.","supporting_citations":[{"cited_title":"In: Yanıkoğlu, B., Buntine, W","cited_arxiv_id":null,"evidence_quote":"defines D-CLOSE, the region-based method whose faithfulness performance anchors the benchmark"},{"cited_title":"Proceedings of the Canadian Con- ference on Artificial Intelligence (2024)","cited_arxiv_id":null,"evidence_quote":"defines G-CAME, the CAM-based method that anchors the localization and runtime side of the trade-off"},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","cited_arxiv_id":null,"evidence_quote":"defines D-RISE, the moderate method used as the third benchmarked explainer"},{"cited_title":"International Journal of Computer Vision126(10), 1084–1102 (2018)","cited_arxiv_id":null,"evidence_quote":"supplies the Pointing Game metric used to measure localization accuracy"},{"cited_title":"In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops","cited_arxiv_id":null,"evidence_quote":"cited as the source of the Energy-based Pointing Game metric used for localization"},{"cited_title":"In: Pattern Recognition and Artificial Intelligence","cited_arxiv_id":null,"evidence_quote":"supplies the Sparsity metric used for the complexity dimension"},{"cited_title":"In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T","cited_arxiv_id":null,"evidence_quote":"provides the MS-COCO dataset used in the benchmark"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the PASCAL VOC dataset used in the benchmark"}],"review_version":1}