{"id":"817ae827-b7cb-40c1-ae30-15048669d79e","arxiv_id":"2506.01059","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"XAI-Units offers synthetic datasets and handcrafted models with known ground-truth attributions to unit-test feature attribution methods, and demonstrates its use by revealing a normalization bug in Captum's DeepLIFT.","lead":"This paper introduces XAI-Units, an open-source benchmark that pairs synthetic datasets with handcrafted models of known behavior to unit-test feature attribution methods. It is a practical toolkit for diagnosing when explainability methods fail, and it led to a concrete finding of a normalization bug in Captum's DeepLIFT.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Pertinent Negatives ground truth (Appendix A.4) assigns zero attribution to zero-valued pertinent-negative features, directly contradicting the stated intent in Section 3.1 that FA methods should return non-zero scores for them; the dataset therefore scores the opposite of the property it…","rationale":"The reader's weakest assumption is exactly the internal contradiction I find most load-bearing. The central claim of the paper is that XAI-Units provides reliable ground-truth attributions for models with known mechanisms, enabling objective comparison. A benchmark's ground truth must be consistent with the behavior it claims to isolate. For Pertinent Negatives, the ground-truth definition (ablation to baseline 0) and the stated ideal (non-zero attribution for zero-valued features) are mutually exclusive for samples where the pertinent-negative feature equals 0. Because the dataset is specifically designed to include such samples, the perfect MSE scores in Table 3 do not validate the methods; they reflect the ground truth assigning zero to the very feature the text says should receive non-zero importance. This is a correctness issue internal to the paper, not a disagreement with an external consensus.\n\nA second, less immediate but related issue is that the ground truth for Conflicting Features and Categorical Feature Interaction (Appendices A.3 and A.6) is defined by a sequential ablation order that is arbitrary; reversing the order yields different ground-truth scores. This means the benchmark's 'objective' ground truth is convention-dependent. However, the Pertinent Negatives contradiction is more serious because it conflicts with the benchmark's own stated expectation and produces scores that are known to be wrong relative to that expectation.\n\nThe DeepLIFT/Captum discrepancy finding appears credible and is independently testable; it does not change my assessment. The benchmark is otherwise useful and open-source, so the concern does not warrant rejection, but the Pertinent Negatives ground truth must be fixed or the dataset's description changed before the benchmark can be considered a reliable ground-truth standard. Since the reader already issued CONDITIONAL, my read leaves that verdict unchanged.","tokens_in":18173,"tokens_out":7004,"duration_ms":78355,"concrete_test":"Add a package-level unit test: generate a Pertinent Negatives sample with a feature i in P_i, x_i=0, and m=2; confirm that the current Appendix A.4 ground-truth array assigns 0 to feature i because M(x)-M(x_-i)=0. Then compute the intended score under the Section 3.1 rationale, e.g., M(x)-M(x with x_i replaced by 1) or a Shapley value with baseline 1, and verify it is non-zero. If the packaged ground truth assigns 0 while the intended score is non-zero and the MSE metric penalizes the intended score, the contradiction is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's central claim is that its synthetic model/dataset pairs provide reliable ground-truth attributions for objective FA comparison. The Pertinent Negatives unit test undermines that claim. Section 3.1 states that FA methods 'should ideally return non-zero attributions' when an output depends on a feature being zero (e.g., heart rate 0 as asystole). However, Appendix A.4 defines the ground truth by ablation to the baseline x_ref=0: F_A_xi(x)=M(x)-M(x_-i). For a pertinent-negative feature with value 0, x_-i equals x, so the attribution is identically 0, regardless of the multiplier m that makes the zero value behaviorally meaningful. Thus a method that returns 0 for that feature receives a perfect MSE score, and a method that returns the non-zero score the text says is ideal is penalized. Table 3 shows exactly this: DeepLIFT, IntegratedGradients, KernelSHAP, ShapleyValueSampling, and LIME(Linear) all report 0.000 MSE on the handcrafted Pertinent Negatives model, which is an artifact of the self-contradictory ground truth, not evidence they satisfy the intended property. The fix is straightforward (choose a baseline other than the pertinent-negative value, or define a mask-based expectation), but without it this unit test cannot be used as a ground-truth standard.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces XAI-Units, an open-source Python benchmark for evaluating feature attribution (FA) methods. It provides synthetic datasets paired with handcrafted neural network models whose internal mechanisms are known, defines ground-truth attributions (mostly by ablating inputs to a zero baseline), and evaluates several popular FA methods from Captum on tabular, image, and text datasets. The main claims are that the benchmark enables objective FA evaluation and that it uncovered a DeepLIFT normalization discrepancy in Captum. The paper also includes trained models for comparison and discusses limitations of synthetic evaluation.","tokens_in":18477,"tokens_out":9644,"duration_ms":89862,"significance":"If the ground-truth definitions were internally consistent, the benchmark would be a useful contribution to the XAI community: it is open-source, extensible, and provides atomic 'unit tests' for specific model behaviours. The identification of a possible DeepLIFT normalization bug in Captum is a concrete, reproducible finding that does not depend on the contested ground-truth choices. The authors are transparent about the benchmark's scope (synthetic, technical evaluation) and explicitly disclaim claims about real-world performance or human interpretability. However, the benchmark's reliability as a ground-truth standard is compromised by the Pertinent Negatives self-contradiction described below.","major_comments":[{"comment":"The Pertinent Negatives ground truth contradicts the stated intent. Section 3.1 says that for a pertinent negative feature with a meaningful zero value, FA methods 'should ideally return non-zero attributions.' However, Appendix A.4 defines the ground truth as ablation to the baseline x_ref=0, F_A_xi(x)=M(x)-M(x_-i). For a pertinent negative feature with value 0, x_-i equals x, so the attribution is identically zero. Consequently, methods that return zero for that feature receive a perfect MSE score, and methods that return the non-zero score the text says is ideal are penalized. Table 3 confirms this: DeepLIFT, IntegratedGradients, KernelSHAP, ShapleyValueSampling, and LIME(Linear) all report 0.000 MSE on the handcrafted Pertinent Negatives model, which is an artifact of the contradictory ground truth rather than evidence of satisfying the intended property. The dataset must be redesigned (e.g., using a non-zero baseline for pertinent negatives, or a mask-based expectation) before it can serve as a ground-truth standard.","section":"Section 3.1 and Appendix A.4, Table 3"},{"comment":"The ground truth for the Conflicting Features and Categorical Feature Interaction datasets uses a sequential ablation order without justification. For example, in Appendix A.6, F_A_xi is computed while holding the categorical feature at its observed value, while F_A_ci is computed after the continuous feature has been ablated to baseline. Reversing the order would produce different attributions. Since the paper presents these as objective ground truths, it should either justify the chosen ordering (e.g., as reflecting a specific causal or computational graph semantics) or explicitly acknowledge that the ground truth is one of several possible decompositions and discuss how this affects the interpretation of benchmark scores.","section":"Appendix A.6 (and A.3)"}],"minor_comments":[{"comment":"The sentence 'All FA methods performed well on the simplest test case, the Weighted Continuous models, but struggled on models with gradient discontinuities, such as those for Shattered Gradients and Pertinent Negatives' is not supported by Table 3 for the handcrafted Pertinent Negatives model, where five of seven methods achieve 0.000 MSE; after the ground truth is corrected, this sentence should be revisited.","section":"Section 4.1"},{"comment":"The table lists a single 'Boolean Formula' row but Section 3.1 describes generic Boolean formulas plus separate AND/OR entries; clarify how these map to the table's rows.","section":"Table 2"},{"comment":"The claim of being 'the first within the research community' to provide an end-to-end pipeline with synthetic datasets and handcrafted models is too strong given the prior existence of XAI-Bench and AttributionLab, which also use synthetic ground truths; the novelty claim should be qualified.","section":"Section 5"},{"comment":"In the Boolean AND/OR ground-truth formula, the notation b- is used in the denominator before it is defined in the following line; reorder the presentation for clarity.","section":"Appendix A.8"},{"comment":"The heart-rate example describes a continuous 0 value, but the model assumes pertinent negative features are categorical with values 0 or 1; clarify whether the benchmark's pertinent negative features are indeed binary in the generated data.","section":"Section 3.1, Pertinent Negatives paragraph"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid engineering contribution with a valuable open-source package, but the ground-truth issues, especially the Pertinent Negatives self-contradiction, need to be resolved before the benchmark can be used as an objective standard. The Captum bug finding is the strongest result and could stand alone as a short paper. The authors should also consider discussing the philosophical status of 'ground truth' in synthetic benchmarks and whether their choices reflect one of many possible decompositions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [name],\n\nThe paper is worth engaging with. XAI-Units is a genuinely useful package: it pairs procedurally generated models with synthetic datasets to test feature attribution methods against atomic behaviors, and the experiments surface a real Captum DeepLIFT normalization discrepancy that I had not seen elsewhere. The LLM trigger-injection unit test is also a clever addition. The open-source implementation and reproducibility are strong points.\n\nThe main thing to know is that the Pertinent Negatives unit test is internally inconsistent. Section 3.1 says FA methods 'should ideally return non-zero attributions' for a zero-valued pertinent-negative feature. But Appendix A.4 defines ground truth by ablating to baseline 0: F_A_xi(x)=M(x)-M(x_-i). If xi=0, then x_-i=x, so the attribution is identically 0 no matter how much the output depends on that zero. The result is that DeepLIFT, IntegratedGradients, KernelSHAP, ShapleyValueSampling, and LIME(Linear) all report 0.000 MSE on the handcrafted Pertinent Negatives model. That is not evidence they satisfy the intended property; it is an artifact of the ground truth definition. Perfect scores on this unit test are meaningless as currently constructed.\n\nThe fix is straightforward: choose a baseline other than the pertinent-negative value, or define the ground truth as a mask/expected change, as they do for the Uncertainty and image datasets. Without that, this one unit test should not be cited as a ground-truth standard.\n\nThe other soft spot is smaller: for Conflicting Features and Categorical Feature Interaction, the ground-truth attribution is allocated by a sequential ablation order (ablate categorical first, then continuous). That ordering is arbitrary, and the paper does not discuss why that decomposition is the right one. This is a modeling choice that should be justified or at least flagged, but it is not a contradiction.\n\nOverall, the central idea holds up. The Captum bug finding is independent of the ground-truth definitions, and the rest of the benchmark (mask-based unit tests, Boolean AND/OR, linear weighted features) is on solid ground. I would accept this for peer review with the expectation that the Pertinent Negatives issue gets fixed. The paper is not a rewrite of the field, but it is a solid subfield contribution that will be useful to anyone building or evaluating FA methods.\n\nRecommendation: send to review, but flag the Appendix A.4 problem to the authors.","headline":"Useful benchmark with a real Captum find, but the Pertinent Negatives ground truth contradicts its own stated goal and needs fixing.","tokens_in":18957,"tokens_out":2272,"would_cite":true,"duration_ms":22524,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmark of handcrafted synthetic models with known mechanisms provides ground-truth attributions that let feature attribution methods be compared objectively, and it catches a DeepLIFT implementation bug in Captum.","keywords":["feature attribution","explainable AI","benchmark","ground-truth attributions","synthetic datasets","unit testing","DeepLIFT","Captum"],"falsifier":"For a Pertinent Negatives sample with a pertinent-negative feature at $x_i = 0$, the benchmark's ground truth is $FA_{x_i} = M(x) - M(x^{-i}) = 0$ by ablation to 0, so any FA method that returns a non-zero attribution for that feature—exactly what Section 3.1 says a good method should do—is scored as wrong; a single run showing no method can be both perfect and non-zero on that feature would settle whether the benchmark tests its own stated property.","tokens_in":18024,"feed_emoji":"🧪","tokens_out":6810,"duration_ms":62550,"temperature":0.7,"pith_summary":"This paper tries to establish that feature attribution methods can be compared objectively when the model being explained is handcrafted, deterministic, and perfectly aligned with a synthetic dataset. It introduces XAI-Units, an open-source benchmark where each dataset-model pair acts as a unit test for one atomic behaviour—weighted inputs, feature conflicts, cancellations, abrupt discontinuities, categorical interactions, irrelevant features, Boolean logic, image foreground/background structure, and trigger tokens in text—with ground-truth attribution scores defined in advance by ablating inputs to a baseline reference. Running common FA methods through the benchmark shows where each class of method fails, and it exposes a concrete implementation discrepancy in the popular Captum library's version of DeepLIFT. The value of the work is that it turns the disagreement problem among FA methods into testable claims about specific model behaviours rather than an unresolved philosophical debate.","feed_headline":"Synthetic unit tests expose a DeepLIFT bug in Captum","feed_subtitle":"Handcrafted models with known mechanisms give feature attribution methods a ground truth and reveal failure modes.","key_machinery":"The load-bearing mechanism is the dataset-model unit test: each synthetic data generator is paired with a handcrafted neural network whose internal computation is given as an explicit formula and diagram in Appendix A. Ground truth is produced by a single rule—ablate an input to its baseline reference (almost always 0) and measure the change in output, with per-model formalizations for continuous, categorical, and Boolean features—and where exact attributions are undefined, the benchmark substitutes masks (image foreground, text trigger tokens, relevant features). Built-in evaluation metrics translate this ground truth into a score, which is what lets the benchmark attribute a poor score to the FA method rather than to unknown model behaviour.","core_discovery":"The central claim is that with fully known model mechanisms, ground truth for explanations exists and can be scored. For each synthetic dataset the authors handcraft a neural network whose forward logic is specified exactly; ground-truth attribution for a feature is defined as the change in output when that feature is ablated to a baseline reference value, usually 0, and the benchmark's default metrics (MSE, Mask Error, Mask Proportion, SensitivityMax, Infidelity) compare FA output against those expectations. On the seven tabular unit tests, gradient-based methods fail on zero-gradient conflicts and shattered gradients, linear-surrogate methods (LIME, KernelSHAP) fail on nonlinearity, and the Uncertainty model reveals that Captum's DeepLIFT normalizes at the wrong stage, yielding a high Mask Error that drops to exactly 0 when the normalization is applied to logits as the DeepLIFT paper prescribes.","pith_inferences":["The ablation-to-baseline ground truth encodes one specific semantics of importance—change in output—so methods built on Shapley values, counterfactuals, or necessary-and-sufficient conditions may score poorly without being 'wrong'; the benchmark ranks methods against this single semantics.","The Pertinent Negatives test appears internally inconsistent: zero-valued features expected to receive non-zero attributions are assigned zero by the benchmark's own ground truth, so perfect scores on that test may not reward the intended behaviour.","The DeepLIFT/Captum finding implies that published FA comparisons can be confounded by implementation details rather than method design, and the same unit-test procedure could audit other libraries for similar discrepancies.","Extending the unit-test idea to semi-synthetic data, where some real inputs enter an otherwise handcrafted model, could keep partial ground truth while gaining realism, though exact attribution would have to be relaxed."],"forward_implications":["The benchmark can verify whether an FA method meets its design specification, as demonstrated by the Captum DeepLIFT normalization bug.","Practitioners can identify, for any FA method, the atomic behaviour that causes it to fail, such as zero gradients in Conflicting Features or discontinuity in Shattered Gradients.","Handcrafted models give cleaner ground truth than trained models, since imperfect training can misalign an otherwise correct explanation with the data distribution.","The benchmark is extensible to custom FA methods, metrics, and new unit tests, covering tabular, image, and text/LLM modalities.","Relative rankings of FA methods on two LLM variants were consistent, suggesting the unit-test signal is stable across models of the same task."],"supporting_citations":[{"why":"Defines DeepLIFT, including the logit-normalization recommendation that the benchmark uses to show Captum's implementation applies it in the wrong place.","marker":"Shrikumar et al., 2017"},{"why":"Provides the Captum library whose DeepLIFT, IntegratedGradients, LIME, and KernelSHAP implementations are the tested FA methods.","marker":"Kokhlikyan et al., 2020"},{"why":"Defines SHAP and the baseline-relative notion of attribution that motivates the benchmark's ablation-based ground truth.","marker":"Lundberg and Lee, 2017"},{"why":"Introduces Integrated Gradients and the baseline axiom, used both as a tested method and as justification for baseline-relative ground truth.","marker":"Sundararajan et al., 2017"},{"why":"Supplies the Rashomon-effect argument that synthetic ground truth is only trustworthy when paired with a handcrafted model.","marker":"Breiman, 2001"},{"why":"Presents synthetic explainable classifiers with ground truth, the approach the benchmark extends toward atomic behaviours.","marker":"Guidotti, 2021"},{"why":"Defines explanation infidelity and sensitivity, the metrics used as defaults for the Boolean and Shattered Gradient unit tests.","marker":"Yeh et al., 2019"},{"why":"Formulates the blame problem that the benchmark claims to circumvent by isolating method performance from model behaviour.","marker":"Hossein and Rahnama, 2024"}],"fun_headline_variants":["Unit tests for explainability reveal DeepLIFT flaw","Synthetic unit tests give attribution methods a ground truth","Ground truth for XAI? Unit tests say yes, find failures","DeepLIFT bug exposed by known-model unit tests","Benchmark unit tests uncover why attribution methods clash"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground-truth scores assume that a feature's importance equals how much the model output changes when that feature is replaced by a baseline value (usually 0), an assumption that in the Pertinent Negatives dataset assigns zero importance to a zero-valued feature even though the paper says such features should ideally receive non-zero attribution.","fun_headline_variants_meta":{"raw":{"variants":["Unit tests for explainability reveal DeepLIFT flaw","Synthetic unit tests give attribution methods a ground truth","Ground truth for XAI? Unit tests say yes, find failures","DeepLIFT bug exposed by known-model unit tests","Benchmark unit tests uncover why attribution methods clash"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000714,"raw_usage":{"total_tokens":3199,"prompt_tokens":919,"completion_tokens":2280,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":2202}},"tokens_in":535,"tokens_out":2280,"duration_ms":19531,"temperature":1.0,"reasoning_tokens":2202,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:51:29.773546+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a Pertinent Negatives sample with a pertinent-negative feature at $x_i = 0$, the benchmark's ground truth is $FA_{x_i} = M(x) - M(x^{-i}) = 0$ by ablation to 0, so any FA method that returns a non-zero attribution for that feature—exactly what Section 3.1 says a good method should do—is scored as wrong; a single run showing no method can be both perfect and non-zero on that feature would settle whether the benchmark tests its own stated property.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines DeepLIFT, including the logit-normalization recommendation that the benchmark uses to show Captum's implementation applies it in the wrong place."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Captum library whose DeepLIFT, IntegratedGradients, LIME, and KernelSHAP implementations are the tested FA methods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines SHAP and the baseline-relative notion of attribution that motivates the benchmark's ablation-based ground truth."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Integrated Gradients and the baseline axiom, used both as a tested method and as justification for baseline-relative ground truth."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Rashomon-effect argument that synthetic ground truth is only trustworthy when paired with a handcrafted model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents synthetic explainable classifiers with ground truth, the approach the benchmark extends toward atomic behaviours."},{"cited_title":"S., Inouye, D","cited_arxiv_id":null,"evidence_quote":"Defines explanation infidelity and sensitivity, the metrics used as defaults for the Boolean and Shattered Gradient unit tests."},{"cited_title":"and Rahnama, A","cited_arxiv_id":null,"evidence_quote":"Formulates the blame problem that the benchmark claims to circumvent by isolating method performance from model behaviour."}],"review_version":1}