{"id":"7a43c676-ebb9-49d1-ac9d-a7149f17b6e2","arxiv_id":"2504.14631","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Survey evidence shows developers want XAI support, yet most participants using LIME or SHAP failed to detect deliberately injected data faults in a housing price model.","lead":"For software developers who build AI-powered products, are today's explanation tools actually useful? Three surveys asked developers about explainable AI and tested whether they could find deliberately planted data bugs using LIME and SHAP. The result: developers report a real need for explanations, but most could not detect the injected faults.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Survey 3 lacks a control condition and an independent check that the injected faults are discoverable in the LIME/SHAP outputs, so the low detection rates do not yet establish that XAI tools fail developers at fault-finding.","rationale":"I read the paper as an exploratory, three-survey report arguing that developers need explainability but current popular XAI tools are not designed for debugging. The paper is clear about its limitations and does not overclaim in the body; the empirical findings are honestly reported. However, the strongest empirical pillar is Survey 3's fault-detection result, and that pillar is unvalidated: the experiment lacks a control condition, and the authors never demonstrate that the injected faults are discoverable from the exact LIME/SHAP outputs participants saw. The income skew can be invisible to local attribution methods because the model is trained on the corrupted distribution; the label swap requires domain reasoning outside the explanatory output. Thus the low detection rates could be an artifact of task design, not a property of XAI tooling. This is not a claim of inconsistency, but the evidence-to-conclusion gap is real. The reader's CONDITIONAL verdict already asks for stronger evidence; my concern adds two specific requirements: a control/baseline and a detectability audit. I therefore keep the verdict unchanged.","tokens_in":7364,"tokens_out":7113,"duration_ms":62570,"concrete_test":"Run a follow-up of Survey 3 with an added no-XAI control (participants get the raw dataset and summary statistics only) and have two independent XAI experts inspect the exact LIME/SHAP outputs for the two injected faults before the study; if experts cannot identify both faults from those outputs, or if the no-XAI control yields a similar detection rate, the Survey 3 result is a task artifact rather than evidence about XAI's utility for debugging.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LIME and SHAP are not engineered for fault-finding rests on Survey 3 (Sec. 3), where only 1/21 (LIME) and 2/21 (SHAP) participants detected an injected fault. The load-bearing issue is that the study has no control condition and no evidence that the injected faults are actually visible in the explanations the tools produce. The income-skew fault is a distributional property of the training data; a local feature-attribution explanation of individual predictions can remain perfectly plausible for a model trained on that skewed data, so the fault need not appear in any LIME or SHAP output. The label-swap fault is detectable only if participants bring domain knowledge and notice that a semantically wrong feature has high importance; it is not a signal that LIME or SHAP is designed to emit. Without a baseline condition (e.g., no-XAI tools, or a non-explanation-based control) and without an expert check that the exact outputs supplied to participants make the faults discoverable, the low detection rates cannot be attributed to a general failure of XAI tooling. The paper's own Sec. 5 caveat that the tools 'were not designed as dedicated debugging tools' further underscores that the experiment may be measuring a mismatch between task and tool rather than a defect in the tools.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports three online surveys with software developers to assess whether mainstream explainable AI (XAI) tools meet developers' needs. Survey 1 measures demand for explanations for AI and non-AI applications. Survey 2 (n=17) asks developers about the usefulness of LIME and SHAP. Survey 3 (n=21) tests whether developers can use LIME or SHAP to detect deliberately injected faults in the California Housing dataset, including an income-skew fault and a label swap. The results show high demand for explanations, mixed perceived usefulness of the two XAI tools, and very low fault-detection rates (1/21 for LIME, 2/21 for SHAP). The paper concludes that LIME and SHAP, and by implication many XAI tools, are not engineered for fault-finding in AI systems, leaving an unmet need for developer-facing XAI support.","tokens_in":7555,"tokens_out":4528,"duration_ms":42048,"significance":"If the results hold, the paper identifies a practically important gap: popular local explanation tools may not help developers detect the kinds of faults that arise in AI systems. The fault-detection design is concrete and falsifiable, and the direction of the results is consistent with the paper's claim. The paper also deserves credit for using established XAI tools (LIME and SHAP), randomizing condition order, and grounding Survey 1 in a published measurement scale. However, the current evidence is too weak to support the strong formulation of the conclusion: Survey 3 lacks a control condition and a discoverability check, the survey instruments appear unvalidated, the samples are small and mostly student-based, and no inferential statistics are reported. The central contribution is therefore better seen as a plausible hypothesis and a promising study design than as a demonstrated result.","major_comments":[{"comment":"The central claim, stated in Sec. 5, that popular XAI tools are designed for a specific initial use case and not for generally exploring faults rests on the low fault-detection rates in Survey 3. However, Survey 3 has no control condition (e.g., no tool, or a non-explanation-based debugging aid) and no independent check that the injected faults are actually discoverable in the LIME/SHAP outputs given to participants. The income-skew fault is a distributional property of the training data and may not perturb local feature-attribution explanations, while the longitude/number-of-households swap is only detectable if participants bring strong domain knowledge. Without a baseline or a pilot validation of discoverability, the 1/21 and 2/21 detection rates could reflect the task, the tool outputs, or participant unfamiliarity rather than a general property of XAI tooling.","section":"Sec. 3, Survey 3"},{"comment":"The results section reports \"no significant differences between the two XAI methods\" and several group comparisons, such as experienced versus less experienced participants, without presenting any statistical tests, confidence intervals, or effect sizes. With n=17 and n=21 and with further subdivisions into small subgroups, these comparative claims are unsupported. This is load-bearing because the paper uses the similarity between LIME and SHAP outcomes and the experience-based differences to argue that the problem is general rather than tool-specific.","section":"Sec. 4, Results"},{"comment":"The demand-for-explainability scale in Survey 1 is the authors' prior instrument (Weber et al. [28]), and the Survey 2 statements are described as \"based on\" and \"rephrased\" from that scale. The manuscript reports no reliability or validity evidence for these adapted measures, such as Cronbach's alpha, factor analysis, or pilot testing. Consequently, the \"increased need\" and \"mixed benefit\" findings rely on unvalidated self-report scales, and the need measurement is not independent of the authors' earlier operationalization of the same construct.","section":"Sec. 3, Surveys 1 and 2"},{"comment":"The generalization from two tools, one dataset, and two injected faults to \"the popular XAI tools we presented in our survey, but also many more\" goes beyond the evidence. The sample is also majority students, and while the final paragraph of Sec. 5 acknowledges this, the abstract and conclusion state the unmet need categorically. The central claim should be scoped to the tested tools and task, or supported by additional empirical evidence, before it can be presented as a general finding about XAI tooling.","section":"Sec. 5, Discussion"}],"minor_comments":[{"comment":"Please clarify how \"detected\" a fault was defined and coded. For example, state whether the counts (1/21 for LIME, 2/21 for SHAP) refer to participants who explicitly identified the injected fault in the modified condition, and explain how responses in the unmodified condition were treated.","section":"Sec. 3.1 and Sec. 4"},{"comment":"If a significance test underlies the statement that there were \"no significant differences\" between LIME and SHAP, report the test statistic and p-value; otherwise, remove the phrase or rephrase the observation descriptively.","section":"Sec. 4"},{"comment":"The study involves human participants, but no ethics approval statement or institutional review board information is given. Please add this information in line with standard reporting requirements for empirical human-subject research.","section":"Sec. 3"},{"comment":"The axis labels in the embedded figure appear garbled or corrupted in the manuscript text; please provide a clean vector figure with legible axis labels.","section":"Fig. 1"},{"comment":"The compensation phrase \"equivalent of 10 $US per hour\" should be written as \"US$10 per hour\" for clarity, and the demographic description of Survey 2 should state exactly how many participants were students versus professionals.","section":"Sec. 3.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is very short for a journal article and reads like an extended abstract or workshop paper. The core idea is timely and the fault-detection study is worth pursuing, but the experimental design needs substantial strengthening—particularly the addition of a control condition or a discoverability check—before the strong conclusion about XAI tools is warranted. If the authors can add such evidence or considerably restrict the scope of their claims, the paper could become a useful empirical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this short arXiv paper by Weber. It reports three online surveys, and the last one has a striking result: 21 developers, given LIME or SHAP on the California Housing dataset with two injected faults, almost never found them (1/21 with LIME, 2/21 with SHAP). That is a new, concrete data point in the XAI-for-debugging discussion. The paper is honest and doesn't oversell the tools.\n\nThe good: the question matters, the Survey 3 design is straightforward, and the detection rates speak for themselves. It also usefully reframes a known debate—LIME and SHAP were built for specific use cases, not general fault-finding. The discussion in Section 5 is measured and includes the right caveats: the tools aren't dedicated debugging tools, the example use cases weren't complex, and the sample was mostly students.\n\nThe soft spots are real. Survey 3 has no control condition. Participants never used a no-XAI or non-explanation baseline, so we don't know how many faults developers would find by inspecting the data or the model alone. More importantly, there is no independent check that the injected faults are visible in the LIME/SHAP outputs the participants saw. The income skew is a distributional property; a local feature attribution can look plausible on a model trained on skewed data. The label swap might be detectable if you know longitude matters for housing price, but that is domain knowledge, not an XAI signal. The stress-test note is right: the low detection rates don't yet establish that XAI tools fail developers at fault-finding.\n\nAlso, the samples are small (n=17, n=21), mostly students, and there are no statistical tests, confidence intervals, or effect sizes. The abstract's \"increased need\" lacks a clear baseline, and the demand scale is self-cited from Weber et al. No materials or data are released, which would be easy to fix.\n\nWho is this for? Anyone working on XAI evaluation or developer tooling. It's a preliminary result, not a definitive one. I think it deserves a serious referee—the core finding is worth taking seriously, but the revision needs controls, a sanity check on fault visibility, and a stats pass. If those land, this could be a solid conference paper or workshop contribution. As is, it's an interesting data point to cite with caution.\n\nI'd probably bring it to a reading group to discuss what evidence would actually convince us that XAI does or doesn't help debugging.","headline":"A genuinely new empirical data point—developers rarely spot injected data faults with LIME/SHAP (1/21 and 2/21)—but the study lacks controls and statistical inference, so the central claim is plausible rather than proven.","tokens_in":8110,"tokens_out":2163,"would_cite":true,"duration_ms":19928,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Popular XAI tools explain AI models but rarely help developers find the faults in them, three surveys show.","keywords":["explainable AI","software developers","debugging","LIME","SHAP","fault detection","user study","data-driven development"],"falsifier":"Give experienced professional developers a similar fault-injection task with realistic faults such as distribution shift, label noise, or data leakage, and compare LIME and SHAP conditions against a no-explanation baseline that inspects raw feature distributions; if the XAI conditions do not outperform the baseline at detecting faults, the paper's conclusion is supported, but if they do, the low rates in Survey 3 reflect the task or sample rather than a general limitation of the tools.","tokens_in":7097,"feed_emoji":"🐛","tokens_out":5366,"duration_ms":46359,"temperature":0.7,"pith_summary":"The paper reports three surveys with software developers aimed at determining whether explainable AI (XAI) tools help the people who build and maintain AI-powered software. It argues that developers do want explanations for AI systems, but mainstream tools like LIME and SHAP are designed for their original use cases—explaining individual predictions—rather than as debugging aids. In a fault-detection test on the California Housing dataset with two deliberately injected errors, only one of 21 participants using LIME and two using SHAP spotted a fault, even though participants reported high understanding, plausibility, and confidence. The paper concludes that current XAI systems fall short of the aspiration of supporting developers in finding faults, leaving an unmet need for human-centered XAI debugging tools.","feed_headline":"Only 3 of 21 developers found injected AI faults with LIME or SHAP","feed_subtitle":"They rated explanations plausible and useful, yet the deliberately planted faults stayed hidden.","key_machinery":"The load-bearing object is the Survey 3 fault-detection task: a California Housing regression dataset with two injected faults, a skewed median income over a quarter of the high-value records and a label swap between longitude and number of households, presented to 21 developers who inspected LIME or SHAP explanations of both the unmodified and the modified dataset. The mechanism that carries the argument is the contrast between perceived usefulness and measured detection: participants rate the explanations as plausible and confidence-building, while the planted faults remain nearly invisible, separating the subjective experience of XAI from its debugging value.","core_discovery":"The central claim is that popular XAI tools are not currently engineered for software developers who need to locate faults in AI systems or their underlying data. The evidence comes from a fault-detection experiment in which 21 developers inspected LIME or SHAP explanations of a California Housing dataset that had been corrupted in two ways: median income was skewed over a quarter of the high-value records, and the labels for longitude and number of households were swapped. Only one participant using LIME and two using SHAP detected a fault, even though participants rated their understanding, the plausibility of the explanations, and their confidence in the model as generally high. The author interprets this gap between subjective usefulness and actual fault discovery as evidence that LIME and SHAP serve a specific initial use case—explaining model behavior—rather than general exploration of potential faults, and that even experienced developers may mainly use XAI to confirm existing suspicions.","pith_inferences":["A direct extension, not tested in the paper, would be to measure whether adding LIME or SHAP explanations to a developer's debugging workflow changes the time to find a fault or the number of defects shipped; the paper's results suggest any such benefit would be small.","The low detection rates hint that feature-attribution methods may miss faults that are spread across many predictions rather than visible in any single explanation; future tools could aggregate attributions over a dataset or compare observed explanations against expected ones.","The paper's account suggests that data-focused debugging approaches, such as inspecting raw feature distributions and comparing them to expected patterns, may catch the kinds of faults that LIME and SHAP missed, and that combining these with XAI could be a productive direction."],"forward_implications":["Software developers should not treat current LIME and SHAP output as a reliable way to find data or model faults, because the study found detection rates of only 1 in 21 for LIME and 2 in 21 for SHAP.","XAI designers need to evaluate tools on fault-discovery tasks, not only on whether explanations seem plausible or increase user confidence.","Existing XAI tools may confirm what developers already suspect rather than reveal unknown problems, so relying on them for debugging could reinforce mistaken mental models.","Novice developers are especially underserved, since the tools require prior knowledge to be useful and the study sample included many users with little experience.","Future XAI debugging support should be designed for the developer's task of locating errors in data and models, not only for explaining a single prediction."],"supporting_citations":[{"why":"Supplies LIME, one of the two XAI tools whose fault-detection utility is tested in the surveys.","marker":"[25]"},{"why":"Supplies SHAP, the other XAI tool tested in the surveys.","marker":"[21]"},{"why":"Provides the California Housing dataset into which the two faults were injected.","marker":"[17]"},{"why":"Provides the survey scale used in Survey 1 to gauge the demand for explainability.","marker":"[28]"},{"why":"Supports the argument that existing XAI systems are narrow in scope and inadequately address user needs.","marker":"[7]"},{"why":"Establishes the idea of explanatory debugging for machine-learned programs, the aspiration the paper tests against actual tools.","marker":"[19]"}],"fun_headline_variants":["AI explanations feel useful, but devs miss 86% of faults","LIME and SHAP fail devs: only 3 of 21 spot injected AI faults","Confidence in XAI high, fault detection low: 3 of 21 devs succeed","Explainability's promise vs reality: 3 of 21 devs catch planted faults"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on the Survey 3 setup being a fair stand-in for real developer debugging: the specific injected faults, the short tool introduction, and a sample dominated by students must approximate how working developers actually use LIME and SHAP.","fun_headline_variants_meta":{"raw":{"variants":["AI explanations feel useful, but devs miss 86% of faults","LIME and SHAP fail devs: only 3 of 21 spot injected AI faults","Confidence in XAI high, fault detection low: 3 of 21 devs succeed","Explainability's promise vs reality: 3 of 21 devs catch planted faults"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001365,"raw_usage":{"total_tokens":5487,"prompt_tokens":846,"completion_tokens":4641,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":4549}},"tokens_in":462,"tokens_out":4641,"duration_ms":29534,"temperature":1.0,"reasoning_tokens":4549,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:43:40.623364+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give experienced professional developers a similar fault-injection task with realistic faults such as distribution shift, label noise, or data leakage, and compare LIME and SHAP conditions against a no-explanation baseline that inspects raw feature distributions; if the XAI conditions do not outperform the baseline at detecting faults, the paper's conclusion is supported, but if they do, the low rates in Survey 3 reflect the task or sample rather than a general limitation of the tools.","supporting_citations":[{"cited_title":"In: Ardito, C., Lanzilotti, R., Malizia, A., Petrie, H., Pic cinno, A., Desolda, G., Inkpen, K","cited_arxiv_id":null,"evidence_quote":"Provides the survey scale used in Survey 1 to gauge the demand for explainability."}],"review_version":1}