REVIEW 4 major objections 5 minor 30 references
Explainability for Embedding AI: Aspirations and Actuality
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Popular XAI tools explain AI models but rarely help developers find the faults in them, three surveys show.
desk verdict A genuinely new empirical data point—developers rarely spot injected data faults with LIME/SHAP (1/21 and 2/21)—but the study lacks controls and statistical inference, so the central claim is plausible rather than proven. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Survey 3 fault-detection task: a California Housing regression dataset with two injected faults, a skewed median income over a quarter of the high-value records and a label swap between longitude and number of households, presented to 21 developers who inspected LIME or SHAP explanations of both the unmodified and the modified dataset. The mechanism that carries the argument is the contrast between perceived usefulness and measured detection: participants rate the explanations as plausible and confidence-building, while the planted faults remain nearly invisible, separating the subjective experience of XAI from its debugging value.
What would settle it
Give experienced professional developers a similar fault-injection task with realistic faults such as distribution shift, label noise, or data leakage, and compare LIME and SHAP conditions against a no-explanation baseline that inspects raw feature distributions; if the XAI conditions do not outperform the baseline at detecting faults, the paper's conclusion is supported, but if they do, the low rates in Survey 3 reflect the task or sample rather than a general limitation of the tools.
Extended reading notes
Core claim
The central claim is that popular XAI tools are not currently engineered for software developers who need to locate faults in AI systems or their underlying data. The evidence comes from a fault-detection experiment in which 21 developers inspected LIME or SHAP explanations of a California Housing dataset that had been corrupted in two ways: median income was skewed over a quarter of the high-value records, and the labels for longitude and number of households were swapped. Only one participant using LIME and two using SHAP detected a fault, even though participants rated their understanding, the plausibility of the explanations, and their confidence in the model as generally high. The author interprets this gap between subjective usefulness and actual fault discovery as evidence that LIME and SHAP serve a specific initial use case—explaining model behavior—rather than general exploration of potential faults, and that even experienced developers may mainly use XAI to confirm existing suspicions.
Load-bearing premise
The result depends on the Survey 3 setup being a fair stand-in for real developer debugging: the specific injected faults, the short tool introduction, and a sample dominated by students must approximate how working developers actually use LIME and SHAP.
Editorial extensions
If this is right
- Software developers should not treat current LIME and SHAP output as a reliable way to find data or model faults, because the study found detection rates of only 1 in 21 for LIME and 2 in 21 for SHAP.
- XAI designers need to evaluate tools on fault-discovery tasks, not only on whether explanations seem plausible or increase user confidence.
- Existing XAI tools may confirm what developers already suspect rather than reveal unknown problems, so relying on them for debugging could reinforce mistaken mental models.
- Novice developers are especially underserved, since the tools require prior knowledge to be useful and the study sample included many users with little experience.
- Future XAI debugging support should be designed for the developer's task of locating errors in data and models, not only for explaining a single prediction.
Reading between the lines
- A direct extension, not tested in the paper, would be to measure whether adding LIME or SHAP explanations to a developer's debugging workflow changes the time to find a fault or the number of defects shipped; the paper's results suggest any such benefit would be small.
- The low detection rates hint that feature-attribution methods may miss faults that are spread across many predictions rather than visible in any single explanation; future tools could aggregate attributions over a dataset or compare observed explanations against expected ones.
- The paper's account suggests that data-focused debugging approaches, such as inspecting raw feature distributions and comparing them to expected patterns, may catch the kinds of faults that LIME and SHAP missed, and that combining these with XAI could be a productive direction.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports three online surveys with software developers to assess whether mainstream explainable AI (XAI) tools meet developers' needs. Survey 1 measures demand for explanations for AI and non-AI applications. Survey 2 (n=17) asks developers about the usefulness of LIME and SHAP. Survey 3 (n=21) tests whether developers can use LIME or SHAP to detect deliberately injected faults in the California Housing dataset, including an income-skew fault and a label swap. The results show high demand for explanations, mixed perceived usefulness of the two XAI tools, and very low fault-detection rates (1/21 for LIME, 2/21 for SHAP). The paper concludes that LIME and SHAP, and by implication many XAI tools, are not engineered for fault-finding in AI systems, leaving an unmet need for developer-facing XAI support.
Significance. If the results hold, the paper identifies a practically important gap: popular local explanation tools may not help developers detect the kinds of faults that arise in AI systems. The fault-detection design is concrete and falsifiable, and the direction of the results is consistent with the paper's claim. The paper also deserves credit for using established XAI tools (LIME and SHAP), randomizing condition order, and grounding Survey 1 in a published measurement scale. However, the current evidence is too weak to support the strong formulation of the conclusion: Survey 3 lacks a control condition and a discoverability check, the survey instruments appear unvalidated, the samples are small and mostly student-based, and no inferential statistics are reported. The central contribution is therefore better seen as a plausible hypothesis and a promising study design than as a demonstrated result.
major comments (4)
- [Sec. 3, Survey 3] The central claim, stated in Sec. 5, that popular XAI tools are designed for a specific initial use case and not for generally exploring faults rests on the low fault-detection rates in Survey 3. However, Survey 3 has no control condition (e.g., no tool, or a non-explanation-based debugging aid) and no independent check that the injected faults are actually discoverable in the LIME/SHAP outputs given to participants. The income-skew fault is a distributional property of the training data and may not perturb local feature-attribution explanations, while the longitude/number-of-households swap is only detectable if participants bring strong domain knowledge. Without a baseline or a pilot validation of discoverability, the 1/21 and 2/21 detection rates could reflect the task, the tool outputs, or participant unfamiliarity rather than a general property of XAI tooling.
- [Sec. 4, Results] The results section reports "no significant differences between the two XAI methods" and several group comparisons, such as experienced versus less experienced participants, without presenting any statistical tests, confidence intervals, or effect sizes. With n=17 and n=21 and with further subdivisions into small subgroups, these comparative claims are unsupported. This is load-bearing because the paper uses the similarity between LIME and SHAP outcomes and the experience-based differences to argue that the problem is general rather than tool-specific.
- [Sec. 3, Surveys 1 and 2] The demand-for-explainability scale in Survey 1 is the authors' prior instrument (Weber et al. [28]), and the Survey 2 statements are described as "based on" and "rephrased" from that scale. The manuscript reports no reliability or validity evidence for these adapted measures, such as Cronbach's alpha, factor analysis, or pilot testing. Consequently, the "increased need" and "mixed benefit" findings rely on unvalidated self-report scales, and the need measurement is not independent of the authors' earlier operationalization of the same construct.
- [Sec. 5, Discussion] The generalization from two tools, one dataset, and two injected faults to "the popular XAI tools we presented in our survey, but also many more" goes beyond the evidence. The sample is also majority students, and while the final paragraph of Sec. 5 acknowledges this, the abstract and conclusion state the unmet need categorically. The central claim should be scoped to the tested tools and task, or supported by additional empirical evidence, before it can be presented as a general finding about XAI tooling.
minor comments (5)
- [Sec. 3.1 and Sec. 4] Please clarify how "detected" a fault was defined and coded. For example, state whether the counts (1/21 for LIME, 2/21 for SHAP) refer to participants who explicitly identified the injected fault in the modified condition, and explain how responses in the unmodified condition were treated.
- [Sec. 4] If a significance test underlies the statement that there were "no significant differences" between LIME and SHAP, report the test statistic and p-value; otherwise, remove the phrase or rephrase the observation descriptively.
- [Sec. 3] The study involves human participants, but no ethics approval statement or institutional review board information is given. Please add this information in line with standard reporting requirements for empirical human-subject research.
- [Fig. 1] The axis labels in the embedded figure appear garbled or corrupted in the manuscript text; please provide a clean vector figure with legible axis labels.
- [Sec. 3.1] The compensation phrase "equivalent of 10 $US per hour" should be written as "US$10 per hour" for clarity, and the demographic description of Survey 2 should state exactly how many participants were students versus professionals.
Circularity Check
No circular derivation: conclusions rest on fresh survey data; the only self-citation is a reused measurement scale, which is not load-bearing.
full rationale
The paper's central claims are empirical: developers report a need for explanations (Survey 1) and rarely detect injected faults with LIME/SHAP (1/21 and 2/21 in Survey 3). Survey 1 reuses the scale from Weber et al. [28], a genuine self-citation, but only as a measurement instrument; the responses are new data and the scale is not defined in terms of this paper's conclusions. Survey 3's fault-detection rates come from a manipulated California Housing dataset and fresh participant responses; no equation, fitted parameter, or prior result is imported to generate them. The skeptic's point about missing control conditions and unverified discoverability of the faults is a validity concern, not circularity: the low detection rates are not equivalent to the inputs by construction. No self-definitional, fitted-input-called-prediction, uniqueness-import, ansatz-by-citation, or renaming pattern appears. The only minor self-citation justifies a score of 2 rather than 0, but the derivation chain is otherwise self-contained.
Assumptions & free parameters
assumptions (5)
- domain assumption Self-reported Likert scale responses reflect genuine need for and value of explanations (Weber et al. 2021 scale).
- domain assumption The brief introduction of LIME and SHAP gives participants enough familiarity to judge their usefulness for debugging.
- domain assumption The injected faults in Survey 3 (income skew and label swap) are representative of realistic bugs developers would encounter.
- domain assumption Computer science students and a small number of professionals are an adequate proxy for software developers in practice.
- domain assumption LIME and SHAP are representative of popular XAI systems, so conclusions extend to XAI tools in general.
Cite this review
Pith. "Pith review of Explainability for Embedding AI: Aspirations and Actuality." pith.science (2026). https://pith.science/paper/V3YTKD4A
@misc{pith2026250414631,
author = {Pith},
title = {Pith review of: Explainability for Embedding AI: Aspirations and Actuality},
year = {2026},
howpublished = {\url{https://pith.science/paper/V3YTKD4A}},
note = {Machine review of arXiv:2504.14631}
}
read the original abstract
With artificial intelligence (AI) embedded in many everyday software systems, effectively and reliably developing and maintaining AI systems becomes an essential skill for software developers. However, the complexity inherent to AI poses new challenges. Explainable AI (XAI) may allow developers to understand better the systems they build, which, in turn, can help with tasks like debugging. In this paper, we report insights from a series of surveys with software developers that highlight that there is indeed an increased need for explanatory tools to support developers in creating AI systems. However, the feedback also indicates that existing XAI systems still fall short of this aspiration. Thus, we see an unmet need to provide developers with adequate support mechanisms to cope with this complexity so they can embed AI into high-quality software in the future.
Reference graph
Works this paper leans on
-
[28]
In: Ardito, C., Lanzilotti, R., Malizia, A., Petrie, H., Pic cinno, A., Desolda, G., Inkpen, K
Weber, T., Hußmann, H., Eiband, M.: Quantifying the dema nd for explainability. In: Ardito, C., Lanzilotti, R., Malizia, A., Petrie, H., Pic cinno, A., Desolda, G., Inkpen, K. (eds.) Human-Computer Interaction – INTERACT 20 21. pp. 652–661. Springer International Publishing, Cham (2021)
work page 2021
-
[1]
IEEE Access 6, 52138–52160 (2018) Explainability for Embedding AI: Aspirations and Actualit y 7
Adadi, A., Berrada, M.: Peeking inside the black-box: A su rvey on explainable artificial intelligence (XAI). IEEE Access 6, 52138–52160 (2018) Explainability for Embedding AI: Aspirations and Actualit y 7
work page 2018
-
[2]
Apruzzese, G., Laskov, P., Montes de Oca, E., Mallouli, W. , Brdalo Rapa, L., Grammatopoulos, A.V., Di Franco, F.: The role of machine learning in cybersecurity. Digital Threats: Research and Practice 4(1), 1–38 (Mar 2023). https://doi.org/10.1145/3545574
doi:10.1145/3545574 2023
-
[3]
Barbalau, A., Cosma, A., Ionescu, R.T., Popescu, M.: A gen eric and model-agnostic exemplar synthetization framework for explainable ai (202 0)
-
[4]
Berk, R., Heidari, H., Jabbari, S., Kearns, M., Roth, A.: F airness in criminal justice risk assessments: The state of the art (2017)
work page 2017
-
[5]
Biswas, S., Rajan, H.: Do the machine learning models on a c rowd sourced platform exhibit bias? an empirical study on model fairness . In: Pro- ceedings of the 28th ACM Joint Meeting on European Software E ngi- neering Conference and Symposium on the Foundations of Soft ware Engi- neering. p. 642–653. ESEC/FSE 2020, Association for Comput ing Machin- ery...
arXiv 2020
-
[6]
Foundations and Trends in M arketing 14, 173–236 (01 2020)
Brei, V.: Machine learning in marketing: Overview, learn ing strategies, applica- tions, and future developments. Foundations and Trends in M arketing 14, 173–236 (01 2020). https://doi.org/10.1561/1700000065
-
[7]
Brennen, A.: What do people really want when they say they w ant "explainable ai?" we asked 60 stakeholders. In: Extended Ab stracts of the 2020 CHI Conference on Human Factors in Computing Sys- tems. p. 1–7. CHI EA ’20, Association for Computing Machiner y, New York, NY, USA (2020). https://doi.org/10.1145/3334480.3383047, https://doi.org/10.1145/3334480.3383047
arXiv 2020
Show all 30 references
-
[8]
SSRN Electronic Journal (201 9)
Cohen, I.G., Graver, H.: A doctor’s touch: What big data in health care can teach us about predictive policing. SSRN Electronic Journal (201 9)
-
[9]
Du, M., Liu, N., Hu, X.: Techniques for interpretable mach ine learning. Commun. ACM 63(1), 68–77 (Dec 2019)
2019
-
[10]
In: Fu, W., Pan, S., Brdiczka, O., Chau, P., Calvary, G
Eiband, M., Völkel, S.T., Buschek, D., Cook, S., Hussman n, H.: When people and algorithms meet: user-reported problems in intelligent ev eryday applications. In: Fu, W., Pan, S., Brdiczka, O., Chau, P., Calvary, G. (eds.) Pr oceedings of the 24th International Conference on I...
2019
-
[11]
In: Proceedings of the 25th ACM SIGKDD Internatio nal Conference on Knowledge Discovery & Data Mining
Gade, K., Geyik, S.C., Kenthapadi, K., Mithal, V., Taly, A.: Explainable ai in industry. In: Proceedings of the 25th ACM SIGKDD Internatio nal Conference on Knowledge Discovery & Data Mining. p. 3203–3204. KDD ’19, As sociation for Computing Machinery, New York, NY, USA (2019)
2019
-
[12]
(eds.) Machine Learning and Knowledg e Extraction
Goebel, R., Chander, A., Holzinger, K., Lecue, F., Akata , Z., Stumpf, S., Kieseberg, P., Holzinger, A.: Explainable ai: The new 42? In: Holzinger , A., Kieseberg, P., Tjoa, A.M., Weippl, E. (eds.) Machine Learning and Knowledg e Extraction. pp. 295–303. Springer International...
2018
-
[13]
Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., Yang, G.: XAI - explain- able artificial intelligence. Sci. Robotics 4(37) (2019)
2019
-
[14]
Journal of Environmental Economics and Manageme nt 5(1), 81–102 (1978)
Harrison, D., Rubinfeld, D.L.: Hedonic housing prices a nd the demand for clean air. Journal of Environmental Economics and Manageme nt 5(1), 81–102 (1978). https://doi.org/https://doi.org/10.1016/0095-0696(78)90006-2 , https://www.sciencedirect.com/science/article/pii/0095069...
1978
-
[15]
CoRR abs/1812.04608 (2018)
Hoffman, R.R., Mueller, S.T., Klein, G., Litman, J.: Metr ics for explainable AI: challenges and prospects. CoRR abs/1812.04608 (2018)
2018 arXiv
-
[16]
: What do we need to build explainable AI systems for the medical domain? CoRR abs/1712.09923 (2017) 8 Weber
Holzinger, A., Biemann, C., Pattichis, C.S., Kell, D.B. : What do we need to build explainable AI systems for the medical domain? CoRR abs/1712.09923 (2017) 8 Weber
2017 arXiv
-
[17]
Statistics & Probability Letters 33(3), 291–297 (1997)
Kelley Pace, R., Barry, R.: Sparse spatial autoregres- sions. Statistics & Probability Letters 33(3), 291–297 (1997). https://doi.org/https://doi.org/10.1016/S0167-7152(96)00140-X , https://www.sciencedirect.com/science/article/pii/S016771529600140X
1997 doi
-
[18]
In: Pro- ceedings of the 20th International Conference on Intellige nt User Inter- faces
Kulesza, T., Burnett, M., Wong, W.K., Stumpf, S.: Princi ples of ex- planatory debugging to personalize interactive machine le arning. In: Pro- ceedings of the 20th International Conference on Intellige nt User Inter- faces. p. 126–137. IUI ’15, Association for Computing Machi...
2015
-
[19]
In: 2010 I EEE Sympo- sium on Visual Languages and Human-Centric Computing
Kulesza, T., Stumpf, S., Burnett, M., Wong, W.K., Riche, Y., Moore, T., Oberst, I., Shinsel, A., McIntosh, K.: Explanatory debuggi ng: Supporting end-user debugging of machine-learned programs. In: 2010 I EEE Sympo- sium on Visual Languages and Human-Centric Computing. pp. 4 1...
2010 doi
-
[20]
CoRR abs/1905.04610 (2019)
Lundberg, S.M., Erion, G.G., Chen, H., DeGrave, A., Prut kin, J.M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., Lee, S.: Explainable A I for trees: From local explanations to global understanding. CoRR abs/1905.04610 (2019)
2019 arXiv
-
[21]
CoRR abs/1705.07874 (2017), http://arxiv.org/abs/1705.07874
Lundberg, S.M., Lee, S.: A unified approach to interpreti ng model predictions. CoRR abs/1705.07874 (2017), http://arxiv.org/abs/1705.07874
2017 arXiv
-
[22]
Miller, T., Howe, P., Sonenberg, L.: Explainable ai: Bew are of inmates running the asylum or: How i learnt to stop worrying and love the socia l and behavioural sciences (2017)
2017
-
[23]
Mittelstadt, B.D., Floridi, L.: Transparent, explaina ble, and accountable AI for robotics. Sci. Robotics 2(6) (2017)
2017
-
[24]
In: 2021 International Joint Conference on Neural Networks (IJCNN)
Rasouli, P., Yu, I.C.: Explainable debugger for black-b ox machine learning models. In: 2021 International Joint Conference on Neural Networks (IJCNN). pp. 1–10 (2021). https://doi.org/10.1109/IJCNN52387.2021.9533944
2021
-
[25]
why should i tru st you?
Ribeiro, M.T., Singh, S., Guestrin, C.: "why should i tru st you?": Ex- plaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discover y and Data Mining. p. 1135–1144. KDD ’16, Association for Computing Ma chine...
2016
-
[26]
A., Shambour, M.K.Y., Alsalibi, A.I., Gandomi, A.H.: Machine l earn- ing in medical applications: A review of state-of-the-art m eth- ods
Shehab, M., Abualigah, L., Shambour, Q., Abu-Hashem, M. A., Shambour, M.K.Y., Alsalibi, A.I., Gandomi, A.H.: Machine l earn- ing in medical applications: A review of state-of-the-art m eth- ods. Computers in Biology and Medicine 145, 105458 (2022). https://doi.org/https://doi....
2022
-
[27]
In: Brewster, S.A., Fitzpatrick, G., Cox, A .L., Kostakos, V
Wang, D., Yang, Q., Abdul, A.M., Lim, B.Y.: Designing the ory-driven user-centric explainable AI. In: Brewster, S.A., Fitzpatrick, G., Cox, A .L., Kostakos, V. (eds.) Proceedings of the 2019 CHI Conference on Human Factors in Co mputing Systems, CHI 2019, Glasgow, Scotland, UK...
2019
-
[29]
Wing, J.M.: Computational thinking and thinking about c omput- ing. In: 22nd IEEE International Symposium on Parallel and D is- tributed Processing, IPDPS 2008, Miami, Florida USA, April 14-18, Explainability for Embedding AI: Aspirations and Actualit y 9
2008
-
[2008]
p. 1. IEEE (2008). https://doi.org/10.1109/IPDPS.2008.4536091, https://doi.org/10.1109/IPDPS.2008.4536091 Ϭ ϱ ϭϬ ϭϱ ϮϬ Ϯϱ ϯϬ ϯϱ ϰϬ ϰϱ ϱϬ Ϭ ϱ ϭϬ ϭϱ ϮϬ Ϯϱ ϯϬ ĂƚĂ ĂƚĂ
2008
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.