Pith. sign in

REVIEW 3 major objections 6 minor 51 references

Integrating Explainable AI in Medical Devices: Technical, Clinical and Regulatory Insights and Recommendations

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that adding local explanations from LIME to a black-box clinical decision support system increases clinicians' trust and diagnostic accuracy while also producing over-reliance on AI suggestions, a safety concern for…

desk verdict A useful multidisciplinary workshop report whose pilot evidence is being oversold in the abstract; the qualitative regulatory-clinical insights are the real contribution, not the causal claim about explanations improving accuracy. read the letter →

arxiv 2505.06620 v1 pith:JNFZC3TM submitted 2025-05-10 cs.HC cs.AI

Dima Alattal , Asal Khoshravan Azar , Puja Myles , Richard Branson , Hatim Abdulhussein , Allan Tucker This is my paper · ORCID
classification cs.HCcs.AI
keywords explainableAIclinicaldecisionsupportsystemsmedicaldeviceregulationtrustcalibrationLIMEcounterfactualexplanationsautomationbiaspilotstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that local explainability, specifically LIME explanations attached to a black-box neural network, improves clinicians' trust and diagnostic accuracy when they use a clinical decision support system, but that the same explanations create measurable over-reliance on AI suggestions. The evidence comes from an expert working group of regulators, clinicians, and data scientists, plus a pilot study in which eight clinicians diagnosed ten synthetic patient records before and after seeing AI predictions with explanations. The paper argues that regulators and clinicians need different kinds of transparency, and it offers recommendations for safely adopting AI decision support, including preferring simpler models when performance is comparable and using global and counterfactual explanations carefully.

What carries the argument

The central object is the pilot study design: eight practicing clinicians each reviewed ten synthetic patient records, made an initial diagnosis, then saw the neural network's prediction with LIME local explanations and confidence scores, and were allowed to revise their diagnosis. LIME serves as the local explainer, generating feature-importance bars by fitting an interpretable surrogate model around each instance, while ExMatrix provides counterfactual visualizations from the random forest. The before-and-after comparison of clinician diagnoses, interpreted by the expert group, is the mechanism that carries the argument.

What would settle it

A randomized trial with a representative clinician sample and real or realistic patient cases, comparing a group that sees AI predictions with LIME explanations against a group that sees only the predictions, could settle the claim: if the explanation group shows no gain in diagnostic accuracy or a higher rate of accepting incorrect AI recommendations, the paper's central conclusion fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that local explanations increase clinicians' confidence and diagnostic accuracy, especially for low-risk cases, but that this benefit is accompanied by automation bias: in five of six instances where AI and clinician diagnoses differed, clinicians shifted their diagnosis toward the AI, including one case where the AI was wrong. The discovery is that trust calibration, rather than explanation fidelity alone, is the key safety variable in AI-based clinical decision support, and that transparency needs diverge between regulators, who value global feature importance for model selection, and clinicians, who value local and counterfactual explanations for individual cases.

Load-bearing premise

The load-bearing premise is that the behavior of eight self-selected clinicians on ten deliberately chosen synthetic patient records, with no control condition and no statistical analysis, represents what clinicians in general would do when given local explanations.

Editorial extensions

If this is right

  • If local explanations improve accuracy but induce over-reliance, then AI clinical decision support should include safeguards such as override prompts, uncertainty display, and training in when to discard AI recommendations.
  • Regulators should evaluate human-AI interaction evidence, not just model performance metrics, when approving AI-based medical devices.
  • Divergent global feature rankings across models mean model selection should involve clinical review of top-ranked features, not accuracy metrics alone.
  • Counterfactual explanations that suggest changing immutable features like age or medical history are clinically unusable and should be filtered before presentation.
  • Simpler, interpretable models with comparable performance are preferable to black-box models in high-risk clinical settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The pilot's lack of a control condition and its small self-selected sample mean the observed accuracy gain could plausibly come from task familiarity or regression to the mean rather than from explanations; a controlled replication is needed before acting on the recommendation.
  • If explanations increase agreement with AI, they may amplify systematic model errors in biased or under-represented subgroups; regulators might therefore require subgroup-specific interaction testing, not just overall accuracy.
  • Local explanations such as LIME may be better suited as training tools for junior clinicians than as real-time decision aids for experts, since time pressure was reported as a barrier to using them in practice.
  • The observed divergence between clinician and model feature rankings suggests a testable extension: showing clinicians the model's top features before their own diagnosis could either reduce anchoring bias or increase it, depending on how the information is framed.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript reports a multidisciplinary expert working group convened by the UK MHRA to evaluate AI/XAI-based clinical decision support for heart attack risk prediction. It presents three models (logistic regression, random forest, neural network) on a synthetic CPRD dataset, global, local, and counterfactual explanation methods, qualitative insights from regulators and clinicians, and a pilot study in which eight clinicians diagnosed ten synthetic records before and after seeing AI outputs. The headline empirical claim is that local explanations increase clinicians' trust and diagnostic accuracy, with an automation-bias caveat. The paper concludes with recommendations on model selection, explanation design, stakeholder training, and regulatory safeguards.

Significance. If the empirical claim were properly supported, the paper would offer useful evidence for AI-CDSS design and regulation. The workshop-based qualitative insights and the concrete recommendations around model selection, explanation presentation, and trust calibration are valuable, and the use of a public synthetic dataset is a strength. However, the central quantitative claim about local explanations improving accuracy and trust is currently not supported by the pilot design and analysis, and the paper's own HRA tool output says the findings are not generalisable. The paper would still be a useful multidisciplinary position piece if the empirical claims were appropriately downgraded and clearly labelled as exploratory.

major comments (3)
  1. [Abstract and §3.2] The abstract states that 'the study reveals an overall increase in clinicians' trust and diagnostic accuracy when using local explanations,' but the pilot described in §3.1 and §3.2 cannot support a causal attribution to local explanations. Each clinician first diagnosed from raw records and was then shown the AI diagnosis, confidence, and all explanation types (global, LIME local, and ExMatrix counterfactual) together, with no control condition and no arm isolating local explanations. Appendix A explicitly records 'No' to generalisability, which is the paper's own admission that the empirical basis is not generalisable. The sentence should be revised to describe an exploratory observation or be supported by a properly controlled study.
  2. [§3.2] The claim that 'five out of six instances' of clinician adjustment produced 'an overall improvement in diagnostic accuracy' is unsupported by the reported data. No per-case or per-clinician before/after accuracy table, no paired comparison, and no statistical test are provided, and the acknowledged automation-bias case could offset any accuracy gain. The authors should report the full contingency of clinician diagnoses before and after AI/XAI exposure and, if appropriate, a paired analysis; otherwise the conclusion should be limited to a qualitative observation.
  3. [§2.2.1 and Table 1] The comparison that 'random forest outperformed the others' is made from point estimates of sensitivity, specificity, precision, and AUC without confidence intervals, bootstrap resampling, or cross-validation variability. Because the workshop discussion and the pilot selection depend on model performance, the authors should either provide uncertainty measures for these metrics or soften the comparison to descriptive reporting.
minor comments (6)
  1. [§2.2.2 and Table 2] The text and Table 2 use 'odd ratios' where 'odds ratios' is the correct term; this should be corrected throughout.
  2. [§2.2.1] There are typographical errors such as 'ans it considered the simplest model' and 'the complexiest'; these should read 'as it is considered' and 'the most complex.'
  3. [§4] The lesson numbering jumps from Lesson 5 to Lesson 7; the lessons should be renumbered or Lesson 6 should be added.
  4. [§2.3 and Table 3] The caption 'T able 3' and the phrase 'as in shown in Table 1' are formatting/grammatical issues that should be cleaned up.
  5. [§3.1] The sentence 'ensuring that they were provided with exactly the same data that has been used by the AI/ML models' is grammatically awkward and should be rephrased.
  6. [References] References [23] and [27] appear to cite the same Herm et al. paper; this duplicate should be merged or disambiguated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical workshop/pilot observations, not derivations from fitted inputs or self-citation chains.

full rationale

The paper does not present a formal derivation in which a predicted quantity is defined in terms of the data used to produce it. The abstract's claim that local explanations increase trust and diagnostic accuracy is an interpretation of the Section 3 pilot: clinicians made an initial diagnosis, were shown the AI prediction with global, local and counterfactual explanations and confidence, and then were asked whether and how their assessment changed. No parameter is fitted and then renamed as a prediction, and no uniqueness theorem or explanatory ansatz is imported from the authors' prior work. The only author-overlapping citation is [41] for the synthetic CPRD-derived dataset used to train the models; that citation is data provenance, not a load-bearing theoretical premise, and the dataset is an external CPRD resource. The pilot's small sample and lack of a control arm, together with the HRA decision tool in Appendix A recording that the findings are 'not going to be generalisable', undermine the strength of the causal wording in the abstract, but that is an external-validity and over-interpretation concern, not a circularity concern. The discussion's supporting references to prior studies [17, 50, 5, 11, 49] are outside citations for observed effects, not self-referential proofs. No step in the paper reduces by construction to its own inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims are qualitative and empirical, so there is no formal derivation and no invented entity. The hand-chosen inputs that matter are the undisclosed model hyperparameters and the ten selected pilot cases; the model performance metrics are fitted values presented to the experts. The main domain assumptions are the representativeness of the synthetic data, the clinicians, and the case selection, plus reliance on self-reports of confidence.

free parameters (3)
  • ML model hyperparameters = not reported
    The paper says 'We explored a number of parameterisations and train-test regimes' in Section 2.2.1 but does not report the final hyperparameters for logistic regression, random forest, or the neural network. The explanations shown to clinicians depend on these fitted models.
  • Ten pilot patient records = 10 hand-picked records
    The clinical team selected ten cases as 'diverse and clinically interesting' in Section 3.1 rather than sampling at random, and this selection affects the pilot outcomes.
  • Model performance values in Table 1 = LR AUC 0.90, RF AUC 0.96, ANN AUC 0.80
    These fitted metrics were presented to the expert group and inform the interpretability and performance trade-off discussion that shapes the recommendations.
assumptions (4)
  • domain assumption The high-fidelity synthetic CPRD dataset is a valid proxy for real primary care data when studying clinician trust in XAI outputs.
    The paper relies on this in Section 2.1 for all model training, explanation generation, and pilot cases; unrealistic data would undermine the ecological validity of the clinical judgments.
  • domain assumption The eight recruited clinicians are representative of the broader clinician population that would use an AI-CDSS.
    Recruitment and experience levels are not reported in Section 3.1, and the paper itself cites prior work showing clinician preferences vary with background; small self-selected samples limit generalization.
  • domain assumption Self-reported confidence and stated agreement with AI explanations reflect actual trust and diagnostic behavior.
    In Section 3.2, 'all clinicians agreed that these explanations boosted their confidence' is a subjective report; no behavioral measure of trust or decision time is provided.
  • domain assumption The ten deliberately selected patient records cover a sufficiently diverse range of diagnostic scenarios to test the effect of AI and XAI outputs on clinician behavior.
    The selection in Section 3.1 prioritizes clinical interest over random sampling, so the pilot can only speak to these cases, not to a population of cases.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Integrating Explainable AI in Medical Devices: Technical, Clinical and Regulatory Insights and Recommendations." pith.science (2026). https://pith.science/paper/JNFZC3TM

@misc{pith2026250506620,
  author       = {Pith},
  title        = {Pith review of: Integrating Explainable AI in Medical Devices: Technical, Clinical and Regulatory Insights and Recommendations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JNFZC3TM}},
  note         = {Machine review of arXiv:2505.06620}
}
read the original abstract

There is a growing demand for the use of Artificial Intelligence (AI) and Machine Learning (ML) in healthcare, particularly as clinical decision support systems to assist medical professionals. However, the complexity of many of these models, often referred to as black box models, raises concerns about their safe integration into clinical settings as it is difficult to understand how they arrived at their predictions. This paper discusses insights and recommendations derived from an expert working group convened by the UK Medicine and Healthcare products Regulatory Agency (MHRA). The group consisted of healthcare professionals, regulators, and data scientists, with a primary focus on evaluating the outputs from different AI algorithms in clinical decision-making contexts. Additionally, the group evaluated findings from a pilot study investigating clinicians' behaviour and interaction with AI methods during clinical diagnosis. Incorporating AI methods is crucial for ensuring the safety and trustworthiness of medical AI devices in clinical settings. Adequate training for stakeholders is essential to address potential issues, and further insights and recommendations for safely adopting AI systems in healthcare settings are provided.

Figures

Figures reproduced from arXiv: 2505.06620 by the authors.

Figure 4
Figure 4. LIME local explanations for two selected cases predicted by the neural network model: (a) local explanation for a false positive output, (b) local explanation for a true positive output. For counterfactual generation, we used the Explainable Matrix ”ExMatrix” tech￾nique to extract counterfactual explanations from random forest. ExMatrix identifies decision paths within the random forest that lead to different predic… view at source ↗
Figure 5
Figure 5. Exmatrix counterfactual cases for a true positive case. Figure (a), Original matrix visualisation . Figure (b), simplified 1D plot of ExMatrix. 2.3 Results & Expert Working Group Insights In this study, we presented the AI/ML models employed to predict heart attack risk to the expert group. Performance metrics for these algorithms were subsequently shared with them, as detailed in [PITH_FULL_IMAGE:figures/full_fig_… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 43 canonical work pages

  1. [1]

    In: Hagen, H

    Preim, B.: HCI in Medical Visualization. In: Hagen, H. (ed.) Scientific Visu- alization: Interactions, Features, Metaphors vol. 2, pp. 292–310 (2011). https: //doi.org/10.4230/DFU.Vol2.SciViz.2011.292

    work page doi:10.4230/dfu.vol2.sciviz.2011.292 2011

  2. [2]

    BMC medical informatics and decision making 20, 1–9 (2020)

    Amann, J., Blasimme, A., Vayena, E., Frey, D., Madai, V.I., Consortium, P.: Explainability for artificial intelligence in healthcare: a multidisciplinary perspective. BMC medical informatics and decision making 20, 1–9 (2020)

    work page 2020

  3. [3]

    BMC medical informatics and decision making 21(1), 178 (2021)

    Alam, L., Mueller, S.: Examining the effect of explanation on satisfaction and trust in ai diagnostic systems. BMC medical informatics and decision making 21(1), 178 (2021)

    work page 2021

  4. [4]

    Computers and Biomedical Research 14(6), 542–558 (1981)

    Teach, R.L., Shortliffe, E.H.: An analysis of physician attitudes regard- ing computer-based clinical consultation systems. Computers and Biomedical Research 14(6), 542–558 (1981)

    work page 1981

  5. [5]

    Available at SSRN 4773350 (2024)

    Rosenbacke, R.: Errors in physician-ai collaboration: Insights from a mixed- methods study of explainable ai and trust in clinical decision-making. Available at SSRN 4773350 (2024)

    work page 2024

  6. [6]

    Science 366(6464), 447–453 (2019) 40

    Obermeyer, Z., Powers, B., Vogeli, C., Mullainathan, S.: Dissecting racial bias in an algorithm used to manage the health of populations. Science 366(6464), 447–453 (2019) 40

    work page 2019

  7. [7]

    Information Fusion 77, 29–52 (2022)

    Yang, G., Ye, Q., Xia, J.: Unbox the black-box for the medical explainable ai via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond. Information Fusion 77, 29–52 (2022)

    work page 2022

  8. [8]

    Hasan, Z., Serajuddin, M., Alvi, S.A.M., Khan, A., Ayub, R.: Explainable deep learning models for healthcare decision support

    work page

Show all 51 references
  1. [9]

    Health and Technology 14(1), 1–14 (2024)

    Pagallo, U., O’Sullivan, S., Nevejans, N., Holzinger, A., Friebe, M., Jeanquartier, F., Jean-Quartier, C., Miernik, A.: The underuse of ai in the health sector: Oppor- tunity costs, success stories, risks and recommendations. Health and Technology 14(1), 1–14 (2024)

    2024

  2. [10]

    In: Proceedings of the First International Symposium on Trustworthy Autonomous Systems

    Li, P., Williams, R., Gilbert, S., Anderson, S.: Regulating ai/ml-enabled medi- cal devices in the uk. In: Proceedings of the First International Symposium on Trustworthy Autonomous Systems. TAS ’23. Association for Computing Machin- ery, New York, NY, USA (2023). https://doi....

    2023

  3. [11]

    Rosenbacke, R.: Heuristics and errors in xai-augmented clinical decision-making: Moving beyond algorithmic appreciation and aversion (2024)

    2024

  4. [12]

    arXiv preprint arXiv:2202.01602 (2022)

    Krishna, S., Han, T., Gu, A., Pombra, J., Jabbari, S., Wu, S., Lakkaraju, H.: The disagreement problem in explainable machine learning: A practitioner’s perspective. arXiv preprint arXiv:2202.01602 (2022)

    2022 arXiv

  5. [13]

    World Wide Web 24(5), 1857–1884 (2021)

    Naiseh, M., Al-Thani, D., Jiang, N., Ali, R.: Explainable recommendation: when design meets trust calibration. World Wide Web 24(5), 1857–1884 (2021)

    2021

  6. [14]

    IEEE Access 8, 220335–220351 (2020) 41

    Okamura, K., Yamada, S.: Empirical evaluations of framework for adaptive trust calibration in human-ai cooperation. IEEE Access 8, 220335–220351 (2020) 41

    2020

  7. [15]

    Patterns 1(4) (2020)

    Tomsett, R., Preece, A., Braines, D., Cerutti, F., Chakraborty, S., Srivastava, M., Pearson, G., Kaplan, L.: Rapid trust calibration through interpretable and uncertainty-aware ai. Patterns 1(4) (2020)

    2020

  8. [16]

    Plos one 15(2), 0229132 (2020)

    Okamura, K., Yamada, S.: Adaptive trust calibration for human-ai collaboration. Plos one 15(2), 0229132 (2020)

    2020

  9. [17]

    International Journal of Human-Computer Studies 169, 102941 (2023)

    Naiseh, M., Al-Thani, D., Jiang, N., Ali, R.: How the different explanation classes impact trust calibration: The case of clinical decision support systems. International Journal of Human-Computer Studies 169, 102941 (2023)

    2023

  10. [18]

    In: Workshop” Automation Experience Across Domains” in Conjunction with CHI2020 (2020)

    Schrammel, J., Fr¨ ohlich, P., Mirnig, A.G., Dinica, O., Lindley, A., Woitsch, R., Falconi, D., Baldauf, M.: Investigating communication techniques to support trust calibration for automated systems. In: Workshop” Automation Experience Across Domains” in Conjunction with CHI20...

    2020

  11. [19]

    BMJ Health & Care Informatics 30(1) (2023)

    Oppermann, I.: Regulating ai for health. BMJ Health & Care Informatics 30(1) (2023)

    2023

  12. [20]

    In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

    Panigutti, C., Beretta, A., Giannotti, F., Pedreschi, D.: Understanding the impact of explanations on advice-taking: A user study for ai-based clinical decision sup- port systems. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. CHI ’22. Associ...

    2022

  13. [21]

    IEEE Transactions on Emerging Topics in Computing 10(4), 1746–1760 (2022)

    Jia, Y., McDermid, J., Lawton, T., Habli, I.: The role of explainability in assuring safety of machine learning in healthcare. IEEE Transactions on Emerging Topics in Computing 10(4), 1746–1760 (2022)

    2022

  14. [22]

    arXiv preprint arXiv:1606.05386 (2016) 42

    Ribeiro, M.T., Singh, S., Guestrin, C.: Model-agnostic interpretability of machine learning. arXiv preprint arXiv:1606.05386 (2016) 42

    2016 arXiv

  15. [24]

    Neural computing and applications 32(24), 18069–18083 (2020)

    Vellido, A.: The importance of interpretability and visualization in machine learning for applications in medicine and health care. Neural computing and applications 32(24), 18069–18083 (2020)

    2020

  16. [25]

    Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 10(5), 1379 (2020)

    Stiglic, G., Kocbek, P., Fijacko, N., Zitnik, M., Verbert, K., Cilar, L.: Inter- pretability of machine learning-based prediction models in healthcare. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 10(5), 1379 (2020)

    2020

  17. [26]

    The Lancet Digital Health 2(9), 489–492 (2020)

    Futoma, J., Simons, M., Panch, T., Doshi-Velez, F., Celi, L.A.: The myth of gen- eralisability in clinical research and machine learning in health care. The Lancet Digital Health 2(9), 489–492 (2020)

    2020

  18. [27]

    International Journal of Information Management 69, 102538 (2023)

    Herm, L.-V., Heinrich, K., Wanner, J., Janiesch, C.: Stop ordering machine learning algorithms by their explainability! a user-centered investigation of per- formance and explainability. International Journal of Information Management 69, 102538 (2023)

    2023

  19. [28]

    In: 2021 IEEE 15th International Conference on Semantic Computing (ICSC), pp

    Singla, K., Biswas, S.: Machine learning explanability method for the multi-label classification model. In: 2021 IEEE 15th International Conference on Semantic Computing (ICSC), pp. 337–340 (2021). IEEE

    2021

  20. [29]

    IEEE access 6, 52138–52160 (2018)

    Adadi, A., Berrada, M.: Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE access 6, 52138–52160 (2018)

    2018

  21. [30]

    43 Artificial intelligence 267, 1–38 (2019)

    Miller, T.: Explanation in artificial intelligence: Insights from the social sciences. 43 Artificial intelligence 267, 1–38 (2019)

    2019

  22. [31]

    Applied Sciences 11(11), 5088 (2021)

    Antoniadi, A.M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B.A., Mooney, C.: Current challenges and future opportunities for xai in machine learning-based clinical decision support systems: a systematic review. Applied Sciences 11(11), 5088 (2021)

    2021

  23. [32]

    In: Proceedings of the 22nd ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining, pp

    Ribeiro, M.T., Singh, S., Guestrin, C.: ” why should i trust you?” explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining, pp. 1135–1144 (2016)

    2016

  24. [33]

    PhD thesis, Michigan Technological University (2020)

    Alam, L.: Investigating the impact of explanation on repairing trust in ai diag- nostic systems for re-diagnosis. PhD thesis, Michigan Technological University (2020)

    2020

  25. [34]

    In: Handbook of Artificial Intelligence in Healthcare: Vol 2: Practicalities and Prospects, pp

    Manresa-Yee, C., Roig-Maim´ o, M.F., Ramis, S., Mas-Sans´ o, R.: Advances in xai: Explanation interfaces in healthcare. In: Handbook of Artificial Intelligence in Healthcare: Vol 2: Practicalities and Prospects, pp. 357–369. Springer, ??? (2021)

    2021

  26. [35]

    npj Digital Medicine 6(1), 94 (2023)

    Bienefeld, N., Boss, J.M., L¨ uthy, R., Brodbeck, D., Azzati, J., Blaser, M., Willms, J., Keller, E.: Solving the explainable ai conundrum by bridging clinicians’ needs and developers’ goals. npj Digital Medicine 6(1), 94 (2023)

    2023

  27. [36]

    Advances in neural information processing systems 30 (2017)

    Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. Advances in neural information processing systems 30 (2017)

    2017

  28. [37]

    In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp

    Panigutti, C., Perotti, A., Pedreschi, D.: Doctor xai: an ontology-based approach to black-box sequential data classification explanations. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 629–639 (2020) 44

    2020

  29. [38]

    Brankovic, A., Cook, D., Rahman, J., Huang, W., Khanna, S.: Evaluation of popular xai applied to clinical prediction models: Can they be trusted? arXiv preprint arXiv:2306.11985 (2023)

    2023 arXiv

  30. [39]

    arXiv preprint arXiv:2010.10596 (2020)

    Verma, S., Boonsanong, V., Hoang, M., Hines, K.E., Dickerson, J.P., Shah, C.: Counterfactual explanations and algorithmic recourses for machine learning: A review. arXiv preprint arXiv:2010.10596 (2020)

    2020 arXiv

  31. [40]

    Informa- tion Sciences 655, 119898 (2024)

    Del Ser, J., Barredo-Arrieta, A., D´ ıaz-Rodr´ ıguez, N., Herrera, F., Saranti, A., Holzinger, A.: On generating trustworthy counterfactual explanations. Informa- tion Sciences 655, 119898 (2024)

    2024

  32. [41]

    In: Third International Workshop on Learning with Imbalanced Domains: Theory and Applications, pp

    Draghi, B., Wang, Z., Myles, P., Tucker, A.: Bayesboost: identifying and han- dling bias using synthetic data generators. In: Third International Workshop on Learning with Imbalanced Domains: Theory and Applications, pp. 49–62 (2021). PMLR

    2021

  33. [42]

    URL: https://christophm

    Molnar, C.: A guide for making black box models explainable. URL: https://christophm. github. io/interpretable-ml-book 2(3), 10 (2018)

    2018

  34. [43]

    In: 2017 IEEE 41st Annual Com- puter Software and Applications Conference (COMPSAC), vol

    Yildirim, P.: Chronic kidney disease prediction on imbalanced data by multilayer perceptron: Chronic kidney disease prediction. In: 2017 IEEE 41st Annual Com- puter Software and Applications Conference (COMPSAC), vol. 2, pp. 193–198 (2017). IEEE

    2017

  35. [44]

    Expert Systems with Applications 30(2), 272–281 (2006)

    Yan, H., Jiang, Y., Zheng, J., Peng, C., Li, Q.: A multilayer perceptron-based medical decision support system for heart disease diagnosis. Expert Systems with Applications 30(2), 272–281 (2006)

    2006

  36. [45]

    Advances in neural information processing systems 34, 23928– 23941 (2021)

    Kadra, A., Lindauer, M., Hutter, F., Grabocka, J.: Well-tuned simple nets excel on 45 tabular datasets. Advances in neural information processing systems 34, 23928– 23941 (2021)

    2021

  37. [46]

    Information Fusion 81, 84–90 (2022)

    Shwartz-Ziv, R., Armon, A.: Tabular data: Deep learning is not all you need. Information Fusion 81, 84–90 (2022)

    2022

  38. [47]

    IEEE Transactions on Neural Networks and Learning Systems (2022)

    Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., Kasneci, G.: Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems (2022)

    2022

  39. [48]

    Sensors 22(22), 8615 (2022)

    Mavrogiorgou, A., Kiourtis, A., Kleftakis, S., Mavrogiorgos, K., Zafeiropoulos, N., Kyriazis, D.: A catalogue of machine learning algorithms for healthcare risk predictions. Sensors 22(22), 8615 (2022)

    2022

  40. [49]

    Applied Sciences 12(20), 10323 (2022)

    Du, Y., Antoniadi, A.M., McNestry, C., McAuliffe, F.M., Mooney, C.: The role of xai in advice-taking from a clinical decision support system: A comparative user study of feature contribution-based and example-based explanations. Applied Sciences 12(20), 10323 (2022)

    2022

  41. [50]

    IEEE Transactions on Artificial Intelligence (2021)

    Wu, H., Ruan, W., Wang, J., Zheng, D., Liu, B., Geng, Y., Chai, X., Chen, J., Li, K., Li, S., et al.: Interpretable machine learning for covid-19: An empirical study on severity prediction task. IEEE Transactions on Artificial Intelligence (2021)

    2021

  42. [51]

    Fisher, A., Rudin, C., Dominici, F.: All models are wrong, but many are useful: Learning a variable’s importance by studying an entire class of prediction models simultaneously. J. Mach. Learn. Res. 20(177), 1–81 (2019)

    2019

  43. [52]

    IEEE Transactions on Visualization and Computer Graphics 27(2), 1427–1437 (2020) 46 Appendix A: Ethical Approval Requirements for the Pilot Study) Go straight to content

    Neto, M.P., Paulovich, F.V.: Explainable matrix-visualization for global and local interpretability of random forest classification ensembles. IEEE Transactions on Visualization and Computer Graphics 27(2), 1427–1437 (2020) 46 Appendix A: Ethical Approval Requirements for the ...

    2020

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.