A comparative analysis of machine learning models in SHAP analysis

Julia Fukuyama; Justin Lin

arxiv: 2604.07258 · v1 · submitted 2026-04-08 · 💻 cs.LG

A comparative analysis of machine learning models in SHAP analysis

Justin Lin , Julia Fukuyama This is my paper

Pith reviewed 2026-05-10 18:40 UTC · model grok-4.3

classification 💻 cs.LG

keywords SHAP analysisexplainable AImachine learning modelswaterfall plotmulti-class classificationmodel interpretabilitycomparative analysisblack-box models

0 comments

The pith

SHAP interpretations depend on the machine learning model, so analysts need tailored procedures and a generalized waterfall plot for multi-class cases.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper conducts a comparative study of SHAP explanations applied to different machine learning models and datasets. This matters because black-box models handle complex data but remain untrustworthy without clear accounts of how features drive each prediction. The authors map out the nuances that arise in practice to help users navigate model-specific differences. They also introduce an extension of the waterfall plot that visualizes feature contributions across multiple output classes at once.

Core claim

SHAP analysis lacks a universal procedure because the meaning of the values changes with the underlying model; a detailed comparison across models and datasets illustrates these differences, and a novel generalization of the waterfall plot extends the visualization to multi-class classification so that class-specific contributions can be inspected together.

What carries the argument

The generalized waterfall plot, which extends the standard SHAP waterfall visualization to display how each feature shifts the predicted probability for every class in a multi-class problem.

Load-bearing premise

That SHAP values have no single interpretation method that works for all models and that the proposed multi-class waterfall plot will deliver useful new insight without separate validation.

What would settle it

Running the generalized waterfall plot on a standard multi-class dataset such as Iris and finding that the resulting visualization fails to separate class-specific feature effects or contradicts known domain relationships.

Figures

Figures reproduced from arXiv: 2604.07258 by Julia Fukuyama, Justin Lin.

**Figure 2.** Figure 2: Raw simulated data embedded in two dimensions with UMAP colored by target class. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Figure 3: Stacked bar chart of average absolute SHAP values for the simulated data. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 4.** Figure 4: UMAP embedding of SHAP vectors for all three classifiers trained on the simulated data. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

**Figure 5.** Figure 5: SHAP vectors colored according to an HDBSCAN cluster analysis of the XGBoost SHAP vectors [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

**Figure 6.** Figure 6: High-dimensional clustered waterfall plots of SHAP vectors averaged across HDBSCAN clustering [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

**Figure 7.** Figure 7: Heat map of raw data averaged across clusters (simulated data). [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗

**Figure 8.** Figure 8: Raw MNIST data embedded in two dimensions with UMAP colored by digit. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗

**Figure 9.** Figure 9: Stacked bar chart of average absolute SHAP values for the MNIST data. [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗

**Figure 10.** Figure 10: UMAP embedding of SHAP vectors for all three classifiers trained on the MNIST data. [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗

**Figure 11.** Figure 11: High-dimensional clustered waterfall plots of SHAP vectors averaged across digit (MNIST). [PITH_FULL_IMAGE:figures/full_fig_p012_11.png] view at source ↗

**Figure 12.** Figure 12: Raw ADNI data embedded in two dimensions with UMAP colored by patient status. [PITH_FULL_IMAGE:figures/full_fig_p012_12.png] view at source ↗

**Figure 13.** Figure 13: Stacked bar chart of average absolute SHAP values for the ADNI data. [PITH_FULL_IMAGE:figures/full_fig_p013_13.png] view at source ↗

**Figure 14.** Figure 14: UMAP embedding of SHAP vectors for all three classifiers trained on the ADNI data. [PITH_FULL_IMAGE:figures/full_fig_p013_14.png] view at source ↗

**Figure 15.** Figure 15: SHAP vectors colored according to an HDBSCAN cluster analysis of the XGBoost SHAP vectors [PITH_FULL_IMAGE:figures/full_fig_p014_15.png] view at source ↗

**Figure 16.** Figure 16: High-dimensional clustered waterfall plots of SHAP vectors averaged across HDBSCAN clustering [PITH_FULL_IMAGE:figures/full_fig_p014_16.png] view at source ↗

read the original abstract

In this growing age of data and technology, large black-box models are becoming the norm due to their ability to handle vast amounts of data and learn incredibly complex data patterns. The deficiency of these methods, however, is their inability to explain the prediction process, making them untrustworthy and their use precarious in high-stakes situations. SHapley Additive exPlanations (SHAP) analysis is an explainable AI method growing in popularity for its ability to explain model predictions in terms of the original features. For each sample and feature in the data set, an associated SHAP value quantifies the contribution of that feature to the prediction of that sample. Analysis of these SHAP values provides valuable insight into the model's decision-making process, which can be leveraged to create data-driven solutions. The interpretation of these SHAP values, however, is model-dependent, so there does not exist a universal analysis procedure. To aid in these efforts, we present a detailed investigation of SHAP analysis across various machine learning models and data sets. In uncovering the details and nuance behind SHAP analysis, we hope to empower analysts in this less-explored territory. We also present a novel generalization of the waterfall plot to the multi-classification problem.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper runs a SHAP comparison across models and asserts a multi-class waterfall plot extension, but the abstract supplies no definition, examples, or results for the new plot.

read the letter

The paper does a comparative analysis of SHAP across various machine learning models and datasets, and it claims to introduce a generalization of the waterfall plot for multi-class problems. The comparative investigation is the main substance here. It could give practitioners some practical guidance on interpreting SHAP values when the model type changes, since the authors note that interpretation is model-dependent. The new element is the multi-class waterfall plot. If it actually combines class probabilities or feature contributions in a useful way, that might help analysts who deal with classification tasks beyond binary. That said, the abstract states the generalization exists but does not describe how it works, what it looks like, or how it differs from applying SHAP per class and plotting separately. There are no examples, no validation on real data, and no comparison to existing multi-class explainability tools. This makes it tough to tell if the extension adds real value or just restates standard approaches. The paper's claims about empowering analysts rest on this unshown work, so the central argument can't be fully evaluated from the given text. The citation pattern isn't detailed, but the abstract doesn't reference specific prior multi-class SHAP papers, which might be a gap. This kind of work is aimed at applied machine learning users who want more tools for model explanation. A reader who needs a quick overview of SHAP behavior across models might get something out of the comparison section, but anyone looking for a rigorously developed new visualization technique will find it thin. I don't think this rises to the level for peer review yet. The authors would need to define the generalization clearly, show it on data, and demonstrate what new insight it provides before it merits referee time.

Referee Report

2 major / 1 minor

Summary. The manuscript conducts a comparative investigation of SHAP analysis applied to various machine learning models across multiple datasets. It highlights that SHAP value interpretations are inherently model-dependent with no universal procedure, and it proposes a novel generalization of the waterfall plot to multi-class classification problems intended to deliver new insight for analysts working with black-box models.

Significance. A well-executed comparative study could provide practical guidance on model-specific nuances in SHAP usage. If the multi-class waterfall generalization is rigorously defined, shown to differ substantively from per-class SHAP visualizations, and demonstrated via concrete examples to yield previously unavailable interpretability, the work would strengthen explainable AI tooling for multi-class tasks. The paper's emphasis on the absence of a universal analysis procedure is a fair observation already present in the XAI literature.

major comments (2)

Abstract: The claim of a 'novel generalization of the waterfall plot to the multi-classification problem' is load-bearing for the paper's contribution, yet the abstract (and by extension the manuscript framing) supplies no definition of the construction—e.g., how base values, per-class SHAP contributions, or probability outputs are combined or ordered in the plot—nor any comparison to existing multi-class SHAP visualization techniques such as class-specific force plots or stacked bar summaries.
Abstract / Results framing: The manuscript asserts that the generalization 'will provide valuable new insight' in a model-dependent setting, but presents neither the algorithmic steps, pseudocode, nor any empirical demonstration on a concrete dataset showing superior analyst utility or new findings relative to standard per-class SHAP explanations. This absence prevents evaluation of whether the extension is substantive or merely cosmetic.

minor comments (1)

The abstract would be strengthened by naming the specific machine learning models (e.g., random forests, neural nets) and datasets employed in the comparative analysis, allowing readers to assess the scope of the investigation immediately.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed review. The comments highlight important areas where the presentation of our novel multi-class waterfall plot generalization can be strengthened. We address each major comment below and commit to revisions that will improve clarity and rigor without altering the core findings of the comparative SHAP analysis.

read point-by-point responses

Referee: Abstract: The claim of a 'novel generalization of the waterfall plot to the multi-classification problem' is load-bearing for the paper's contribution, yet the abstract (and by extension the manuscript framing) supplies no definition of the construction—e.g., how base values, per-class SHAP contributions, or probability outputs are combined or ordered in the plot—nor any comparison to existing multi-class SHAP visualization techniques such as class-specific force plots or stacked bar summaries.

Authors: We agree that the abstract would benefit from a concise definition of the construction to better frame the contribution. In the revised manuscript we will expand the abstract to briefly describe how base values are computed from class probabilities, how per-class SHAP contributions are ordered and aggregated in the waterfall, and how the plot differs from standard per-class visualizations. We will also add a short comparison to class-specific force plots and stacked bar summaries. The full algorithmic definition already appears in Section 3.2, but the abstract framing will be updated to make this immediately accessible. revision: yes
Referee: Abstract / Results framing: The manuscript asserts that the generalization 'will provide valuable new insight' in a model-dependent setting, but presents neither the algorithmic steps, pseudocode, nor any empirical demonstration on a concrete dataset showing superior analyst utility or new findings relative to standard per-class SHAP explanations. This absence prevents evaluation of whether the extension is substantive or merely cosmetic.

Authors: We acknowledge that a more explicit demonstration is required to substantiate the claim of new insight. In the revision we will add pseudocode for the multi-class waterfall construction, expand the algorithmic steps in the methods section, and include a dedicated empirical subsection. This will apply the plot to a concrete multi-class dataset (e.g., a standard benchmark such as the Iris or Wine dataset) and directly compare the interpretability gains against per-class SHAP force plots and stacked summaries, highlighting previously unavailable cross-class feature contributions. These additions will allow readers to assess the substantive value of the extension. revision: yes

Circularity Check

0 steps flagged

No circularity in derivation chain; empirical investigation with no self-referential reductions

full rationale

The paper conducts a comparative empirical analysis of SHAP values across ML models and datasets and introduces a generalization of the waterfall plot for multi-class problems. No equations, derivations, fitted parameters, or self-citations are invoked as load-bearing steps in the provided text. The generalization is presented as a novel visualization aid without any claim that it follows by construction from prior definitions or inputs within the paper itself. The central claims rest on investigation results rather than any reduction to self-defined quantities, making the work self-contained against external benchmarks with no detectable circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper is an empirical comparative study with no mathematical derivations, free parameters, axioms, or invented entities described in the abstract.

pith-pipeline@v0.9.0 · 5510 in / 1054 out tokens · 39707 ms · 2026-05-10T18:40:41.197502+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

[1]

Augustine, M. T. (2024). A survey on universal approximation theorems.arXiv [preprint]

work page 2024
[2]

Becker, B. and R. Kohavi (1996). Uci machine learning repository

work page 1996
[3]

Campello, R. J. G. B., D. Moulavi, and J. Sander (2013). Density-based clustering based on hierarchical density estimates.PAKDD, Lecture Notes in Computer Science

work page 2013
[4]

Chen, T. and C. Guestrin (2016). XGBoost: A scalable tree boosting system.arXiv [preprint]

work page 2016
[5]

Coenen, A. and A. Pearce (2019). Understanding umap.https://pair-code.github.io/ understanding-umap/. Accessed: 2026-01-16

work page 2019
[6]

Huan, and J

Cohen, J., X. Huan, and J. Ni (2024). Shapley-based explainable AI for clustering applications in fault diagnosis and prognosis.Journal of Intelligent Manufacturing 35, 4071–4086

work page 2024
[7]

Cooper, A. (2022). Supervised clustering: How to use SHAP values for better cluster analysis.https: //www.aidancooper.co.uk/supervised-clustering-shap-values/. Accessed: 2026-01-16

work page 2022
[8]

Doyle, and A

Cooper, A., O. Doyle, and A. Bourke (2022). Supervised clustering for subgroup discovery: An application to COVID-19 symptomatology.Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 408–422

work page 2022
[9]

Deng, L. (2012). The MNIST database of handwritten digit images for machine learning research.IEEE Signal Processing Magazine 29(6)

work page 2012
[10]

Lundberg, S. M., G. G. Erion, and S.-I. Lee (2019). Consistent individualized feature attribution for tree ensembles.arXiv [preprint]

work page 2019
[11]

Lundberg, S. M. and S.-I. Lee (2017). A unified approach to interpreting model predictions.Proceedings of the 31st International Conference on Neural Information Processing Systems, 4768–4777

work page 2017
[12]

Healy, and J

McInnes, L., J. Healy, and J. Melville (2020). UMAP: Uniform manifold approximation and projection for dimension reduction.arXiv [preprint]. O’Bryant, S. E., S. C. Waring, C. M. Cullum, J. Hall, L. Lacritz, P. J. Massman, P. J. Lupo, J. S. Reisch, and R. Doody (2008). Staging dementia using clinical dementia rating scale sum of boxes scores.JAMA Neurology...

work page 2020

[1] [1]

Augustine, M. T. (2024). A survey on universal approximation theorems.arXiv [preprint]

work page 2024

[2] [2]

Becker, B. and R. Kohavi (1996). Uci machine learning repository

work page 1996

[3] [3]

Campello, R. J. G. B., D. Moulavi, and J. Sander (2013). Density-based clustering based on hierarchical density estimates.PAKDD, Lecture Notes in Computer Science

work page 2013

[4] [4]

Chen, T. and C. Guestrin (2016). XGBoost: A scalable tree boosting system.arXiv [preprint]

work page 2016

[5] [5]

Coenen, A. and A. Pearce (2019). Understanding umap.https://pair-code.github.io/ understanding-umap/. Accessed: 2026-01-16

work page 2019

[6] [6]

Huan, and J

Cohen, J., X. Huan, and J. Ni (2024). Shapley-based explainable AI for clustering applications in fault diagnosis and prognosis.Journal of Intelligent Manufacturing 35, 4071–4086

work page 2024

[7] [7]

Cooper, A. (2022). Supervised clustering: How to use SHAP values for better cluster analysis.https: //www.aidancooper.co.uk/supervised-clustering-shap-values/. Accessed: 2026-01-16

work page 2022

[8] [8]

Doyle, and A

Cooper, A., O. Doyle, and A. Bourke (2022). Supervised clustering for subgroup discovery: An application to COVID-19 symptomatology.Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 408–422

work page 2022

[9] [9]

Deng, L. (2012). The MNIST database of handwritten digit images for machine learning research.IEEE Signal Processing Magazine 29(6)

work page 2012

[10] [10]

Lundberg, S. M., G. G. Erion, and S.-I. Lee (2019). Consistent individualized feature attribution for tree ensembles.arXiv [preprint]

work page 2019

[11] [11]

Lundberg, S. M. and S.-I. Lee (2017). A unified approach to interpreting model predictions.Proceedings of the 31st International Conference on Neural Information Processing Systems, 4768–4777

work page 2017

[12] [12]

Healy, and J

McInnes, L., J. Healy, and J. Melville (2020). UMAP: Uniform manifold approximation and projection for dimension reduction.arXiv [preprint]. O’Bryant, S. E., S. C. Waring, C. M. Cullum, J. Hall, L. Lacritz, P. J. Massman, P. J. Lupo, J. S. Reisch, and R. Doody (2008). Staging dementia using clinical dementia rating scale sum of boxes scores.JAMA Neurology...

work page 2020