Pith. sign in

REVIEW 3 major objections 6 minor 10 references

Enhancing ML Model Interpretability: Leveraging Fine-Tuned Large Language Models for Better Understanding of AI

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Pith's one-line read A fine-tuned large language model that translates SHAP values into conversational text can make machine-learning explanations usable by non-experts.

desk verdict Useful fine-tuning recipe and open-source prototype for a SHAP-explaining chatbot, but the human-interpretability claim rests on weak subjective survey evidence. read the letter →

arxiv 2505.02859 v1 pith:R6SSJYWV submitted 2025-05-02 cs.CL cs.AI

classification cs.CLcs.AI
keywords eXplainableAIlargelanguagemodelsfine-tuningSHAPchatbotbatteryStateofHealthmodelinterpretabilitydesignscienceresearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a chatbot built on a fine-tuned large language model can make machine-learning explanations accessible to people unfamiliar with explainable-AI tools. It proposes a reference architecture in which SHAP values, the numeric contributions of each feature to a prediction, are fed into a fine-tuned LLM that answers user questions in natural language. The design is instantiated for battery State-of-Health prediction and evaluated through an online survey with 61 participants. The reported results indicate that the fine-tuned chatbot was rated higher than both a SHAP waterfall plot and a non-fine-tuned chatbot on clarity, interpretability, and cognitive effort, with the largest advantage among users with the least XAI experience.

What carries the argument

The carrying mechanism is a three-stage fine-tuning pipeline applied to a general-purpose LLM with low-rank adaptation, so only a small set of parameters is updated. Stage one enriches the model's background knowledge of SHAP and battery health from unstructured texts; stage two exposes the model to a summary of global SHAP findings; stage three trains it in a supervised way on structured examples that mirror real user questions about the most important features and their SHAP values. After each stage, the model is evaluated by perplexity or loss. The final model is wired into a reference architecture that computes a prediction from a pre-trained CatBoost model, derives SHAP values, and inserts them into an information prompt so the chatbot can answer inferential questions while referring to the XAI visualization.

What would settle it

A randomized experiment in which users answer objective questions about a battery prediction after seeing either the standard SHAP chart or the fine-tuned chatbot's explanation would settle the claim: if chatbot users rate the text as clearer but answer follow-up and prediction questions no better than chart readers, the claimed interpretability gain would not be supported. Counterbalancing the presentation order would also rule out learning effects.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that domain-specific fine-tuning of an LLM substantially improves its ability to turn SHAP outputs into understandable textual explanations, and that this translation is what makes ML interpretable to non-experts. The authors report a three-step fine-tuning process: unsupervised in-domain training on SHAP and battery material, unsupervised training on a written summary of global SHAP findings, and supervised human-alignment training on structured instruction-context-output pairs. Each step is measured by perplexity or loss, with the final alignment step producing the largest reported improvement. In the battery State-of-Health use case, a CatBoost model produces the prediction, SHAP computes feature contributions, and the fine-tuned chatbot converts those values into conversational answers. The survey evidence is interpreted as showing that the prototype enhances human interpretability of ML, especially for users with less experience with XAI.

Load-bearing premise

The central claim rests on the assumption that participants' self-reported impressions that an explanation is clearer and easier to follow reflect genuine understanding, since the survey contained no comprehension task and always showed the standard chart before the chatbot text.

Editorial extensions

If this is right

  • A user without statistical training can query a model's prediction in natural language and receive a domain-aware explanation instead of having to read a SHAP plot.
  • Fine-tuning an LLM on in-domain text plus task-format examples can be monitored step by step with perplexity and loss, and the supervised human-alignment stage gives the largest reported drop in loss.
  • The reference architecture is designed to be domain-agnostic, so other use cases could substitute a different ML model, XAI technique, or LLM while keeping the same interaction pattern.
  • Unlike template-based conversational XAI systems, a fine-tuned LLM can answer inferential and follow-up questions rather than being restricted to a fixed question bank.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported preference is driven by the fluent tone of generated text rather than by comprehension, a task-based study asking users to predict the model's behavior on new inputs would separate those explanations; the current survey only measured self-report.
  • The three-step fine-tuning recipe could plausibly transfer to other XAI outputs such as LIME or counterfactual explanations, but that transfer is not tested in the paper.
  • A prompt-only condition would clarify how much of the gain comes from fine-tuning versus from the information prompt that already contains the SHAP values; the paper's ablation compares fine-tuned against non-fine-tuned but does not isolate prompt engineering.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a reference architecture for an interactive chatbot that translates SHAP-based explanations of a machine learning model into natural language, powered by a fine-tuned LLM (Llama-2-13b-chat). The architecture is instantiated for a battery State-of-Health prediction use case. The authors follow a design science research process: they derive design objectives from literature and expert interviews, implement a prototype, and evaluate it through an ex-ante expert interview (n=16), an intermediate demonstration and competing-artifact analysis, and an ex-post online survey (n=61). The survey compares a SHAP waterfall plot, a non-fine-tuned chatbot, and the fine-tuned chatbot on self-reported clarity, interpretability, and cognitive effort. The authors report that the fine-tuned chatbot outperforms both baselines on all three metrics, especially for participants with low XAI proficiency. They also present a three-step fine-tuning ablation in Table 1, with perplexity and loss improvements, including a 51.5% loss reduction for the human-alignment step. The central claim is that the prototype enhances human interpretability of ML models, particularly for users less experienced with XAI.

Significance. If the central claim were convincingly established, the contribution would be valuable: a concrete, reproducible pipeline for making SHAP outputs accessible to non-experts through fine-tuned LLMs, with potential implications for XAI practice in domains like battery health monitoring. The paper's strengths include a clearly described reference architecture, a publicly available prototype (GitHub repository), and a use case that is both relevant and non-trivial. The design objectives are grounded in the literature and expert input, and the three-step fine-tuning procedure is a useful engineering contribution. However, the significance is conditional on the evaluation actually supporting the claim that the chatbot improves human understanding, and the current evidence is not sufficient: the ex-post survey measures subjective preference rather than comprehension, and the key fine-tuning result is evaluated on data generated by the same procedure used to create the training data. These issues are load-bearing for the central claim.

major comments (3)
  1. [Section 5.3, Figures 5 and 6] The abstract's and conclusion's central claim that the fine-tuned chatbot 'enhances human interpretability' rests on self-reported Likert ratings of clarity, interpretability, and cognitive effort. These are subjective preference measures, not measures of whether participants actually understand the model's behavior. The survey includes no comprehension-grounded task, so an explanation that sounds fluent but does not improve the user's ability to make correct inferences about the prediction would still appear to perform well. Furthermore, the presentation order is fixed: all participants evaluate the SHAP plot first, then the non-fine-tuned chatbot, then the fine-tuned chatbot. The sentence 'To minimize order effects, all participants first evaluated the SHAP plot, followed by the chatbot-generated explanations' is contradictory; a fixed order does not minimize order effects and confounds condition with practice and familiarity. No confidence intervals, significance tests, or effect sizes are reported for the comparisons in Figures 5 and 6. Because this survey is the only evidence for the central claim, the claim is not supported as it stands.
  2. [Section 4.3.3, Step 3 and Table 1] The 51.5% loss improvement from human-alignment fine-tuning is measured on a 'Context Q&A Dataset' generated with the same automated procedure used to create the training data. This evaluation therefore measures how well the model reproduces the authors' own answer style in the generated format, not the correctness, quality, or usefulness of the explanations. It is a circular evaluation that cannot, by itself, demonstrate that fine-tuning improves explanation quality. In addition, the global explanation fine-tuning step yields only a 1.3% perplexity improvement (Table 1), so the Introduction's assertion of 'a substantial improvement across all evaluated metrics' is not supported by the quantitative results; at most one step shows a large movement, and that movement is on a non-independent metric.
  3. [Section 5.3, proficiency-level analysis] The claim that low-proficiency users benefit 'especially' from the fine-tuned chatbot rests on the disparity analysis in Figure 6, but no statistical tests, confidence intervals, or effect sizes are provided for the differences between proficiency groups. The paper reports only the observed pattern of decreasing disparities with increasing proficiency. Without inference statistics, it is impossible to determine whether this pattern is robust or due to chance, and the absence of a control for the fixed presentation order further weakens the interpretation. This is a load-bearing claim because the abstract explicitly singles out 'users with less experience with XAI' as the main beneficiaries.
minor comments (6)
  1. [Section 2.1] The paper uses 'LoRa' in the text but 'LoRA' in the fine-tuning footnotes; please standardize to 'LoRA'.
  2. [References] The reference 'Vaswani, A. (2017). Attention is all you need' is incomplete; it should include the full author list and venue, e.g., Advances in Neural Information Processing Systems 30.
  3. [Section 5.3] The citation of Doshi-Velez and Kim (2017) to justify the fixed presentation order is inappropriate, since that reference concerns interpretability evaluation, not survey order effects; a methods reference on experimental design would be more suitable.
  4. [Table 1] The column headers and row labels in Table 1 are ambiguous; for example, it is not clear which evaluation document is used for each row and why some perplexity values remain unchanged across fine-tuning steps. Please clarify the layout and the meaning of each value.
  5. [Figure 5] The caption 'Survey results for overall interpretability and clarity' mentions only two of the three metrics; the figure appears to include cognitive effort as well, so the caption should list all three.
  6. [Section 4.3.3] The text refers to 'Oobabooga text-generation-webui' without a version or a proper citation; please provide a version number or repository reference.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity in the human-alignment fine-tuning evaluation: the 51.5% loss improvement is measured on a Q&A document generated by the same automatic pipeline as the training data, so it largely measures reproduction of the authors' answer style; the central interpretability claim, however, rests on an independent survey.

  1. fitted input called prediction [Section 4.3.3 (Fine-Tuning, step 3) and Section 4.3.4 / Table 1]
    "To support this process, we automatically generate a structured JSON document for training that includes instructions, relevant contextual information related to the query – such as brief data descriptions, the 20 most important features with their values and SHAP values – and the correct output in alpaca format. ... Using the same approach, we generate an evaluation document (context Q&A) to have matching training and evaluation prompts. Finally , we tested against the evaluation document in a loss evaluation to assess the impact of 3. Human-Alignment fine-tuning."

    The training corpus and the evaluation corpus for human-alignment fine-tuning are both produced by the same automated pipeline, with the evaluation explicitly constructed to have 'matching training and evaluation prompts.' The model is optimized to emit the authors' generated 'correct output,' and the reported 51.5% loss reduction is computed on a second sample from that same generator. The metric therefore measures how faithfully the model reproduces the authors' own answer format and style, not whether explanations improve human understanding or any externally grounded quality.

full rationale

The central claim that the fine-tuned chatbot enhances human interpretability rests mainly on the ex-post survey (Section 5.3), which compares the SHAP waterfall plot, a non-fine-tuned chatbot, and the fine-tuned chatbot using subjective Likert ratings. That survey is not circular, though its self-report design and fixed presentation order are validity concerns rather than circularity. The in-domain fine-tuning evaluations on external documents (SHAP guide, Wikipedia battery articles) are independent benchmarks and show real perplexity gains. The reported 51.5% loss improvement for human-alignment fine-tuning is the one step that reduces by construction: training and evaluation Q&A documents are generated with the same automated approach, so the metric largely measures the model's fit to the authors' own answer-generation style. The paper's self-citations (Schneider 2024; Bokstaller et al. 2023/2024) motivate the gap and provide the battery use case but are not load-bearing for the interpretability claim. Overall this is partial circularity in one quantitative evaluation step, while the main human-facing claim retains independent (though imperfect) survey evidence.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The core artifact is a software system, not a mathematical derivation, so the ledger is short. No numeric constants are fitted to data. The paper rests on three domain assumptions: SHAP values faithfully represent the CatBoost model's reasoning; the authors' self-generated Q&A data is a valid training and evaluation target; and self-reported survey ratings measure interpretability. Each is standard in applied XAI research but none is independently verified. The human-alignment evaluation is the most fragile because the training and evaluation documents share the same generation procedure.

assumptions (4)
  • domain assumption SHAP values computed on the pre-trained CatBoost model accurately reflect feature contributions to the SoH prediction.
    The chatbot explanation flow (Section 4.2) feeds SHAP values into the LLM as the ground-truth input; if SHAP misrepresents the model, all generated explanations inherit the error. No independent check of SHAP validity for this CatBoost model is provided.
  • ad hoc to paper The authors' self-generated Q&A documents contain correct explanations of SHAP values and are a valid target for both fine-tuning and loss evaluation.
    Section 4.3.3 step 3 states that training and evaluation documents are automatically generated in the same way; the 51.5% loss reduction in Table 1 depends on this target being correct and representative. The ground truth is author-defined and not externally validated.
  • domain assumption Self-reported Likert ratings of clarity, interpretability, and cognitive effort measure genuine human interpretability.
    Section 5.3 uses survey ratings as the outcome measure without a comprehension task. Perceived fluency may differ from actual understanding.
  • domain assumption Llama-2-13b-chat is a representative general-purpose LLM whose fine-tuning behavior transfers to other LLMs.
    Section 4.3.2 selects Llama-2-13b-chat and generalizes the results to 'an LLM' throughout; all evidence is collected on this single model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing ML Model Interpretability: Leveraging Fine-Tuned Large Language Models for Better Understanding of AI." pith.science (2026). https://pith.science/paper/R6SSJYWV

@misc{pith2026250502859,
  author       = {Pith},
  title        = {Pith review of: Enhancing ML Model Interpretability: Leveraging Fine-Tuned Large Language Models for Better Understanding of AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R6SSJYWV}},
  note         = {Machine review of arXiv:2505.02859}
}
read the original abstract

Across various sectors applications of eXplainableAI (XAI) gained momentum as the increasing black-boxedness of prevailing Machine Learning (ML) models became apparent. In parallel, Large Language Models (LLMs) significantly developed in their abilities to understand human language and complex patterns. By combining both, this paper presents a novel reference architecture for the interpretation of XAI through an interactive chatbot powered by a fine-tuned LLM. We instantiate the reference architecture in the context of State-of-Health (SoH) prediction for batteries and validate its design in multiple evaluation and demonstration rounds. The evaluation indicates that the implemented prototype enhances the human interpretability of ML, especially for users with less experience with XAI.

Figures

Figures reproduced from arXiv: 2505.02859 by the authors.

Figure 1
Figure 1. Our proposed artifact uses a fine-tuned LLM to enhance ML model interpretability visualized as a UML activity diagram (Dumas et al., 2001). Initially, the developer fine-tunes a chosen general-purpose LLM in three steps to improve its ability to process and respond to domain-specific content and thus, to meet the DOs. Within the application, the user first decides to utilize the chatbot either for domain-specific qu… view at source ↗
Figure 2
Figure 2. Ex-ante evaluation to rate the importance of the research question (RQ) and DOs [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Ex-ante evaluation to rank the importance of the DOs [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 6 canonical work pages

  1. [1]

    Abid, A., Abdalla, A., Abid, A., Khan, D., Alfozan, A., & Zou, J. (2019). Gradio: Hassle-free sharing and testing of ml models in the wild. ArXiv, abs/1906.02569. Arrieta, A. B., Diaz-Rodriguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020). Explainable Artificial Intelligence (X...

  2. [6]

    16 Peffers, K., Tuunanen, T., Rothenberger, M

    The final version will be available in the ECIS 2025 proceedings. 16 Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2017). A Design Science Research Methodology for I nformation Systems Research. Journal of Management Information Systems, 24(3), 45–77. Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). Catbo...

  3. [26]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few -shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901. Chakraborty, S., Tomsett, R., Raghavendra, R., Harborne, D., Alzantot, M., Cerutti, F. (2017). Interpretabilit...

  4. [30]

    Venable, J., Pries -Heje, J., & Baskerville, R. (2016). FEDS: a framework for evaluation in design science research. European Journal of Information Systems, 25(1), 77–89. Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., & Mann, G. (2023). BloombergGPT: A Large Language Model for Finance. arXiv, abs /2303....

  5. [31]

    Räuker, T., Ho, A., Casper, S., & Hadfield- Menell, D. (2023). Toward Transparent AI: A Survey on Interpreting the Inner Structures of D eep Neural Networks. Proceedings of the 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), Raleigh, NC, USA, 464–483. Schneider, J. (2024). Explainable Generative AI (GenXAI): A survey, conceptualiz...

  6. [52]

    Mazumder, S., Ma, N., & Liu, B. (2018). Towards a Continuous Knowledge Learning Engine for Chatbots. ArXiv, abs/1802.06024. Meske, C., Bunde, E., Schneider, J., & Gersch, M. (2022). Explainable Artificial Intelligence: Objectives, Stakeholders, and F uture Research Opportunities. Information Systems Management, 39(1), 53–63. Meta. (2023). Llama 2: Open so...

  7. [289]

    Schneider, J., & Handali, J. (2019). Personalized explanation in machine learning: A conceptualization. Proceedings of the European Conference on Information Systems (ECIS), Stockholm, Sweden. Schwalbe, G., & Finzel, B. (2023). A comprehensive taxonomy for explainable artificial intelligence: A systematic survey of surveys on methods and concepts . Data M...

  8. [2025]

    15 Hu, E

    The final version will be available in the ECIS 2025 proceedings. 15 Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. (2022). Lora: Low-Rank Adaptation of Large Language Models. Proceedings of the T enth International Conference on Learning Representations, Virtual Event, Austria. Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E...

Show all 10 references
  1. [4777]

    Mayer, R. E. & Moreno R. (2010). Nine Ways to Reduce Cognitive Load in Multimedia Learning. Educational Psychlogist, 38(1), 43 –

  2. [4813]

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.- A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023). LLaMA: Open and efficient F oundation Language Models. ArXiv, abs/2302.13971. Van den Broeck, G., Lykov, A., Schleich, M., & Suciu, ...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.