REVIEW 2 major objections 5 minor 34 references
Don't Just Translate, Agitate: Using Large Language Models as Devil's Advocates for AI Explanations
T0 review · 2 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read LLM-based XAI translation can cause overreliance; this paper argues LLMs should act as devil's advocates that challenge AI explanations.
desk verdict A short, honest workshop position paper proposing LLMs as devil's advocates for XAI explanations; the idea is promising but the central claim is untested and there is no mechanism to keep adversarial critiques accurate. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a set of adversarial prompting strategies, summarized as a table, that re-frame the LLM's persona from helpful assistant to helpful adversary. The strategies are uncertainty awareness, alternative explanations, bias detection, counterfactual thinking, scrutinizing assumptions, explanation audit, and user-calibrated depth, with example prompts that tell the LLM to surface what a smooth translation would hide. This devil's-advocate prompt regime does the work in the argument: it converts the LLM's tendency to smooth over inconsistencies into a means of surfacing them, and it is offered as a springboard for few-shot training and future empirical testing.
What would settle it
A controlled study comparing the same explainer output delivered as a fluent narrative versus an adversarial devil's-advocate form, measuring users' decision accuracy and trust calibration on cases where the model is wrong; if adversarial users overrely just as much as translation users, the central claim fails.
Extended reading notes
Core claim
The central claim is that LLMs should stop being passive translators and become constructive agitators. In a decision-support setting, an LLM should not take the explainer's output as ground truth and dress it up as a readable story; it should challenge the output, present multiple interpretations, point out where the explanation may be unreliable, and offer counterfactual probes. The paper grounds this claim in studies showing that natural-language and conversational explanations raise trust without improving understanding, so users overrely on wrong predictions. It also points to a prior devil's advocate study as evidence that adversarial LLMs can be seen as useful collaborators. The paper is explicitly a position paper with proposed prompting strategies; it does not report a new user study confirming them.
Load-bearing premise
The central proposal depends on users actually engaging with adversarial output and recalibrating their trust from it, and on the LLM's adversarial claims being accurate enough to be worth believing.
Editorial extensions
If this is right
- LLM-based XAI systems should be judged by whether they reduce overreliance and improve trust calibration, not by how fluent or preferred their explanations are.
- Conversational XAI assistants should include an adversarial mode that challenges the model's own outputs, not only a mode that answers user questions in a supportive tone.
- Generated explanations should carry uncertainty statements, alternative explanations, and known explainer limitations rather than a single plausible narrative.
- Designers can use game-like structures or progressive difficulty to make adversarial engagement rewarding enough to offset the lower satisfaction users give to cognitive forcing.
- Different audiences, such as clinicians versus patients, could receive adversarial explanations calibrated to their expertise surfaced from the same XAI output.
Reading between the lines
- Editorial inference: because the paper provides no user study, the decisive empirical test is to compare final decision quality under translated versus adversarial explanations; that test would separate the value of the role change from the value of showing any extra information at all.
- Editorial inference: the paper's own cited evidence implies the adversarial claims themselves could be wrong; a robust version might add a second verification pass that grounds every adversarial statement in explainer output, otherwise a confident but false critique could destabilize trust further.
- Editorial inference: an untested corollary is that adversarial prompting could aid model developers debugging a classifier, since they may tolerate the cognitive friction that end users reject.
- Editorial inference: the overreliance problem may be dominated by users' default trust in any fluent authority, so the devil's advocate role only works if users read it as an adversarial persona and not as just another confident AI voice.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This position paper argues that the current trend of using large language models (LLMs) to translate XAI outputs such as SHAP or LIME values into fluent natural-language narratives is insufficient and potentially harmful, because such translations can make AI decisions appear more plausible than transparent and can increase user overreliance and misplaced trust. The authors propose instead that LLMs should act as 'devil's advocates' that actively interrogate AI explanations by surfacing uncertainty, alternative interpretations, potential biases, counterfactual scenarios, and limitations of the explainer itself. The paper reviews prior empirical work on LLM-based explanation translation, conversational XAI assistants, and counterfactual explanations; presents a table of prompting strategies for a devil's advocate; and discusses future directions for workflow design, persona-based explanations, and the tradeoff between cognitive friction and user satisfaction.
Significance. If the proposal is validated, it would reframe the role of LLMs in XAI from passive fluency providers to active critics, directly targeting the documented problem of overreliance caused by persuasive but shallow explanations. The paper is honest about the lack of empirical evaluation and explicitly states that 'empirical evaluations are ultimately needed' (Sec. 3). Its concrete contribution is a set of prompt exemplars in Table 1 that can serve as a springboard for future studies, and it connects the proposal to prior cognitive-forcing research, thereby giving the community a testable starting point. The paper also correctly identifies a key tradeoff: adversarial interaction may reduce overreliance but lower user satisfaction, citing Buçinca et al. [2]. These strengths make the proposal a useful addition to the XAI-HCI discourse, provided the identified gaps are addressed.
major comments (2)
- [Sec. 3, Table 1] The central mechanism requires that the devil's advocate's adversarial claims—counterfactuals, bias analyses, and explainer audits—are factually grounded in the actual model and data, but the paper provides no mechanism to ensure this. Prompts such as 'If Feature X were reduced by 10%, the model's decision would flip' assume the LLM has access to the fitted classifier's decision boundary, which is not the case for a pre-trained LLM. The paper acknowledges (Sec. 3) that LLMs can produce 'unfaithful or placebic explanations' and yet relies on the same model class to critique them, so the concern that the adversary may hallucinate plausible-but-wrong claims is load-bearing for the proposal. The manuscript should either propose a grounding or verification strategy (e.g., constraining the LLM to only restate information computed by a trustworthy explainer, or including explicit uncertainty markers on every claim) or clearly frame this as an open research question that must be solved before the approach can be adopted.
- [Sec. 3 and Sec. 4] The causal claim that a devil's advocate will reduce overreliance and improve trust calibration is not tested, and the only directly supporting evidence cited is Chiang et al. [4] in a group decision-making setting, which differs from the XAI explanation scenario. Buçinca et al. [2], cited by the authors themselves, shows that cognitive forcing reduces overreliance but receives the lowest user favorability ratings, so the proposed intervention may be effective only when users tolerate the friction. The manuscript should either temper the abstract's claim that adopting the devil's advocate role will 'reduce overreliance' or specify a falsifiable empirical design, including hypotheses about when the approach helps versus when it backfires (e.g., inducing unwarranted distrust in correct model decisions). As a position paper this is acceptable as a call for research, but the current phrasing overstates the strength of the evidence.
minor comments (5)
- [Sec. 1, first bullet] 'easy of interpretability' should be 'ease of interpretability'.
- [Sec. 2, He et al. paragraph] The sentence 'Four XAI methods were employed (SHAP, PDP [10], Mace [28], the What-If Tool [20], and a Decision Tree [22])' lists five methods; the count or the list should be revised for consistency.
- [Table 1] There are typos in the 'Scrutinize Assumptions' and 'User-Calibrated Depth' rows: 'le/t_ting' should be 'letting' and 'a/t_tributions' should be 'attributions'.
- [References, [6]] The author list for Crisan et al. is incomplete ('Anamaria Crisan, Nathan Butters, and Zoe') and the entry appears truncated; the full author and title should be provided.
- [Sec. 4, 'Design for a Productive Adversarial Process'] 'there will inevitable difficulties' should be 'there will inevitably be difficulties'.
Circularity Check
No circularity found: the paper is a position argument with no derivation chain that reduces to its inputs.
full rationale
This is a workshop position paper that proposes a role for LLMs as devil's advocates in XAI, not a derivation or empirical validation. The central claim is normative: rather than translating XAI outputs into smooth narratives, LLMs should surface uncertainty, alternative explanations, potential biases, and failure cases. That claim is supported by external empirical findings from Buçinca et al., He et al., Crisan et al., Zytek et al., and Chiang et al., and it is not equivalent to any of those inputs. There is no fitted parameter, no equation, and no prediction that is forced by construction. The prompt strategies in Table 1 are introduced as 'a springboard for future research and exploration,' and the authors explicitly state that 'empirical evaluations are ultimately needed to assess the effectiveness of these strategies,' so the proposal is not dressed up as a derived result. The self-citations [23,24] appear in a background discussion of different user needs and are not load-bearing: the devil's advocate argument would stand unchanged without them. No uniqueness theorem is imported, no ansatz is smuggled in via citation, and no known result is merely renamed. Accordingly, the paper exhibits no significant circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption Critical engagement with AI explanations reduces overreliance and improves decision outcomes.
- domain assumption LLMs can produce accurate alternative interpretations, counterfactual explanations, and bias analyses without introducing meaningful new errors.
- domain assumption Users will engage with adversarial LLM outputs and adjust their trust appropriately, rather than ignoring the LLM or deferring to it.
Cite this review
Pith. "Pith review of Don't Just Translate, Agitate: Using Large Language Models as Devil's Advocates for AI Explanations." pith.science (2026). https://pith.science/paper/QOJCI7J7
@misc{pith2026250412424,
author = {Pith},
title = {Pith review of: Don't Just Translate, Agitate: Using Large Language Models as Devil's Advocates for AI Explanations},
year = {2026},
howpublished = {\url{https://pith.science/paper/QOJCI7J7}},
note = {Machine review of arXiv:2504.12424}
}
read the original abstract
This position paper highlights a growing trend in Explainable AI (XAI) research where Large Language Models (LLMs) are used to translate outputs from explainability techniques, like feature-attribution weights, into a natural language explanation. While this approach may improve accessibility or readability for users, recent findings suggest that translating into human-like explanations does not necessarily enhance user understanding and may instead lead to overreliance on AI systems. When LLMs summarize XAI outputs without surfacing model limitations, uncertainties, or inconsistencies, they risk reinforcing the illusion of interpretability rather than fostering meaningful transparency. We argue that - instead of merely translating XAI outputs - LLMs should serve as constructive agitators, or devil's advocates, whose role is to actively interrogate AI explanations by presenting alternative interpretations, potential biases, training data limitations, and cases where the model's reasoning may break down. In this role, LLMs can facilitate users in engaging critically with AI systems and generated explanations, with the potential to reduce overreliance caused by misinterpreted or specious explanations.
Reference graph
Works this paper leans on
-
[2]
Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos . 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-computer Interaction 5, CSCW1 (2021), 1–21
work page 2021
-
[4]
Chun-Wei Chiang, Zhuoran Lu, Zhuoyan Li, and Ming Yin. 20 24. Enhancing AI-Assisted Group Decision Making through LL M-Powered Devil’s Advocate. In Proceedings of the 29th International Conference on Intelli gent User Interfaces . 103–119
-
[1]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, J ared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pran av Shyam, Girish Sas- try, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh , Daniel Ziegler, Jef- frey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler , Mateusz Litwin, ...
work page 2020
-
[3]
Adrian Bussone, Simone Stumpf, and Dympna O’Sullivan. 2 015. The role of explanations on trust and reliance in clinic al decision support systems. In 2015 international conference on healthcare informatics . IEEE, 160–169
work page 2015
-
[5]
Andrew M Colman. 2016. Game theory and experimental games: The study of strategic i nteraction. Elsevier
work page 2016
-
[6]
Anamaria Crisan, Nathan Butters, and Zoe. 2024. Explori ng Subjective Notions of Explainability through Counterfactual Visualization of Sentiment Analysis. In 2024 IEEE Evaluation and Beyond-Methodological Approache s for Visualization (BELIV) . IEEE, 15–24
work page 2024
-
[7]
Google DeepMind. [n. d.]. Gemini 2.0 Flash Thinking. htt ps://deepmind.google/technologies/gemini/flash-thinking/. Accessed: 2025-04-11
work page 2025
-
[8]
Malin Eiband, Daniel Buschek, Alexander Kremer, and Hei nrich Hussmann. 2019. The impact of placebic explanations o n trust in intelligent systems. In Extended abstracts of the 2019 CHI conference on human facto rs in computing systems . 1–6
work page 2019
Show all 34 references
-
[9]
Arturo Fredes and Jordi Vitria. 2024. Using LLMs for Expl aining Sets of Counterfactual Examples to Final Users. arXiv preprint arXiv:2408.15133 (2024)
2024 arXiv
-
[10]
Jerome H Friedman. 2001. Greedy function approximatio n: a gradient boosting machine. Annals of statistics (2001), 1189–1232
2001
-
[11]
Flavio Giorgi, Cesare Campagnano, Fabrizio Silvestri , and Gabriele Tolomei. 2024. Natural Language Counterfact ual Explanations for Graphs Using Large Language Models. arXiv preprint arXiv:2410.09295 (2024)
2024 arXiv
-
[12]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoy u Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek- r1: Incentivizing reasoning capability in LLMs via reinfor cement learning. arXiv preprint arXiv:2501.12948 (2025)
2025 arXiv
-
[13]
Gaole He, Nilay Aishwarya, and Ujwal Gadiraju. 2025. Is Conversational XAI All You Need? Human-AI Decision Making W ith a Conversational XAI Assistant. arXiv preprint arXiv:2501.17546 (2025)
2025 arXiv
-
[14]
Robert R Hoffman, Shane T Mueller, Gary Klein, Mohammadr eza Jalaeian, and Connor Tate. 2023. Explainable AI: roles a nd stakeholders, desire- ments and challenges. Frontiers in Computer Science 5 (2023), 1117848
2023
-
[15]
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, A hmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720 (2024)
2024 arXiv
-
[16]
Abderrazak Khediri, Hamda Slimi, Ayoub Yahiaoui, Makh louf Derdour, Hakim Bendjenna, and Charaf Eddine Ghenai. 20 24. Enhancing Machine Learning Model Interpretability in Intrusion Detection Sy stems through SHAP Explanations and LLM-Generated Descrip tions. In 2024 6th Inter-...
2024
-
[17]
Scott Lundberg. 2017. A unified approach to interpretin g model predictions. arXiv preprint arXiv:1705.07874 (2017)
2017 arXiv
-
[18]
Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jo rdanous. 2024. Is Temperature the Creativity Parameter of Large Language Models? arXiv:2405.00492 [cs.CL] https://arxiv.org/abs/2405.0 0492
2024 arXiv
-
[19]
Matthew Renze and Erhan Guven. 2024. The Effect of Sampli ng Temperature on Problem Solving in Large Language Models. arXiv:2402.05201 [cs.CL] https://arxiv.org/abs/2402.0 5201
2024 arXiv
-
[20]
Google Research. [n. d.]. WhatIf Tool. https://pair-c ode.github.io/what-if-tool/. Accessed: 2025-02-20
2025
-
[21]
Why should i trust you?
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin . 2016. " Why should i trust you?" Explaining the predictions of any classifier. InProceedings of the 22nd ACM SIGKDD international conference on knowledg e discovery and data mining . 1135–1144
2016
-
[22]
scikit learn. [n. d.]. Decision Tree Classifier. https: //scikit-learn.org/stable/modules/generated/sklearn .tree.DecisionTreeClassifier.html. Accessed: 2025-02-20
2025
-
[23]
Ashley Suh, Gabriel Appleby, Erik W Anderson, Luca Fine lli, Remco Chang, and Dylan Cashman. 2023. Are metrics enoug h? guidelines for communicating and visualizing predictive models to subjec t matter experts. IEEE Transactions on Visualization and Computer Graphics (2023)
2023
-
[24]
Ashley Suh, Harry Li, Caitlin Kenney, Kenneth Alperin, and Steven R Gomez. 2024. More Questions than Answers? Lesso ns from Integrating Explainable AI into a Cyber-AI Tool. arXiv preprint arXiv:2408.04746 (2024)
2024 arXiv
-
[25]
Gomez, Kevin K
Harini Suresh, Steven R. Gomez, Kevin K. Nam, and Arvind Satyanarayan. 2021. Beyond Expertise and Roles: A Framewor k to Characterize the Stakeholders of Interpretable Machine Learning and their N eeds. Proc. CHI (2021). https://doi.org/10.1145/3411764.3445088
2021
-
[26]
Gustavo F Tondello and Lennart E Nacke. 2018. Gamificati on: Tools and techniques for motivating users. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems . 1–4
2018
-
[27]
Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017. Counterfactual explanations without opening the bla ck box: Automated decisions and the GDPR. Harv. JL & Tech. 31 (2017), 841. 7 Suh et al
2017
-
[28]
Wenzhuo Yang, Jia Li, Caiming Xiong, and Steven CH Hoi. 2 022. Mace: An efficient model-agnostic framework for counter factual explanation. arXiv preprint arXiv:2205.15540 (2022)
2022 arXiv
-
[29]
Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su , Chenhui Chu, and Dong Yu. 2024. Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601 (2024)
2024 arXiv
-
[30]
It’s a Fair Game
Zhiping Zhang, Michelle Jia, Hao-Ping Lee, Bingsheng Y ao, Sauvik Das, Ada Lerner, Dakuo Wang, and Tianshi Li. 2024. "It’s a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Ben efits When Using LLM-Based Conversational Agents. In Proceedings of the CHI C...
2024
-
[31]
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang , Zican Dong, et al
-
[32]
Alexandra Zytek, Sara Pido, Sarah Alnegheimish, Laure Berti-Equille, and Kalyan Veeramachaneni. 2024. Explingo : Explaining AI Predictions using Large Language Models. In 2024 IEEE International Conference on Big Data (BigData) . IEEE, 1197–1208
2024
-
[33]
Alexandra Zytek, Sara Pidò, and Kalyan Veeramachaneni . 2024. LLMs for XAI: Future Directions for Explaining Expla nations. arXiv preprint arXiv:2405.06064 (2024). 8
2024 arXiv
-
[2023]
arXiv preprint arXiv:2303.18223 (2023)
A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)
2023 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.