{"id":"cff015f4-9017-4d14-b4e2-f96042ce3f94","arxiv_id":"2504.12424","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs should interrogate XAI outputs as constructive adversaries, presenting alternative interpretations and limitations, rather than serving as passive translators of feature-attribution explanations.","lead":"This position paper argues that large language models should act as devil's advocates in explainable AI, challenging AI explanations instead of smoothly translating them into natural language. The authors propose prompting strategies that surface uncertainties, biases, and counterfactuals to help users engage critically with AI outputs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central premise that LLM-generated adversarial critiques are accurate enough to improve trust calibration is unverified; Table 1 prompts can yield plausible-but-hallucinated counterfactuals and bias claims.","rationale":"The reader's CONDITIONAL verdict is sound. The weakest assumption I identify is not exactly the one the reader foregrounded. The reader highlights user engagement and acceptance, citing the Buçinca cognitive-forcing friction tradeoff. The paper already acknowledges that tradeoff and cites evidence that cognitive forcing can reduce overreliance, so effectiveness under engagement has some empirical support. More load-bearing is the unexamined accuracy of the adversarial LLM's claims. The entire proposed mechanism—a constructive agitator—produces content that users are meant to weigh against the original explanation. If that content is hallucinated or generic, the intervention cannot improve calibration and can actively worsen it by inducing misplaced distrust. The paper is honest that empirical evaluation is needed, but it does not flag this specific failure mode or propose any grounding or verification component. My proposed benchmark would directly test it. I do not move the verdict: CONDITIONAL remains appropriate because the empirical accuracy question is exactly the condition that needs to be met. I mark agreement as partial because the reader mentions output accuracy as 'another fragile premise,' but the primary weakest assumption listed is user engagement; I place the accuracy premise first as the more fundamental risk.","tokens_in":8565,"tokens_out":4793,"duration_ms":54522,"concrete_test":"Use an open dataset (e.g., UCI Adult) to fit a logistic regression or GBM with a deliberately injected bias and known ground-truth feature effects. For a random sample of instances, compute SHAP and LIME outputs and feed them to an LLM using the Table 1 adversarial prompts. Enumerate every checkable generated claim—counterfactual flips ('if X changed by delta, prediction flips'), statements about SHAP/LIME assumptions, and bias/disparity assertions—and verify each against the fitted model and data via brute-force perturbation and statistical tests. Pre-register a tolerance (e.g., at least 95% of counterfactual and assumption claims must be correct).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's causal claim is that an LLM devil's advocate will reduce overreliance by actively interrogating XAI outputs—presenting alternative interpretations, biases, training-data limitations, and failure cases. For this to work, those adversarial statements must be factually grounded in the actual model and data. The paper provides no mechanism to ensure this. Table 1 instructs the LLM to say, e.g., 'If Feature X were reduced by 10%, the model's decision would flip' or 'SHAP assumes feature independence, which may not hold,' but a pre-trained LLM has no reliable access to the fitted classifier, its training distribution, or the validity conditions of the explainer. The same generative tendencies that the paper identifies as dangerous in translation—smoothing over inconsistencies, producing persuasive but unfaithful narratives—can operate in the adversary's output, producing a confident but incorrect critique. Users are told to maintain 'healthy skepticism' (Sec. 3), but that does not help them distinguish a true counterfactual from a hallucinated one when they lack domain or model knowledge. The paper acknowledges (Sec. 3) that LLMs can produce 'unfaithful or placebic explanations' and nevertheless relies on the same model class to critique them. Thus the intervention could replace one specious narrative with another, or induce misplaced distrust in correct model decisions. This is the load-bearing empirical premise: the accuracy of the adversarial content itself. It is distinct from the user-acceptance concern raised by the reader, because even a fully engaged user is harmed if the adversarial claims are wrong.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This position paper argues that the current trend of using large language models (LLMs) to translate XAI outputs such as SHAP or LIME values into fluent natural-language narratives is insufficient and potentially harmful, because such translations can make AI decisions appear more plausible than transparent and can increase user overreliance and misplaced trust. The authors propose instead that LLMs should act as 'devil's advocates' that actively interrogate AI explanations by surfacing uncertainty, alternative interpretations, potential biases, counterfactual scenarios, and limitations of the explainer itself. The paper reviews prior empirical work on LLM-based explanation translation, conversational XAI assistants, and counterfactual explanations; presents a table of prompting strategies for a devil's advocate; and discusses future directions for workflow design, persona-based explanations, and the tradeoff between cognitive friction and user satisfaction.","tokens_in":8836,"tokens_out":3105,"duration_ms":31537,"significance":"If the proposal is validated, it would reframe the role of LLMs in XAI from passive fluency providers to active critics, directly targeting the documented problem of overreliance caused by persuasive but shallow explanations. The paper is honest about the lack of empirical evaluation and explicitly states that 'empirical evaluations are ultimately needed' (Sec. 3). Its concrete contribution is a set of prompt exemplars in Table 1 that can serve as a springboard for future studies, and it connects the proposal to prior cognitive-forcing research, thereby giving the community a testable starting point. The paper also correctly identifies a key tradeoff: adversarial interaction may reduce overreliance but lower user satisfaction, citing Buçinca et al. [2]. These strengths make the proposal a useful addition to the XAI-HCI discourse, provided the identified gaps are addressed.","major_comments":[{"comment":"The central mechanism requires that the devil's advocate's adversarial claims—counterfactuals, bias analyses, and explainer audits—are factually grounded in the actual model and data, but the paper provides no mechanism to ensure this. Prompts such as 'If Feature X were reduced by 10%, the model's decision would flip' assume the LLM has access to the fitted classifier's decision boundary, which is not the case for a pre-trained LLM. The paper acknowledges (Sec. 3) that LLMs can produce 'unfaithful or placebic explanations' and yet relies on the same model class to critique them, so the concern that the adversary may hallucinate plausible-but-wrong claims is load-bearing for the proposal. The manuscript should either propose a grounding or verification strategy (e.g., constraining the LLM to only restate information computed by a trustworthy explainer, or including explicit uncertainty markers on every claim) or clearly frame this as an open research question that must be solved before the approach can be adopted.","section":"Sec. 3, Table 1"},{"comment":"The causal claim that a devil's advocate will reduce overreliance and improve trust calibration is not tested, and the only directly supporting evidence cited is Chiang et al. [4] in a group decision-making setting, which differs from the XAI explanation scenario. Buçinca et al. [2], cited by the authors themselves, shows that cognitive forcing reduces overreliance but receives the lowest user favorability ratings, so the proposed intervention may be effective only when users tolerate the friction. The manuscript should either temper the abstract's claim that adopting the devil's advocate role will 'reduce overreliance' or specify a falsifiable empirical design, including hypotheses about when the approach helps versus when it backfires (e.g., inducing unwarranted distrust in correct model decisions). As a position paper this is acceptable as a call for research, but the current phrasing overstates the strength of the evidence.","section":"Sec. 3 and Sec. 4"}],"minor_comments":[{"comment":"'easy of interpretability' should be 'ease of interpretability'.","section":"Sec. 1, first bullet"},{"comment":"The sentence 'Four XAI methods were employed (SHAP, PDP [10], Mace [28], the What-If Tool [20], and a Decision Tree [22])' lists five methods; the count or the list should be revised for consistency.","section":"Sec. 2, He et al. paragraph"},{"comment":"There are typos in the 'Scrutinize Assumptions' and 'User-Calibrated Depth' rows: 'le/t_ting' should be 'letting' and 'a/t_tributions' should be 'attributions'.","section":"Table 1"},{"comment":"The author list for Crisan et al. is incomplete ('Anamaria Crisan, Nathan Butters, and Zoe') and the entry appears truncated; the full author and title should be provided.","section":"References, [6]"},{"comment":"'there will inevitable difficulties' should be 'there will inevitably be difficulties'.","section":"Sec. 4, 'Design for a Productive Adversarial Process'"}],"recommendation":"major_revision","confidential_remarks":"This is a workshop-style position paper, so the lack of a user study is not by itself disqualifying; however, the proposal's core mechanism depends on the factual accuracy of LLM-generated adversarial content, and the current manuscript neither solves nor fully frames this problem. The authors should be asked to clarify the epistemic status of the devil's advocate's claims and to provide a concrete validation agenda. The paper is well-written and the prompt table is useful, so with those revisions it could be a solid contribution to the HCXAI community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a workshop paper, so I read it as a position statement rather than a finished result. What's new is the application of Chiang et al.'s LLM devil's advocate to XAI explanation interrogation, plus the concrete prompting strategies in Table 1. That table is a genuinely useful starting point: it gives researchers something to try, not just a vague suggestion. The paper also does a good job synthesizing recent work on LLM-based explanation translation, especially the evidence that fluency can increase overreliance, and it is honest that no empirical evaluation exists.\n\nThe main soft spot is exactly what the stress-test note says: the adversarial LLM's critiques must be accurate for the intervention to help, and the paper provides no mechanism to ensure that. Table 1 includes prompts like 'If Feature X were reduced by 10%, the model's decision would flip,' but a pretrained LLM has no reliable access to the classifier's decision boundary or the explainer's validity conditions unless explicitly grounded with tools. The paper acknowledges that LLMs can produce unfaithful or placebic explanations, then proposes using the same model class as the critic. That is a real gap, not a minor one. A user who is fully engaged with an adversarial explanation is still harmed if the critique is a confident hallucination.\n\nA second, related concern is the paper's optimism about 'healthy skepticism' and the devil's advocate framing. As the paper itself notes from Buçinca et al., cognitive forcing interventions can reduce overreliance but get low user favorability. The tradeoff is real and the paper doesn't offer much beyond gamification ideas. Still, that issue is about adoption; the accuracy problem is prior to it.\n\nThe reader's verdict and soundness score look fair to me, but I'd weigh the hallucination concern slightly heavier than the reader did. The paper's own caveat that 'empirical evaluations are ultimately needed' is accurate, though it doesn't fix the absence of a grounding strategy.\n\nWho is this for? A workshop audience or a reading group that wants a concise framing of the devil's advocate direction. It is not a validated intervention and should not be cited as evidence that the approach works. But as a position paper, it deserves serious refereeing: the research question is well posed, the literature context is solid, and the prompt table is a reasonable research scaffold. I would accept it for peer review at a workshop or as a short position paper, and I would tell the authors that empirical validation—especially of the accuracy of adversarial claims—is the part that matters most.","headline":"A short, honest workshop position paper proposing LLMs as devil's advocates for XAI explanations; the idea is promising but the central claim is untested and there is no mechanism to keep adversarial critiques accurate.","tokens_in":9379,"tokens_out":1832,"would_cite":true,"duration_ms":22866,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-based XAI translation can cause overreliance; this paper argues LLMs should act as devil's advocates that challenge AI explanations.","keywords":["large language models","explainable AI","devil's advocate","overreliance","trust calibration","counterfactual explanations","XAI translation","cognitive forcing"],"falsifier":"A controlled study comparing the same explainer output delivered as a fluent narrative versus an adversarial devil's-advocate form, measuring users' decision accuracy and trust calibration on cases where the model is wrong; if adversarial users overrely just as much as translation users, the central claim fails.","tokens_in":8380,"feed_emoji":"⚖️","tokens_out":7186,"duration_ms":71201,"temperature":0.7,"pith_summary":"This paper argues that the standard way of using large language models in explainable AI—translating raw feature-attribution outputs such as SHAP or LIME values into fluent natural-language explanations—can make AI outputs feel more plausible without making them more transparent. Reviewing recent empirical studies, it claims that fluent LLM narratives and conversational XAI assistants increase user engagement and trust but also produce misplaced trust in incorrect predictions, a form of overreliance. The paper's central proposal is to repurpose LLMs as adversarial devil's advocates that actively interrogate explanations, surface uncertainty, biases, limitations, and counterfactuals, and push users to engage critically. If that works, adversarial presentation would reduce overreliance and give users better-calibrated trust in AI explanations.","feed_headline":"LLMs should challenge AI explanations, not just translate them","feed_subtitle":"Fluent XAI narratives feel more trustworthy without improving decisions; adversarial prompts could fix that.","key_machinery":"The mechanism is a set of adversarial prompting strategies, summarized as a table, that re-frame the LLM's persona from helpful assistant to helpful adversary. The strategies are uncertainty awareness, alternative explanations, bias detection, counterfactual thinking, scrutinizing assumptions, explanation audit, and user-calibrated depth, with example prompts that tell the LLM to surface what a smooth translation would hide. This devil's-advocate prompt regime does the work in the argument: it converts the LLM's tendency to smooth over inconsistencies into a means of surfacing them, and it is offered as a springboard for few-shot training and future empirical testing.","core_discovery":"The central claim is that LLMs should stop being passive translators and become constructive agitators. In a decision-support setting, an LLM should not take the explainer's output as ground truth and dress it up as a readable story; it should challenge the output, present multiple interpretations, point out where the explanation may be unreliable, and offer counterfactual probes. The paper grounds this claim in studies showing that natural-language and conversational explanations raise trust without improving understanding, so users overrely on wrong predictions. It also points to a prior devil's advocate study as evidence that adversarial LLMs can be seen as useful collaborators. The paper is explicitly a position paper with proposed prompting strategies; it does not report a new user study confirming them.","pith_inferences":["Editorial inference: because the paper provides no user study, the decisive empirical test is to compare final decision quality under translated versus adversarial explanations; that test would separate the value of the role change from the value of showing any extra information at all.","Editorial inference: the paper's own cited evidence implies the adversarial claims themselves could be wrong; a robust version might add a second verification pass that grounds every adversarial statement in explainer output, otherwise a confident but false critique could destabilize trust further.","Editorial inference: an untested corollary is that adversarial prompting could aid model developers debugging a classifier, since they may tolerate the cognitive friction that end users reject.","Editorial inference: the overreliance problem may be dominated by users' default trust in any fluent authority, so the devil's advocate role only works if users read it as an adversarial persona and not as just another confident AI voice."],"forward_implications":["LLM-based XAI systems should be judged by whether they reduce overreliance and improve trust calibration, not by how fluent or preferred their explanations are.","Conversational XAI assistants should include an adversarial mode that challenges the model's own outputs, not only a mode that answers user questions in a supportive tone.","Generated explanations should carry uncertainty statements, alternative explanations, and known explainer limitations rather than a single plausible narrative.","Designers can use game-like structures or progressive difficulty to make adversarial engagement rewarding enough to offset the lower satisfaction users give to cognitive forcing.","Different audiences, such as clinicians versus patients, could receive adversarial explanations calibrated to their expertise surfaced from the same XAI output."],"supporting_citations":[{"why":"Documents that cognitive forcing interventions reduce overreliance but receive low user favorability, framing the central tradeoff the proposed role must overcome.","marker":"[2]"},{"why":"Shows an LLM-powered devil's advocate was perceived as a useful collaborator in group decisions, the prior existence proof this paper adapts to XAI.","marker":"[4]"},{"why":"Reports an N=10 study where LLM-presented counterfactual explanations with LIME did not improve understanding or trust calibration, motivating the adversarial role.","marker":"[6]"},{"why":"Provides the N=306 study of conversational XAI showing higher engagement and perceived trust but misplaced trust in incorrect predictions.","marker":"[13]"},{"why":"Defines SHAP, the feature-attribution method whose outputs are the raw material being translated or challenged.","marker":"[17]"},{"why":"Defines LIME, the explainer used in the counterfactual studies and a baseline raw output for comparison.","marker":"[21]"},{"why":"Introduces the Explingo pipeline for generating and scoring SHAP-based narratives, showing that few-shot exemplars are needed for sensible translations.","marker":"[32]"},{"why":"Provides the comparative study in which users preferred LLM narrative explanations over SHAP plots, the core translation trend the paper critiques.","marker":"[33]"}],"fun_headline_variants":["LLMs as devil's advocates: challenge AI explanations, don't just translate","AI explanations need agitators, not translators","Make LLMs interrogate AI explanations, not prettify them","Devil's advocate LLMs to fight overreliance on AI","From translator to agitator: LLMs should challenge AI explanations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central proposal depends on users actually engaging with adversarial output and recalibrating their trust from it, and on the LLM's adversarial claims being accurate enough to be worth believing.","fun_headline_variants_meta":{"raw":{"variants":["LLMs as devil's advocates: challenge AI explanations, don't just translate","AI explanations need agitators, not translators","Make LLMs interrogate AI explanations, not prettify them","Devil's advocate LLMs to fight overreliance on AI","From translator to agitator: LLMs should challenge AI explanations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00015,"raw_usage":{"total_tokens":1161,"prompt_tokens":872,"completion_tokens":289,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":203}},"tokens_in":488,"tokens_out":289,"duration_ms":3407,"temperature":1.0,"reasoning_tokens":203,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:31:21.586002+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled study comparing the same explainer output delivered as a fluent narrative versus an adversarial devil's-advocate form, measuring users' decision accuracy and trust calibration on cases where the model is wrong; if adversarial users overrely just as much as translation users, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents that cognitive forcing interventions reduce overreliance but receive low user favorability, framing the central tradeoff the proposed role must overcome."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows an LLM-powered devil's advocate was perceived as a useful collaborator in group decisions, the prior existence proof this paper adapts to XAI."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports an N=10 study where LLM-presented counterfactual explanations with LIME did not improve understanding or trust calibration, motivating the adversarial role."},{"cited_title":"Why should i trust you?","cited_arxiv_id":null,"evidence_quote":"Defines LIME, the explainer used in the counterfactual studies and a baseline raw output for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Explingo pipeline for generating and scoring SHAP-based narratives, showing that few-shot exemplars are needed for sensible translations."}],"review_version":1}