{"id":"3366f496-17dc-4fc6-9d7e-b8b2af35a641","arxiv_id":"2412.10220","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"This paper introduces an automated evaluation framework with extraction-based faithfulness metrics, perplexity for assumptions, and embedding-based human similarity, and shows it can reveal LLM sign self-correction on manipulated SHAP tables.","lead":"Researchers propose a set of automated metrics to score how faithfully and plausibly large language models turn SHAP explanations into plain-language narratives. The metrics can compare different LLMs and detect when models silently correct or distort the explanation, a form of hallucination.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The extraction-model validation only checks coarse 'any error' flags, not exact extracted rank/sign/value; if gpt-4o extraction silently corrects implausible signs, the headline sign self-correction finding is an artifact.","rationale":"The paper presents a coherent and reproducible framework for automated evaluation of LLM-generated XAI narratives, with faithfulness metrics anchored to an external SHAP table and code provided. The central demonstration, however, rests on the sign self-correction result in Section IV B, which depends on exact extraction accuracy. The reported validation in Section IV A only establishes coarse binary error detection, and the same-model generation/extraction design leaves a plausible mechanism for inflated measured sign accuracy on manipulated narratives. This is a sharper version of the reader's concern about circularity: even without same-model circularity, the absence of per-item human validation of extracted signs means the key empirical finding could be an artifact of the extractor. The issue is repairable with additional validation, and the framework remains a useful proof-of-concept, so the conditional verdict remains appropriate. I have not identified a reason to move the verdict to accept or reject.","tokens_in":17570,"tokens_out":4921,"duration_ms":45988,"concrete_test":"Select 30 manipulated and 30 standard narratives across the four generation models from Section IV B. Have two independent human annotators extract, for every feature mentioned, the literal rank/sign/value stated in the text, with disagreements adjudicated. Compute gpt-4o extraction's agreement with this human gold standard per item, not just the binary 'any error' flag. Then recompute RA/SA/VA and the manipulated SA. Separately, run the same extraction with a different extractor (e.g., Claude Sonnet 3.5 or Llama-3-70B) on the same narratives. If gpt-4o sign extraction disagrees with human literal reading on more than a small fraction of manipulated narratives (e.g., >2 of 60), or if the alternative extractor yields substantially lower manipulated SA than gpt-4o, the self-correction claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV A validates the extraction model as a binary classifier: whether a narrative has any rank/sign error (Table I), reporting 60/60 true negatives on gpt-4o-generated manipulated narratives and 1/53 false negatives on human narratives. The downstream faithfulness metrics RA/SA/VA (Eq. 1, Tables II-VI) require exact per-feature rank/sign/value extraction, but exact extraction accuracy is never validated against human labels. This gap is especially dangerous for the paper's key finding: for manipulated SHAP tables, sign accuracy is ~0.30 instead of the ~0.04 expected by chance (Table III), interpreted as LLMs self-correcting implausible signs. If the gpt-4o extractor, when reading 'scoring five goals reduces the probability of winning,' uses its parametric knowledge to extract a positive sign because the statement is implausible, SA is inflated independent of the generation model. The TN test does not control for this: a narrative can contain multiple sign errors and still be correctly flagged if one error is detected. The fact that the same model (gpt-4o) generates and extracts the manipulated narratives compounds the risk of self-consistent bias. No human annotation of exact extractions is reported, so the measured SA on manipulated narratives could reflect extractor bias rather than generator hallucination. The paper's own caveat that 'more thorough experiments are needed' (Section IV A) underscores this gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for automated evaluation of LLM-generated XAI narratives for tabular classification. It introduces an extraction LLM that recovers per-feature rank, sign, value, and assumptions from a narrative, and then computes faithfulness metrics (RA/SA/VA via Eq. 1). It also evaluates assumption quality via perplexity of extracted assumption sentences (Eq. 2) and human similarity via embedding cosine similarity, comparing against BLEURT. The authors validate the extraction model on 60 manipulated and 60 human narratives (Table I), validate perplexity on 47 manually modified assumptions (Fig. 5), and show that embedding-based matching outperforms BLEURT (Fig. 6). They then compare four LLMs and two prompt types, and report that when the SHAP table is manipulated (rank/sign inversion), sign accuracy stays around 0.30 instead of the chance-level ~0.04 they estimate, which they interpret as LLMs self-correcting implausible signs—a hallucination risk.","tokens_in":17727,"tokens_out":4239,"duration_ms":38782,"significance":"If the metrics are trustworthy, the framework provides a scalable, human-free evaluation pipeline for XAI narratives, with the extraction-based faithfulness scores and embedding similarity being concrete and reusable. The paper has several strengths: the faithfulness ground truth is an external SHAP table, no constants are fitted, per-dataset results are reported, and the authors include explicit caveats about limitations (e.g., Section IV A and the Conclusion on perplexity). The self-correction finding, if robust, is a practically important failure mode for XAI narratives. However, the current validation leaves exact extraction accuracy unmeasured and relies on small samples without statistical tests, so the strength of the central claims is not yet established. The framework is promising but needs additional validation before it can be regarded as ready for automated narrative quality control.","major_comments":[{"comment":"The extraction model is validated only as a binary classifier over an 'any error' flag, but the downstream faithfulness metrics in Eq. (1) and Tables II–VI require exact per-feature rank/sign/value extraction. The 60/60 true-negative result does not establish exact-extraction accuracy: a narrative with multiple errors is correctly flagged even if only one error is recovered, and no human annotation of exact extractions is reported. Because gpt-4o is used both to generate and to extract the manipulated narratives, the measured SA of ~0.30 on manipulated tables (Table III) may reflect the extractor's parametric correction of implausible signs rather than the generator's self-correction. This is load-bearing for the central hallucination claim. Please add human-labeled exact extractions (or a second, independent extraction model) and report per-feature agreement for rank, sign, and value.","section":"Section IV A, Table I and Eq. (1)"},{"comment":"The assumption validation uses 47 manually manipulated assumptions and shows a perplexity increase in that idealized setting, but the same metric behaves inconsistently on the manipulated narratives: in Table III, ΔPPL is negative for several models and iterations (e.g., llama-3-70b under both PPL(L) and PPL(M)), and the Conclusion concedes that 'perplexity does not appear to yield consistent behavior across models.' As stated, the claim that perplexity 'can be expected to work' for detecting unreasonable assumptions is not supported for the actual generation scenario. Please provide a validation on generated/manipulated narratives or temper the claim accordingly.","section":"Section IV A, Fig. 5, Table III, Conclusion"},{"comment":"All quantitative comparisons rest on samples of 60 narratives (20 per dataset) and 47 assumptions, with no confidence intervals or significance tests; the ranges in Table III are min|max over four runs at T=0, which is not a statistical error estimate. Consequently, statements such as 'the long prompt consistently achieves a better performance' (Section IV B) and 'cosine similarity outperforms BLEURT' (42/60 vs 29/60, Fig. 6) are not established beyond descriptive comparison. Please add bootstrap confidence intervals or significance tests for the headline comparisons, especially the sign self-correction rates that motivate the hallucination finding.","section":"Section IV A and IV B"}],"minor_comments":[{"comment":"Tables IV and V are identical in the manuscript, both titled 'Student'; one of them presumably should report the Fifa dataset. This needs to be corrected to support the per-dataset analysis.","section":"Tables IV and V"},{"comment":"The true-positive rate is derived by assuming that the false-positive rate is low, but the false-positive rate was measured on manipulated gpt-4o narratives, not on human-written narratives; please clarify why this transfer of an error-rate estimate is justified.","section":"Section IV A"},{"comment":"The denominator n - sum of phi entries is zero if all extracted values for a feature are phi; please specify the convention in that edge case (e.g., the term is skipped).","section":"Eq. (1)"},{"comment":"The 'naive expectation' of about 4% sign accuracy under the manipulation is not derived; please show the baseline calculation (e.g., assuming random sign generation or independent sign flips) so the reader can verify the claimed deviation.","section":"Section IV B, Table III"},{"comment":"The phrase 'filter out all features that occur less than once in all 20 Student instances' is unclear; please rephrase, for example as 'features with fewer than 16 occurrences in the pooled 320 narratives.'","section":"Fig. 8 caption"},{"comment":"There are minor typos: 'give raise' should be 'give rise' (Introduction), and 'explanatizons' should be 'explanations' (Fig. 6 caption).","section":"Introduction and Fig. 6 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the framework is a useful contribution, but the validation gaps around exact extraction accuracy and statistical support are load-bearing for the headline findings. I would encourage the authors to address these with concrete additions (human-annotated exact extractions, a second extraction model, and significance tests) rather than with a rewriting only. The same-model generation/extraction risk is the main correctness concern and should be handled explicitly, not just as a caveat."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper builds a sensible, fully automated evaluation pipeline for LLM-generated XAI narratives and demonstrates it on three tabular datasets. The genuinely new piece is decomposing faithfulness into rank/sign/value agreement using an extraction LLM, and showing that modern embeddings beat BLEURT at matching narratives. The central 'sign self-correction' finding—models correcting implausible manipulated SHAP signs ~30% of the time—is interesting but needs stronger validation before I'd call it a real hallucination risk.\n\nWhat's good: The framework is clearly specified, the SHAP ground truth is external, code and data are in a repo, and the authors are honest about limitations. They run repeated generations and report ranges, set temperature to zero, use two perplexity models, and explicitly flag where perplexity underperforms. The comparison to BLEURT is a concrete, reproducible result.\n\nWhere the soft spots are: The biggest issue is exactly what the stress-test flags. The extraction model is validated only as a binary 'any error' classifier, not for exact per-feature rank/sign/value extraction. Table I shows 60/60 true negatives and 1/53 false negatives, but that doesn't tell us whether the extractor returns the right sign for each feature. If gpt-4o extracts 'scoring five goals reduces probability' and silently flips the sign to positive because the statement is implausible, then the measured sign agreement on manipulated narratives is inflated regardless of what the generation model actually wrote. The fact that the same model generates and extracts the manipulated narratives makes this worse. Without human annotation of exact extractions, the headline sign self-correction number (SA ≈ 0.30 vs expected 0.04) can't be trusted as evidence about the generator.\n\nThe other issues are more minor: small samples (60 narratives per condition, 20 per dataset), no confidence intervals or significance tests, and Tables IV and V appear duplicated—the Student table repeats the Fifa numbers verbatim, which is an obvious error.\n\nDoes the paper hold up? The framework itself is a reasonable proof of concept. The extraction-validation gap doesn't sink the idea; it means the paper overclaims when it says the metrics 'allow us to identify new challenges.' I'd read the self-correction finding as a hypothesis to test with a better-validated extractor, not a result.\n\nRecommendation: Send it to peer review, but with revisions. The authors need to either validate exact extraction against human annotations or temper the hallucination claim, fix the table duplication, and add basic uncertainty estimates. It's the kind of paper a good reviewer can meaningfully improve.","headline":"A useful proof-of-concept framework for automated XAI narrative evaluation, but the headline sign self-correction finding needs better extraction validation before it can be trusted.","tokens_in":18348,"tokens_out":3689,"would_cite":true,"duration_ms":30432,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLM-written explanations of SHAP tables can be scored automatically by extracting each feature's rank, sign, value, and assumption from the narrative, and that doing so reveals that LLMs often silently correct…","keywords":["explainable AI","SHAP narratives","LLM faithfulness evaluation","extraction-based metrics","hallucination detection","text embeddings","perplexity","narrative generation"],"falsifier":"Hand-annotate a fresh sample of narratives from a different generation model family and compare the extraction model's outputs to the human labels; if rank/sign/value agreement with the labels drops well below the roughly 98% reported, then the faithfulness scores in the comparison tables are not portable. Also rerun the manipulated-table experiment with a prompt instruction that the narrative must state the given signs verbatim; if sign agreement then returns near 100%, the self-correction is a style preference, not an irremovable bias.","tokens_in":17274,"feed_emoji":"📊","tokens_out":6709,"duration_ms":64540,"temperature":0.7,"pith_summary":"This paper proposes a fully automated way to grade the stories that LLMs write to explain model predictions from SHAP tables. The key mechanism is an extraction LLM that pulls out each feature's rank, sign, numeric value, and causal assumption from a narrative, then checks them against the original SHAP table. Across three tabular datasets and four LLMs, the metrics behave as expected: structured prompts improve faithfulness, embedding similarity matches the right human reference narrative more often than BLEURT, and when SHAP tables are manipulated, faithfulness scores collapse. A surprising result is that LLMs often 'correct' the manipulated sign of a feature, suggesting they sacrifice fidelity for plausibility. If the metrics are trustworthy, they provide a scalable substitute for human evaluation and expose a previously unnoticed hallucination failure mode.","feed_headline":"Automated metrics catch LLM narrators rewriting SHAP signs","feed_subtitle":"A new extraction-based framework grades XAI stories without human surveys and exposes a subtle hallucination risk.","key_machinery":"The extraction pipeline: a second LLM, always gpt-4o, reads the narrative and outputs a structured dictionary with, for each feature mentioned, a rank $r_j$, a sign $s_j\\in\\{-1,1\\}$, a value $v_j$ (or null), and an assumption sentence (or null). Downstream metrics then score these extractions: rank agreement (RA), sign agreement (SA), and value agreement (VA) against the ground-truth SHAP table, perplexity for assumptions, and cosine similarity of narrative embeddings for human similarity. The extraction step is the load-bearing object: it converts free text into the structured quantities that every faithfulness score depends on.","core_discovery":"This paper claims that narrative quality can be measured automatically by decomposing an LLM-written explanation into an extractable dictionary of feature, rank, sign, value, and assumption, and scoring faithfulness against the source SHAP table, human similarity via embeddings, and assumption plausibility via perplexity. Using gpt-4o as the extractor, the authors report high validation accuracy (60/60 faulty narratives flagged, 1 false negative in 53). The central discovery is the sign self-correction phenomenon: when the SHAP table passed to a generation LLM has its feature signs inverted, the LLM writes a narrative whose implied signs agree with the manipulated table only about a quarter to a third of the time, far below the roughly 96% predicted from normal narratives. The authors interpret this as evidence that the models override the given explanation with their own internal knowledge, a form of hallucination that the faithfulness metrics bring into view.","pith_inferences":["A deployed risk: if an upstream attribution contains errors (or a model has learned spurious correlations), an LLM narrator may 'fix' them into a more coherent but wrong story. The resulting text can sound more faithful than the model actually is, a failure that human readers, and possibly this metric suite, can miss.","The true-negative validation uses gpt-4o for both generating and extracting manipulated narratives; a shared model bias could inflate the reported extraction accuracy. A fairer test would hand-label narratives from several model families and measure extraction accuracy per family.","The extract-then-score design is generic enough to be embedded in agentic pipelines: regenerate the narrative until faithfulness thresholds are met, or apply the same extraction to other explanation formats such as LIME or counterfactual rules.","Embedding similarity could become a proper XAI narrative metric by fine-tuning a small regression head on top of embeddings to predict rank and sign agreement, rather than using raw cosine distance."],"forward_implications":["The metrics can act as an automated validation layer before a narrative reaches a user, replacing or complementing slow, expensive human surveys.","Longer, structured generation prompts measurably improve rank and sign faithfulness across all tested models, giving a concrete lever for prompt design.","Because LLMs tend to silently correct implausible signs, a smooth-sounding narrative can misrepresent the underlying explanation; faithfulness scores on the narrative alone will not reveal that the original SHAP table was wrong.","Perplexity works for detecting deliberately irrational assumptions in isolation, but it does not robustly flag the assumptions inside real generated narratives, so it should not be used as a standalone plausibility check.","Modern embedding-based cosine similarity already matches the correct human narrative more reliably than BLEURT and can partially sense when a narrative was written from manipulated data."],"supporting_citations":[{"why":"Supplies the SHAP explanation that narratives must be faithful to.","marker":"[1]"},{"why":"Defines the narrative-generation prompt style and the three datasets used.","marker":"[7]"},{"why":"Closest prior work on tabular XAI narratives, whose manual faithfulness analysis and BLEURT-based similarity this paper automates and extends.","marker":"[6]"},{"why":"Source of the rank- and sign-agreement metrics adapted as faithfulness downstream metrics.","marker":"[23]"},{"why":"Supports the premise that LLMs can extract structured information at near-human accuracy.","marker":"[19]"},{"why":"Analogous extraction-based faithfulness evaluation for model explanations, grounding the extraction approach.","marker":"[20]"},{"why":"BLEURT, the reference trained similarity metric that embedding cosine similarity is compared against.","marker":"[17]"},{"why":"Empirical results on knowledge conflicts that the sign self-correction hypothesis is tied to.","marker":"[38]"}],"fun_headline_variants":["LLM narrators override SHAP signs, new metric shows","Automated metrics expose LLM XAI hallucination","XAI stories checked by extraction: sign errors surface","No surveys needed: metrics grade AI explanations","When LLMs rewrite SHAP signs: a hidden hallucination"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire faithfulness pipeline assumes the extraction model—gpt-4o in every experiment—correctly recovers rank, sign, value, and assumptions from a narrative, and that the small validation set (60 manipulated plus 60 human stories) is representative enough to guarantee this.","fun_headline_variants_meta":{"raw":{"variants":["LLM narrators override SHAP signs, new metric shows","Automated metrics expose LLM XAI hallucination","XAI stories checked by extraction: sign errors surface","No surveys needed: metrics grade AI explanations","When LLMs rewrite SHAP signs: a hidden hallucination"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000519,"raw_usage":{"total_tokens":2452,"prompt_tokens":824,"completion_tokens":1628,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":1550}},"tokens_in":440,"tokens_out":1628,"duration_ms":11537,"temperature":1.0,"reasoning_tokens":1550,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:12:17.804329+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hand-annotate a fresh sample of narratives from a different generation model family and compare the extraction model's outputs to the human labels; if rank/sign/value agreement with the labels drops well below the roughly 98% reported, then the faithfulness scores in the comparison tables are not portable. Also rerun the manipulated-table experiment with a prompt instruction that the narrative must state the given signs verbatim; if sign agreement then returns near 100%, the self-correction is a style preference, not an irremovable bias.","supporting_citations":[{"cited_title":"The inadequacy of shapley values for explainabil- ity, 2023","cited_arxiv_id":null,"evidence_quote":"Defines the narrative-generation prompt style and the three datasets used."},{"cited_title":"The disagreement problem in ex- plainable machine learning: A practitioner’s perspective","cited_arxiv_id":null,"evidence_quote":"Source of the rank- and sign-agreement metrics adapted as faithfulness downstream metrics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SHAP explanation that narratives must be faithful to."},{"cited_title":"Keane, Eoin M","cited_arxiv_id":null,"evidence_quote":"Supports the premise that LLMs can extract structured information at near-human accuracy."},{"cited_title":"Do models explain them- selves? Counterfactual simulatability of natural language explanations","cited_arxiv_id":null,"evidence_quote":"Analogous extraction-based faithfulness evaluation for model explanations, grounding the extraction approach."},{"cited_title":"Explaining ma- chine learning models with interactive natural language conversations using talktomodel","cited_arxiv_id":null,"evidence_quote":"BLEURT, the reference trained similarity metric that embedding cosine similarity is compared against."},{"cited_title":"Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts","cited_arxiv_id":null,"evidence_quote":"Empirical results on knowledge conflicts that the sign self-correction hypothesis is tied to."}],"review_version":1}