This paper introduces an automated evaluation framework with extraction-based faithfulness metrics, perplexity for assumptions, and embedding-based human similarity, and shows it can reveal LLM sign self-correction on manipulated SHAP tables.
Natural Language Counterfactual Explanations for Graphs Using Large Language Models
1 Pith paper cite this work. Polarity classification is still indexing.
abstract
Explainable Artificial Intelligence (XAI) has emerged as a critical area of research to unravel the opaque inner logic of (deep) machine learning models. Among the various XAI techniques proposed in the literature, counterfactual explanations stand out as one of the most promising approaches. However, these "what-if" explanations are frequently complex and technical, making them difficult for non-experts to understand and, more broadly, challenging for humans to interpret. To bridge this gap, in this work, we exploit the power of open-source Large Language Models to generate natural language explanations when prompted with valid counterfactual instances produced by state-of-the-art explainers for graph-based models. Experiments across several graph datasets and counterfactual explainers show that our approach effectively produces accurate natural language representations of counterfactual instances, as demonstrated by key performance metrics.
citation-role summary
citation-polarity summary
fields
cs.CL 1years
2024 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
How good is my story? Towards quantitative metrics for evaluating LLM-generated XAI narratives
This paper introduces an automated evaluation framework with extraction-based faithfulness metrics, perplexity for assumptions, and embedding-based human similarity, and shows it can reveal LLM sign self-correction on manipulated SHAP tables.