Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness

Jaechang Kim; Jaewoong Cho; Jungseul Ok; Seungjoon Lee; Sunung Mun

REVIEW 1 major objections 2 minor 19 references

FAX raises simulation faithfulness of LLM explanations from 0.20 to 0.46 by verifying claims against faithful tools on a new open-world RL benchmark.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-29 13:20 UTC pith:7MDQTXIR

load-bearing objection FAX adds claim-level verification and a new open-world RL benchmark for agentic XAI, with a reported faithfulness lift that rests on untested tool completeness. the 1 major comments →

arxiv 2605.27879 v1 pith:7MDQTXIR submitted 2026-05-27 cs.AI

Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness

Jaechang Kim , Sunung Mun , Seungjoon Lee , Jaewoong Cho , Jungseul Ok This is my paper

classification cs.AI

keywords faithful XAIagentic XAILLM explanationsverification methodopen-world benchmarkreinforcement learningmodel faithfulnessCRAFTER-XAI-Bench

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces FAX, a framework that decomposes LLM-generated draft explanations into individual claims and cross-checks each against inherently faithful verification tools before final output. It also presents CRAFTER-XAI-Bench, an open-world reinforcement learning benchmark featuring complex policies, diverse goals, and challenging scenarios designed to measure model-specific faithfulness. On this benchmark FAX achieves the reported gain in simulation faithfulness while preserving informativeness, relevance, and fluency. In contrast, three tabular benchmarks allow methods to appear strong by conflating general task accuracy with fidelity to the target model's actual behavior. The central argument is that explicit verification against the model itself is required for reliable agentic XAI.

Core claim

FAX decomposes draft explanations into claims and cross-checks them against inherently faithful tools, filtering unsupported or contradictory claims before final generation. On CRAFTER-XAI-Bench this raises simulation faithfulness from 0.20 for the strongest baseline to 0.46 while maintaining high informativeness, relevance, and fluency. The work further shows that existing tabular benchmarks can conflate task accuracy with model-specific faithfulness, implying that faithfulness benchmarks must test explanations directly against the behavior of the target model.

What carries the argument

Claim decomposition followed by verification against inherently faithful tools that filters unsupported or contradictory claims in LLM explanations.

Load-bearing premise

The verification tools are inherently faithful and sufficient to evaluate every claim the LLM explainer might produce.

What would settle it

A set of explanations that pass all FAX verification steps yet still produce simulation outcomes that diverge from the target model's actual behavior on CRAFTER-XAI-Bench would falsify the claimed faithfulness gain.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Explicit verification of claims is essential for faithful agentic XAI systems.
Faithfulness benchmarks must evaluate explanations against the target model's own behavior rather than proxy tasks.
On complex open-world RL policies, verification improves simulation faithfulness without harming informativeness or fluency.
Prior agentic XAI methods risk amplifying unfaithful explanations when LLMs generate natural-language output from unreliable XAI signals.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same claim-verification pattern could be tested on non-RL domains such as vision or language models where ground-truth simulators exist.
If verification tools miss subtle interactions between claims, the method may still allow partially unfaithful explanations to reach users.
Designers of future agentic systems might integrate verification as a standard filtering layer rather than relying on post-hoc evaluation.
Benchmarks that require matching an explainer's predictions to an executable model policy could become a template for other faithfulness evaluations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

FAX adds claim-level verification and a new open-world RL benchmark for agentic XAI, with a reported faithfulness lift that rests on untested tool completeness.

read the letter

FAX decomposes LLM explanations into claims and drops the ones that fail checks from tools like simulators or policy queries. It also ships CRAFTER-XAI-Bench, an open-world RL environment built to test whether explanations match the target model's actual behavior rather than just task performance.

The benchmark is the clearer advance. Prior tabular setups can reward explanations that are correct on the task but not specific to the model, and the paper flags this issue directly. The headline number on the new benchmark is a rise in simulation faithfulness from 0.20 to 0.46 while holding informativeness, relevance, and fluency steady.

The soft spot is exactly the one in the stress-test note. The whole gain depends on the verification tools catching every unsupported claim without false negatives. In open-world RL with partial observability and diverse goals, that coverage is not shown to be complete, and the abstract gives no separate validation of the tools or experimental details such as error bars. Without those, the 0.46 figure is hard to interpret.

The work is aimed at people building or evaluating LLM-based explanations for agents. It deserves peer review because the verification principle and the benchmark design are worth checking in full, even if the current evidence is preliminary and the tool assumption needs more scrutiny.

Referee Report

1 major / 2 minor

Summary. The paper proposes Faithful Agentic XAI (FAX), a framework that decomposes LLM draft explanations into individual claims and filters unsupported or contradictory ones via cross-checks against inherently faithful tools (e.g., environment simulators and policy queries) prior to final output generation. It introduces CRAFTER-XAI-Bench, a new open-world RL benchmark featuring complex policies, diverse goals, and challenging scenarios, to evaluate model-specific faithfulness. On this benchmark FAX raises simulation faithfulness from 0.20 (strongest baseline) to 0.46 while preserving informativeness, relevance, and fluency; on three tabular benchmarks it remains competitive with prior agentic XAI methods, although the authors observe that tabular settings can conflate task accuracy with model-specific faithfulness.

Significance. If the verification approach can be shown to generalize beyond the reported settings, the work would provide a concrete, tool-based method for mitigating unfaithful natural-language explanations in agentic XAI and would supply a dedicated benchmark that directly tests explanations against target-model behavior rather than proxy task performance. The explicit separation of faithfulness from informativeness/fluency metrics is a useful contribution to evaluation methodology in the field.

major comments (1)

[FAX framework description and experimental setup on CRAFTER-XAI-Bench] The headline quantitative result (simulation faithfulness 0.46 vs. 0.20 on CRAFTER-XAI-Bench) is presented as evidence that explicit verification improves faithfulness. This interpretation rests on the untested premise that the chosen verification tools detect every unsupported or contradictory claim without false negatives. In open-world RL with partial observability and diverse policy behaviors, gaps in tool coverage could allow unfaithful claims to pass, rendering the measured gain sensitive to the particular tool set rather than to the verification principle itself. No independent coverage analysis or ablation on tool completeness is supplied.

minor comments (2)

[Abstract] The abstract states numerical improvements without reference to number of runs, error bars, or statistical tests; the full experimental section should make these details explicit for reproducibility.
[Method] Clarify the precise definition and selection criteria for 'inherently faithful tools' so that readers can assess their applicability to new domains.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for highlighting the dependence of our faithfulness gains on verification-tool coverage. We agree this is an important untested assumption and address it directly below.

read point-by-point responses

Referee: [FAX framework description and experimental setup on CRAFTER-XAI-Bench] The headline quantitative result (simulation faithfulness 0.46 vs. 0.20 on CRAFTER-XAI-Bench) is presented as evidence that explicit verification improves faithfulness. This interpretation rests on the untested premise that the chosen verification tools detect every unsupported or contradictory claim without false negatives. In open-world RL with partial observability and diverse policy behaviors, gaps in tool coverage could allow unfaithful claims to pass, rendering the measured gain sensitive to the particular tool set rather than to the verification principle itself. No independent coverage analysis or ablation on tool completeness is supplied.

Authors: We concur that the reported improvement cannot be attributed solely to the verification principle without evidence that the chosen tools achieve high recall on unsupported claims. The manuscript does not contain an independent coverage analysis or ablation on tool completeness, and we therefore cannot claim the 0.46 score is robust to arbitrary gaps in tool coverage. In the revised version we will (1) add an explicit Limitations subsection that enumerates the specific tools used on CRAFTER-XAI-Bench and the observable state variables they can verify, (2) report a simple coverage proxy (fraction of generated claims that could be checked by at least one tool), and (3) include a sensitivity ablation that randomly drops 20 % and 40 % of tool queries to illustrate how faithfulness scores degrade under reduced coverage. These additions will make the dependence on tool completeness transparent rather than implicit. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation relies on external tools and benchmark comparisons

full rationale

The paper introduces FAX as a verification framework that decomposes explanations and filters claims using inherently faithful tools (simulators, policy queries), then evaluates on the new CRAFTER-XAI-Bench against external baselines. Simulation faithfulness is measured by direct comparison to model behavior on the benchmark, with no equations, self-citations, or reductions that make the central result equivalent to its inputs by construction. The assumption that tools are faithful is an external premise, not a self-definitional loop, and results are reported relative to independent baselines.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that verification tools are faithful and on the new benchmark design; no free parameters or invented entities are described in the abstract.

axioms (1)

domain assumption Verification tools used to cross-check claims are inherently faithful representations of the target model behavior.
This premise is required for the filtering step in FAX to guarantee improved faithfulness.

pith-pipeline@v0.9.1-grok · 5777 in / 1060 out tokens · 33589 ms · 2026-06-29T13:20:38.011669+00:00 · methodology

0 comments

read the original abstract

Explainable AI (XAI) helps users interpret model behavior and identify potential faults. Agentic XAI systems use Large Language Models (LLMs) to make explanations more accessible through natural-language interaction, but they can also produce plausible yet unfaithful explanations. This risk arises because unreliable XAI outputs for complex models can be amplified by LLMs and mislead users. We propose Faithful Agentic XAI (FAX), a framework that improves explanation faithfulness through explicit verification. FAX decomposes draft explanations into claims and cross-checks them against inherently faithful tools, filtering unsupported or contradictory claims before final generation. We also introduce CRAFTER-XAI-Bench, an open-world reinforcement learning benchmark with complex policies, diverse goals, and challenging scenarios for assessing model-specific faithfulness. On CRAFTER-XAI-Bench, FAX improves simulation faithfulness from 0.20 for the strongest baseline to 0.46 while maintaining high informativeness, relevance, and fluency. On three tabular benchmarks, FAX performs competitively with prior Agentic XAI baselines, but our analysis shows that these settings can conflate task accuracy with model-specific faithfulness. These findings show that explicit verification is essential for faithful Agentic XAI and that that faithfulness benchmarks must be designed to test explanations against the behavior of the target model itself.

Figures

Figures reproduced from arXiv: 2605.27879 by Jaechang Kim, Jaewoong Cho, Jungseul Ok, Seungjoon Lee, Sunung Mun.

**Figure 1.** Figure 1: Different XAI methods provide different information. User question types are adopted from XAIQuestionBank (Liao et al., 2020). barrier for non-expert users. To lower this barrier, Agentic XAI systems use a Large Language Model (LLM) to select suitable XAI methods and translate their outputs into natural-language explanations (Slack et al., 2023; He et al., 2025). This accessibility benefit introduces a c… view at source ↗

**Figure 2.** Figure 2: (a) Agentic XAI uses XAI methods suited to a user query and generates a natural-language response. (b) FAX verifies response claims with inherently faithful XAI methods. 1. Planning 2. Tool Execution 3. Response Draft Generation 4. Verification 5. Final Response Generation FAX 1) Claim identification 2) Evidence analysis Verification Why does the model get hit by a skeleton? The model thinks mining diamond… view at source ↗

**Figure 3.** Figure 3: FAX augments an agentic XAI workflow with an explicit verification step that detects and removes unfaithful claims before final generation. that may fail sanity checks (Adebayo et al., 2018). Dashboards and conversational XAI systems lower the expertise barrier by aggregating tools or mapping user questions to XAI operations (Dijk et al., 2023; Yang et al., 2022; Arya et al., 2019; Zytek et al., 2024; Cas… view at source ↗

**Figure 4.** Figure 4: Evaluation scenarios are grouped into four categories. Each category represents a different kind [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

**Figure 5.** Figure 5: Each claim in the response draft is verified using faithful tools. Corresponding contents are [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

**Figure 6.** Figure 6: Different models produce different explanations even when the state and model action are the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

19 extracted references · 2 canonical work pages

[1]

Benchmarking the spectrum of agent capabilities.arXiv preprint arXiv:2109.06780,

oegedijk/explainerdashboard: explainer- dashboard 0.4.2: dtreeviz v2 compatiblity. Dheeru Dua and Casey Graff. 2017. UCI machine learning repository. Danijar Hafner. 2021. Benchmarking the spec- trum of agent capabilities.arXiv preprint arXiv:2109.06780. Peter Hase and Mohit Bansal. 2020. Evaluat- ing explainable ai: Which algorithmic explana- tions help ...

work page arXiv 2017
[2]

arXiv preprint arXiv:2004.03685 , year=

Is conversational xai all you need? human- ai decision making with a conversational xai as- sistant. InProceedings of the 30th International Conference on Intelligent User Interfaces, pages 907–924. 9 Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness?arXiv preprint arXiv:2004....

work page arXiv 2020
[3]

In The Eleventh International Conference on Learn- ing Representations

Post-hoc concept bottleneck models. In The Eleventh International Conference on Learn- ing Representations. Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong-Hui Chen, Jiaqi Chen, Mingchen 10 Zhuge, XinCheng, SiruiHong, JinlinWang, Bing- nan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. 2025a. AFlow: Automating agentic work- flow generation. InThe ...

2023
[4]

under edited states′, the target model chooses action a

Explingo: Explaining AI Predictions us- ing Large Language Models . In2024 IEEE International Conference on Big Data (BigData), pages 1197–1208, Los Alamitos, CA, USA. IEEE Computer Society. 11 Appendix A Extended Related Work A.1 Explainable AI Classical methodsPost-hoc XAI methods in- clude four broad families: (i)feature attribu- tion/saliencythat high...

2014
[5]

The model attacks zombies because its health is high,

Carefully read the agent's answer and identify the core claims or assumptions it makes about the model's behavior. (e.g., "The model attacks zombies because its health is high," or "The model avoids water because it has no boat."),→
[6]

For each claim, devise a "what-if" scenario that can be tested using a state edit
[7]

state_edit

Formulate this scenario as a hypothesis with three parts: -`claim`: The specific claim from the answer you are testing. -`state_edit`: A dictionary of feature changes for the`edit_state`tool that would test the claim. -`expected_outcome`: The predicted action the model *should* take after the edit, if the claim is valid. The outcome should be one of the v...
[8]

Acknowledge that the user wants to learn the agent's general strategy, not just understand a single event.,→

**Understand the User's Goal:** Read the`User Query`and`Final Response`. Acknowledge that the user wants to learn the agent's general strategy, not just understand a single event.,→
[9]

The agent moved left to get the wood

**Analyze the Explanation's Nature:** Analyze the content of the response. Does it describe a specific, one-time action (e.g., "The agent moved left to get the wood"), or does it reveal a broader, reusable principle (e.g., "The agent's policy is to prioritize collecting wood whenever it is nearby"). ,→ ,→
[10]

mental model

**Simulate Future Prediction:** Imagine you are now shown a completely new game state. Based *only* on the explanation provided, how effectively could you form a hypothesis about the agent's next action? Does the explanation give you a "mental model" to work with. ,→ ,→
[11]

It attacked the skeleton because it was there

**Assign a Score:** Based on this perceived predictive power and generalizability, assign a single integer score from 1 to 5 using the rubric below.,→ **4. Predictability Gain Rubric** * **5 (Excellent Predictive Power):** The response provides a clear, generalizable principle or rule about the agent's behavior. A user would feel very confident applying t...
[12]

Read the`User Query`to understand the user's exact intent
[13]

Read the`Generated Response`
[14]

Compare the response directly against the query to judge its relevance
[15]

1", "2",

Choose a single integer score from 1 to 5 that best represents the relevance. **4. Query Relevance Rubric** * **5:** The response directly and completely answers the user's question without any unnecessary information. * **4:** The response accurately answers the user's question but may contain minor irrelevant details. * **3:** The response addresses onl...
[16]

Read the`Generated Response`carefully
[17]

Analyze its grammatical correctness, clarity, and overall readability
[18]

Determine if the response is presented as natural language
[19]

1", "2",

Choose a single integer score from 1 to 5 that best represents its linguistic fluency based on the rubric. **4. Fluency Rubric** * **5:** The response is perfectly written. It is grammatically correct, well-structured, clear, and uses natural language.,→ * **4:** The response is well-written and easy to understand, with only very minor errors that do not ...

[1] [1]

Benchmarking the spectrum of agent capabilities.arXiv preprint arXiv:2109.06780,

oegedijk/explainerdashboard: explainer- dashboard 0.4.2: dtreeviz v2 compatiblity. Dheeru Dua and Casey Graff. 2017. UCI machine learning repository. Danijar Hafner. 2021. Benchmarking the spec- trum of agent capabilities.arXiv preprint arXiv:2109.06780. Peter Hase and Mohit Bansal. 2020. Evaluat- ing explainable ai: Which algorithmic explana- tions help ...

work page arXiv 2017

[2] [2]

arXiv preprint arXiv:2004.03685 , year=

Is conversational xai all you need? human- ai decision making with a conversational xai as- sistant. InProceedings of the 30th International Conference on Intelligent User Interfaces, pages 907–924. 9 Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness?arXiv preprint arXiv:2004....

work page arXiv 2020

[3] [3]

In The Eleventh International Conference on Learn- ing Representations

Post-hoc concept bottleneck models. In The Eleventh International Conference on Learn- ing Representations. Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong-Hui Chen, Jiaqi Chen, Mingchen 10 Zhuge, XinCheng, SiruiHong, JinlinWang, Bing- nan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. 2025a. AFlow: Automating agentic work- flow generation. InThe ...

2023

[4] [4]

under edited states′, the target model chooses action a

Explingo: Explaining AI Predictions us- ing Large Language Models . In2024 IEEE International Conference on Big Data (BigData), pages 1197–1208, Los Alamitos, CA, USA. IEEE Computer Society. 11 Appendix A Extended Related Work A.1 Explainable AI Classical methodsPost-hoc XAI methods in- clude four broad families: (i)feature attribu- tion/saliencythat high...

2014

[5] [5]

The model attacks zombies because its health is high,

Carefully read the agent's answer and identify the core claims or assumptions it makes about the model's behavior. (e.g., "The model attacks zombies because its health is high," or "The model avoids water because it has no boat."),→

[6] [6]

For each claim, devise a "what-if" scenario that can be tested using a state edit

[7] [7]

state_edit

Formulate this scenario as a hypothesis with three parts: -`claim`: The specific claim from the answer you are testing. -`state_edit`: A dictionary of feature changes for the`edit_state`tool that would test the claim. -`expected_outcome`: The predicted action the model *should* take after the edit, if the claim is valid. The outcome should be one of the v...

[8] [8]

Acknowledge that the user wants to learn the agent's general strategy, not just understand a single event.,→

**Understand the User's Goal:** Read the`User Query`and`Final Response`. Acknowledge that the user wants to learn the agent's general strategy, not just understand a single event.,→

[9] [9]

The agent moved left to get the wood

**Analyze the Explanation's Nature:** Analyze the content of the response. Does it describe a specific, one-time action (e.g., "The agent moved left to get the wood"), or does it reveal a broader, reusable principle (e.g., "The agent's policy is to prioritize collecting wood whenever it is nearby"). ,→ ,→

[10] [10]

mental model

**Simulate Future Prediction:** Imagine you are now shown a completely new game state. Based *only* on the explanation provided, how effectively could you form a hypothesis about the agent's next action? Does the explanation give you a "mental model" to work with. ,→ ,→

[11] [11]

It attacked the skeleton because it was there

**Assign a Score:** Based on this perceived predictive power and generalizability, assign a single integer score from 1 to 5 using the rubric below.,→ **4. Predictability Gain Rubric** * **5 (Excellent Predictive Power):** The response provides a clear, generalizable principle or rule about the agent's behavior. A user would feel very confident applying t...

[12] [12]

Read the`User Query`to understand the user's exact intent

[13] [13]

Read the`Generated Response`

[14] [14]

Compare the response directly against the query to judge its relevance

[15] [15]

1", "2",

Choose a single integer score from 1 to 5 that best represents the relevance. **4. Query Relevance Rubric** * **5:** The response directly and completely answers the user's question without any unnecessary information. * **4:** The response accurately answers the user's question but may contain minor irrelevant details. * **3:** The response addresses onl...

[16] [16]

Read the`Generated Response`carefully

[17] [17]

Analyze its grammatical correctness, clarity, and overall readability

[18] [18]

Determine if the response is presented as natural language

[19] [19]

1", "2",

Choose a single integer score from 1 to 5 that best represents its linguistic fluency based on the rubric. **4. Fluency Rubric** * **5:** The response is perfectly written. It is grammatically correct, well-structured, clear, and uses natural language.,→ * **4:** The response is well-written and easy to understand, with only very minor errors that do not ...