Objective Metrics for Human-Subjects Evaluation in Explainable Reinforcement Learning

Balint Gyevnar; Mark Towers

REVIEW 4 major objections 6 minor 21 references

Objective Metrics for Human-Subjects Evaluation in Explainable Reinforcement Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Explainable reinforcement learning should be judged by whether humans can act on an explanation, not just by what they say about it.

desk verdict A useful taxonomy of objective XRL metrics, but the actionability link is asserted rather than shown; still deserves a serious referee. read the letter →

arxiv 2501.19256 v1 pith:QV6U367N submitted 2025-01-31 cs.AI cs.HCcs.RO

Balint Gyevnar , Mark Towers This is my paper · ORCID

classification cs.AIcs.HCcs.RO

keywords ExplainablereinforcementlearningHuman-subjectsevaluationObjectivemetricsSubjectiveActionabilityHuman-agentteamingDebuggingUserstudy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that self-reported subjective metrics—confidence, comprehension, preference—cannot tell researchers whether an explanation actually helps a human act on an agent's behaviour. To fix this, it proposes evaluating explanations through objective, observable human behaviour, defining 'actionability' as the capacity of an explanation to change or guide what a person does. It curates two families of metrics: debugging metrics (next-action, goal, and sub-goal prediction; counterfactual state selection; time taken) and teaming metrics (task completion, inter-agent conflict, time taken), illustrating each in a grid-based mini-world. If adopted, the field would move from opinion polls to measurable behavioural outcomes, making studies more reproducible and comparable. The paper frames this as a first step, not a settled protocol.

What carries the argument

The central object is the concept of actionability, defined as the ability of an explanation to affect or change human behaviour, operationalized through a catalogue of objective behavioural metrics. In debugging, the metrics are next-action prediction (can the user select the agent's action $\pi(s)$ from the action set?), goal prediction, sub-goal prediction, counterfactual policy selection (which state change $s'$ makes $\pi(s) \neq \pi(s')$ or $\pi(s')=A$?), and time taken; in teaming, task completion, inter-agent conflict counts, and time taken. The mini-world grid environment serves as a shared illustration for how each metric would be administered, and the metrics are structured around two application goals—debugging agent behaviour before or after deployment and supporting human-agent teaming—so that the choice of metric is tied to the goal and audience of the explanation.

What would settle it

Run a between-subjects study in mini-world with two explanation mechanisms, one optimised for next-action prediction and one for task completion. If participants using the better-prediction mechanism still fail to spot a deliberately planted agent fault, or complete debugging tasks no better than a no-explanation control, then next-action prediction does not measure actionability.

Watch

Extended reading notes

Core claim

The central claim is that an explanation's quality in XRL should be assessed by its actionability—whether it changes or enables human behaviour—measured through objective tasks rather than subjective self-report. For debugging, users should be able to predict an agent's next action, overall goal, or intermediate sub-goal, or to identify counterfactual state changes that would alter the agent's decision; for teaming, the measures are task completion, the number of inter-agent conflicts, and time taken. These metrics are presented as complements, not replacements, to subjective measures: subjective reports can track preferences and confidence, while objective metrics track practical effectiveness. The paper also recommends a 'No Explanation' control condition and argues that the choice of environment and experimental controls, informed by cognitive psychology, determines whether the metrics measure what they claim.

Load-bearing premise

The crucial assumption is that scoring well on tasks like predicting the agent's next action or completing a team task really does mean the explanation helps people in practice, even though no user study yet verifies that link.

Editorial extensions

If this is right

Researchers evaluating XRL explanations would replace or supplement Likert-style self-reports with behavioural tasks such as next-action, goal, and sub-goal prediction, making results more directly comparable across studies.
Debugging and teaming require different metrics; a metric that validates an explanation for fault investigation will not necessarily validate it for human-agent coordination, so papers must state which goal they target.
Every human-subject study should include a 'No Explanation' baseline, preventing results from being attributed to explanations when users could have succeeded from task description alone.
Subjective and objective metrics are complementary: subjective measures capture preference and confidence, while objective measures capture actionability, and their correlation can reveal when users' beliefs about an explanation diverge from its actual effect.
Because metric effectiveness depends on experimental setup, the field needs standardised benchmarks and shared tasks, ideally in more realistic environments than toy gridworlds, to make cross-paper comparison meaningful.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's explicit claims, the metric list suggests a cost-fidelity ordering: next-action prediction is cheap but narrow, task completion is behaviourally rich but confounded, so practical protocols would pair a low-cost prediction metric with an outcome metric.
The 'No Explanation' baseline, if standardised, would allow effect-size meta-analysis across XRL studies, a step the paper calls for but does not formalise.
A testable extension is a mini-world comparison where objective and subjective rankings of explanation mechanisms diverge; such a divergence would directly demonstrate the paper's central distinction between actionability and opinion.
The debugging/teaming split leaves other explanation goals, such as trust calibration or regulatory accountability, without operationalised metrics; extending the catalogue to those goals would require new behavioural tasks.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a position paper, not an empirical study, and its value is the curation of objective metrics plus the debugging/teaming taxonomy. The central recommendation—prefer objective behavioral metrics over subjective ratings—is sensible, but the link from these metrics to 'actionability' is assumed rather than demonstrated.

The paper does several things well. It defines actionability clearly, separates debugging from teaming, and gives concrete examples of six metrics with pros and cons. The tables are useful reference material. The authors are honest about limitations: they note that task completion can be achieved without an explanation, recommend a no-explanation baseline, and point to cognitive psychology for experimental design. That's more self-awareness than most position papers show.

The soft spot is real, and the stress-test note captures it. Next-action, goal, sub-goal, and counterfactual prediction ask participants to reproduce the agent's choices. Those are fidelity measures—they test whether a user can mimic the policy, not whether they can spot a bug or correct the agent's behavior. Task completion and inter-agent conflict are closer to actionability, but without a baseline they are confounded; the paper acknowledges this. The missing piece is criterion validity: no user study or benchmark shows that better scores on these metrics correspond to better debugging or teaming outcomes. The claim that objective metrics are 'more epistemically grounded' is asserted, not shown. That matters because the whole point of the paper is to change evaluation practice.

That said, for a position paper, the absence of a user study isn't fatal. The contribution is conceptual, and the authors don't oversell the metrics as validated—they explicitly call the paper a first step. The mini-world is illustrative, not a released benchmark; that's fine for this purpose.

I'd like to see the final version engage explicitly with the mimicry concern and propose one experiment that could discriminate between policy mimicry and actionability. But as is, this is a solid, readable position piece that deserves a serious referee. The target reader is anyone designing XRL user studies; it would be a good starting point for a methods section.

Referee Report

4 major / 6 minor

Summary. The paper is a position/vision paper arguing that evaluation of explainable reinforcement learning (XRL) should include objective human-subject metrics based on observable behavior, and that subjective self-reports alone are insufficient. It defines an explanation's actionability as its ability to affect or change human behavior, and it curates a set of objective metrics for two application contexts: debugging (next action prediction, goal prediction, sub-goal prediction, counterfactual policy, time taken) and human-agent teaming (task completion, inter-agent conflict, time taken). The paper illustrates each metric through a small grid-world environment ('mini-world') and discusses how subjective and objective metrics can complement each other, while also recommending baselines and standardized benchmarks for future work.

Significance. If the central claim is accepted, the paper could help shift XRL evaluation practice from self-reported opinions toward measurable behavioral outcomes, improving reproducibility and comparability across studies. Its strengths are its clear definitions, its concrete tabular comparison of metric pros and cons, its worked mini-world examples, and its transparent acknowledgment of limitations, including the need for 'No Explanation' baselines and the variability of metric effectiveness. However, the central claim that the proposed metrics are valid measures of actionability and that they make XRL research 'epistemically grounded' is not demonstrated: no user study, benchmark, or formal derivation supports the construct-validity link. The paper is best read as a proposal rather than an established result, and the gap between the actionability definition and several of the proposed metrics is a load-bearing concern that needs to be addressed.

major comments (4)

[Section 2.2] The debugging metrics Next Action Prediction, Goal Prediction, Sub-Goal Prediction, and Counterfactual Policy all ask participants to predict or reproduce the agent's policy choices. These measure the participant's fidelity of mental simulation of the policy, not whether an explanation changes behavior in a way that improves debugging or teaming outcomes. This sits uneasily with the Section 1 definition of actionability as 'the ability of explanations to affect or change human behaviour': a user who can predict the agent's next action may still fail to identify a policy bug or to take corrective action. Either the definition of actionability must be broadened to include policy comprehension, or the paper must provide an argument or evidence connecting prediction accuracy to downstream task performance.
[Section 2.3 and Table 2] Task Completion is the metric closest to behavioral effectiveness, but the paper itself notes in Table 2 that it 'can be achieved without an explanation.' The Section 3 recommendation of a 'No Explanation' baseline is therefore essential, but the paper does not explain how the metric should be analyzed relative to that baseline, nor does it discuss ceiling or floor effects. A user study that compares task completion with and without explanations, and that reports effect sizes and confidence intervals, is needed to support the claim that this metric captures explanation actionability; as written, the methodology is underspecified.
[Section 3] The abstract and Section 1 assert that objective human metrics enable 'reproducible, comparable, and epistemically grounded research' and 'better reproducibility, incremental work, and more rigorous practices.' Yet Section 3 admits that 'in practice, our proposed metrics may be instantiated in a wide range of ways and their effectiveness in capturing explanation success will vary.' This admission undercuts the strength of the central claim. Since no user study, benchmark, or formal analysis is presented, the claim is a plausible position rather than a demonstrated result; the paper should either qualify the claim accordingly or include a concrete validation agenda specifying the controlled comparisons needed to establish each metric's validity.
[Section 2.2, Sub-Goal Prediction] The paper states that 'we are unaware of an objective method for determining an agent's "true" sub-goal' and footnote 2 notes that the effectiveness of existing approaches 'is unknown.' This means sub-goal labels must be handpicked and annotated by researchers, which introduces experimenter judgment into the supposedly objective measurement. The paper should clarify how the metric remains objective when the ground-truth sub-goal is subjectively assigned, or explicitly classify this metric as relying on a researcher-defined ground truth with its own validation burden.

minor comments (6)

[Section 2.1] Typo: '(i.e., the soldier in the red circle).' should not have a stray comma after the closing parenthesis; the sentence continues with 'but for Section 2.3'.
[Figure 1 caption] The phrase 'passabletiles' is missing a space between 'passable' and 'tiles'.
[Table 2 and Section 2.3] The metric is called 'Task Competition' in Table 2 but 'Task Completion' in Section 2.3; the terminology should be made consistent.
[Section 2.2, Counterfactual Policy] The sentence 'users can mentally simulate an state to check' contains a grammar error; it should read 'a state'.
[Section 1] The reference format '[17, Table 1.]' includes an unnecessary period inside the bracket; it should be '[17, Table 1]'.
[Section 1] The term 'agentic systems' is used without definition; since the paper addresses a general AI audience, it would be helpful to define it or use a more standard term such as 'autonomous agents.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper proposes, rather than derives, objective behavioral metrics, and its self-citations are illustrative and non-load-bearing.

full rationale

The paper is not circular in any of the seven tracked senses. It is a position and curation paper: Section 1 defines actionability as "the ability of explanations to affect or change human behaviour" and Section 2 then proposes a battery of objective metrics (next-action prediction, goal prediction, sub-goal prediction, counterfactual policy, task completion, inter-agent conflict, time taken) as candidate operationalizations. The metrics are not derived from actionability by equation, nor is actionability defined as successful performance on the metrics; the connection is an empirical-validity claim rather than a definitional identity. No parameters are fitted and no quantity is "predicted" from fitted inputs, so the fitted-input pattern does not apply. The self-citations ([7], [8], and [9]) are used as background evidence or as one possible instantiation of the counterfactual-policy evaluation, and they do not carry the central recommendation that subjective metrics alone are insufficient. No uniqueness theorem from the authors is invoked. The paper also explicitly flags its own limitations: Section 3 states that "in practice, our proposed metrics may be instantiated in a wide range of ways and their effectiveness in capturing explanation success will vary on the quality of the experimental setup"; Table 2 concedes that task completion "can be achieved without an explanation"; and Section 2.2 admits "we are unaware of an objective method for determining an agent's 'true' sub-goal." These are construct-validity and measurement-design caveats, not circular reasoning. The skeptical concern that prediction metrics may measure policy mimicry rather than actionability is a substantive validity objection, but the paper does not assert an equivalence; it proposes the metrics as plausible measures to be tested. The derivation chain is therefore self-contained, and the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

No fitted parameters, equations, or derivations. The argument rests on several normative assumptions, listed above. Mini-world is an illustrative environment, not a released benchmark.

assumptions (4)

domain assumption Actionability, defined as the ability of explanations to affect human behaviour, is the right central construct to measure.
Section 1 defines actionability and asserts its importance; this normative premise is never empirically grounded.
domain assumption Debugging and human-agent teaming are the two primary applications of XRL explanations, so metrics for these two contexts are sufficient.
Section 2 opens by declaring these the two primary situations; the paper does not justify exhaustiveness.
domain assumption Objective behavioural metrics are more reproducible, comparable, and epistemically grounded than subjective metrics.
Asserted in the abstract and Section 1; no empirical study or formal argument demonstrates this advantage.
domain assumption Users' prediction accuracy, task completion, conflict counts, and response times are valid indicators of explanation effectiveness.
Each metric in Section 2 assumes the observed behavior reflects explanation quality; validation is left to future work.

invented entities (1)

mini-world grid-based environment
purpose: Provides a concrete illustrative testbed for the proposed objective metrics.
Described in Figure 1 but no code or data release is provided; it is a scenario for examples, not an empirical instrument.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Objective Metrics for Human-Subjects Evaluation in Explainable Reinforcement Learning." pith.science (2026). https://pith.science/paper/QV6U367N

@misc{pith2026250119256,
  author       = {Pith},
  title        = {Pith review of: Objective Metrics for Human-Subjects Evaluation in Explainable Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QV6U367N}},
  note         = {Machine review of arXiv:2501.19256}
}

read the original abstract

Explanation is a fundamentally human process. Understanding the goal and audience of the explanation is vital, yet existing work on explainable reinforcement learning (XRL) routinely does not consult humans in their evaluations. Even when they do, they routinely resort to subjective metrics, such as confidence or understanding, that can only inform researchers of users' opinions, not their practical effectiveness for a given problem. This paper calls on researchers to use objective human metrics for explanation evaluations based on observable and actionable behaviour to build more reproducible, comparable, and epistemically grounded research. To this end, we curate, describe, and compare several objective evaluation methodologies for applying explanations to debugging agent behaviour and supporting human-agent teaming, illustrating our proposed methods using a novel grid-based environment. We discuss how subjective and objective metrics complement each other to provide holistic validation and how future work needs to utilise standardised benchmarks for testing to enable greater comparisons between research.

Figures

Figures reproduced from arXiv: 2501.19256 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 19 canonical work pages

[1]

H. A. Abbass. Social Integration of Artificial Intelligence: Functions, Automation Allocation Logic and Human- Autonomy Trust. Cognitive Computation, 11(2):159–171, Apr. 2019

work page 2019
[2]

M. D. Alicke, D. Rose, and D. Bloom. Causation, Norm Violation, and Culpable Control. Journal of Philosophy , 108(12):670–696, 2011

work page 2011
[3]

R. M. Byrne. Good Explanations in Explainable Artificial Intelligence (XAI): Evidence from Human Explanatory Reasoning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 6536–6544, Macau, SAR China, Aug. 2023. International Joint Conferences on Artificial Intelligence Organization

work page 2023
[4]

H. H. Clark and K. Fischer. Social robots as depictions of social agents. Behavioral and Brain Sciences, 46:e21, Jan. 2023

work page 2023
[5]

Ehsan, K

U. Ehsan, K. Saha, M. De Choudhury, and M. O. Riedl. Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI. Proc. ACM Hum.-Comput. Interact., 7(CSCW1):34:1–34:32, Apr. 2023

work page 2023
[6]

J. I. Fuentes. Computational systems as higher-order mechanisms. Synthese, 203(2):1–26, 2024

work page 2024
[7]

Gyevnar, S

B. Gyevnar, S. Droop, T. Quillien, S. B. Cohen, N. R. Bramley, C. G. Lucas, and S. V . Albrecht. People attribute purpose to autonomous vehicles when explaining their behavior: Insights from cognitive science for explainable AI. In CHI ’25: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, New York, NY, United States, 2025. Ass...

work page 2025
[8]

Gyevnar, N

B. Gyevnar, N. Ferguson, and B. Schafer. Bridging the transparency gap: What can explainable AI learn from the AI act? In 26th European Conference on Artificial Intelligence, pages 964–971. IOS Press, 2023

work page 2023

Show all 21 references

[9]

Gyevnar, C

B. Gyevnar, C. Wang, C. G. Lucas, S. B. Cohen, and S. V . Albrecht. Causal explanations for sequential decision-making in multi-agent systems. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’24, pages 771–779, Richland, S...

2024
[10]

Hitchcock

C. Hitchcock. Portable Causal Dependence: A Tale of Consilience. Philosophy of Science, 79(5):942–951, 2012

2012
[11]

R. R. Hoffman, S. T. Mueller, G. Klein, and J. Litman. Metrics for Explainable AI: Challenges and Prospects. arXiv:1812.04608 [cs], Feb. 2019

2019 arXiv
[12]

Huber, M

T. Huber, M. Demmler, S. Mertes, M. L. Olson, and E. Andr´e. Ganterfactual-rl: Understanding reinforcement learning agents’ strategies through visual counterfactual explanations. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, A...

2023
[13]

Karimi, G

A.-H. Karimi, G. Barthe, B. Sch¨olkopf, and I. Valera. A Survey of Algorithmic Recourse: Contrastive Explanations and Consequential Recommendations. ACM Comput. Surv., 55(5):95:1–95:29, Dec. 2022

2022
[14]

Kirfel, T

L. Kirfel, T. Icard, and T. Gerstenberg. Inference from explanation. Journal of Experimental Psychology: General , 151(7):1481–1501, 2022

2022
[15]

Lombrozo

T. Lombrozo. Causal–explanatory pluralism: How intentions, functions, and mechanisms influence causal ascriptions. Cognitive Psychology, 61(4):303–332, Dec. 2010

2010
[16]

Madumal, T

P . Madumal, T. Miller, L. Sonenberg, and F. Vetere. Explainable reinforcement learning through a causal lens. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2493–2500, Apr. 2020

2020
[17]

Milani, N

S. Milani, N. Topin, M. Veloso, and F. Fang. Explainable Reinforcement Learning: A Survey and Comparative Review. ACM Computing Surveys, 56(7):168:1–168:36, Apr. 2024

2024
[18]

T. Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267:1–38, Feb. 2019

2019
[19]

Mohseni, N

S. Mohseni, N. Zarei, and E. D. Ragan. A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems. ACM Transactions on Interactive Intelligent Systems, 11(3-4):24:1–24:45, Aug. 2021

2021
[20]

Perez-Osorio and A

J. Perez-Osorio and A. Wykowska. Adopting the intentional stance toward natural and artificial agents. Philosophical Psychology, 33(3):369–395, Apr. 2020

2020
[21]

N. Puri, S. Verma, P . Gupta, D. Kayastha, S. Deshmukh, B. Krishnamurthy, and S. Singh. Explain your move: Understanding agent actions using specific and relevant feature attribution. In 8th International Conference on Learning Representations, ICLR 2020, Apr. 2020. 6

2020

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

H. A. Abbass. Social Integration of Artificial Intelligence: Functions, Automation Allocation Logic and Human- Autonomy Trust. Cognitive Computation, 11(2):159–171, Apr. 2019

work page 2019

[2] [2]

M. D. Alicke, D. Rose, and D. Bloom. Causation, Norm Violation, and Culpable Control. Journal of Philosophy , 108(12):670–696, 2011

work page 2011

[3] [3]

R. M. Byrne. Good Explanations in Explainable Artificial Intelligence (XAI): Evidence from Human Explanatory Reasoning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 6536–6544, Macau, SAR China, Aug. 2023. International Joint Conferences on Artificial Intelligence Organization

work page 2023

[4] [4]

H. H. Clark and K. Fischer. Social robots as depictions of social agents. Behavioral and Brain Sciences, 46:e21, Jan. 2023

work page 2023

[5] [5]

Ehsan, K

U. Ehsan, K. Saha, M. De Choudhury, and M. O. Riedl. Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI. Proc. ACM Hum.-Comput. Interact., 7(CSCW1):34:1–34:32, Apr. 2023

work page 2023

[6] [6]

J. I. Fuentes. Computational systems as higher-order mechanisms. Synthese, 203(2):1–26, 2024

work page 2024

[7] [7]

Gyevnar, S

B. Gyevnar, S. Droop, T. Quillien, S. B. Cohen, N. R. Bramley, C. G. Lucas, and S. V . Albrecht. People attribute purpose to autonomous vehicles when explaining their behavior: Insights from cognitive science for explainable AI. In CHI ’25: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, New York, NY, United States, 2025. Ass...

work page 2025

[8] [8]

Gyevnar, N

B. Gyevnar, N. Ferguson, and B. Schafer. Bridging the transparency gap: What can explainable AI learn from the AI act? In 26th European Conference on Artificial Intelligence, pages 964–971. IOS Press, 2023

work page 2023