Counterfactual Explanations for Continuous Action Reinforcement Learning

Lu Feng; Shangtong Zhang; Shuyang Dong

REVIEW 3 major objections 3 minor 1 cited by

Counterfactual Explanations for Continuous Action Reinforcement Learning

T0 review · 3 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A TD3-based method turns counterfactual 'what-if' questions for continuous-action RL into a tractable optimization over action sequences.

desk verdict A straightforward TD3-with-a-distance-penalty method for generating counterfactual action sequences; the idea is real but the central relaxation from a constrained problem to an unconstrained expectation is asserted, not proven, and the paper's own metrics show the method often fails to find any improving trajectory. read the letter →

arxiv 2505.12701 v1 pith:AORIYVFZ submitted 2025-05-19 cs.LG cs.AI

Shuyang Dong , Shangtong Zhang , Lu Feng This is my paper · ORCID

classification cs.LGcs.AI

keywords counterfactualexplanationsreinforcementlearningcontinuousactionspacesTD3trajectoryoptimizationinterpretabilitydiabetescontrolrewardshaping

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to make reinforcement learning policies in continuous action spaces explainable by computing counterfactual trajectories: alternative action sequences that start from the same state, achieve a higher cumulative reward than the observed one, and deviate as little as possible from the observed actions. If it works, a clinician could ask what insulin doses would have produced better glucose control and receive a concrete alternative treatment plan rather than a saliency map. The authors formulate this as a constrained optimization problem, relax the hard reward-improvement constraint into a soft penalty, and solve the relaxed problem by extending the TD3 actor-critic algorithm with a sparse terminal reward equal to the negative action distance. They report experiments in a diabetes simulator and in Lunar Lander in which the method generates counterfactuals for a large share of test trajectories and generalizes across environments.

What carries the argument

The load-bearing object is the soft-penalized objective of Eq. (5), built from the relative distance metric $D$ in Eq. (1), which sums $|a_{t+i} - a'_{t+i}| / (|a_{t+i}| + \delta)$ over time steps and generalizes to $\ell^p$ norms for multi-dimensional actions. The paper replaces the essential supremum in Eq. (4) with an expectation, citing concentration inequalities as justification, and then solves the expectation via TD3, adding $-\lambda D$ as a terminal reward so that the actor learns a deterministic counterfactual policy with minimal deviation. For Problem 2, an augmented MDP removes constrained states and applies the predefined policy $\pi_c$ inside them, so the constrained variant reduces to Problem 1.

What would settle it

Take any test trajectory and run Algorithm 1 with $\lambda$ large enough that the learned counterfactual barely changes the actions; if the resulting trajectory does not improve $G$ for a substantial fraction of trajectories while the training objective keeps increasing, the expectation step and soft relaxation do not preserve the constraint. The paper's own diabetes single-environment $\rho^+$ of 0.53 already indicates that no positive counterfactual is found for about half of the test trajectories.

Watch

Extended reading notes

Core claim

The central claim is that counterfactual explanations for continuous-action RL can be reduced to a trajectory-level optimization: minimize a relative action-sequence distance $D(\alpha(\tau), \alpha(\tau'))$ subject to $G(\tau') > G(\tau)$, and that this constrained problem can be solved pragmatically by optimizing the expectation of $G(\tau'(\mu)) - \lambda D(\alpha(\tau), \alpha(\tau'(\mu)))$ over deterministic policies $\mu$, with the distance injected as a sparse terminal reward. The paper further claims that constrained states, such as glucose levels below 100 mg/dL where a doctor's policy applies, can be folded into an augmented MDP so that the same algorithm handles the constrained variant unchanged. The evaluation claims that all three variants outperform a rollout baseline on positive counterfactual percentage in both domains, and that the unconstrained variant P1 achieves the best effectiveness and efficiency.

Load-bearing premise

The method assumes that maximizing the expected value of the soft-penalized reward-minus-distance objective will, with high probability, produce trajectories that actually satisfy the original hard constraint $G(\tau') > G(\tau)$ and are minimal in distance, even though the hard constraint has been replaced by a soft penalty and the per-trajectory guarantee by an average.

Editorial extensions

If this is right

In diabetes control, a trained counterfactual policy can produce alternative insulin dose sequences that raise cumulative reward while staying close to the original treatment, giving clinicians concrete 'what if' explanations.
Because the output is a deterministic policy rather than single-trajectory edits, the same learned policy generates counterfactuals for many observed trajectories at once.
User-specified constraints, such as a doctor's dosing rule in certain glucose ranges, can be enforced by rebuilding the MDP, so explanations respect clinical or safety rules.
The approach is method-agnostic: trajectories from any RL algorithm can be fed in, so it can explain existing trained policies without retraining them.
Training in multiple environments improves the advantage metric, suggesting that counterfactual generators benefit from diverse training conditions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's own $\rho^+$ values (0.53 in single-environment diabetes) imply that for nearly half of the test trajectories no improving counterfactual was found, which suggests the soft-penalty relaxation does not guarantee the hard constraint; a natural extension is a Lagrangian or constrained-policy formulation that explicitly enforces $G(\tau') > G(\tau)$.
The concentration-inequality step from Eq. (4) to Eq. (5) is asserted rather than derived; in environments with heavy-tailed returns, expectation-maximizing policies may not produce trajectories with a high essential supremum, so the guarantee could fail precisely where counterfactuals are most needed.
The distance metric is scale-dependent on action magnitudes; applying the method across heterogeneous action scales, such as different insulin sensitivities, may require normalizing per-trajectory distances, which is a testable modification.
One could validate counterfactual plausibility by feeding the generated action sequences back into the environment model to check whether they are realizable and actually improve outcomes, rather than relying only on the learned reward estimate.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper fills a genuine gap — counterfactual explanations for continuous-action RL are underexplored, and framing the task as trajectory optimization with a minimal-deviation objective is the right instinct. The authors are honest about the gap and the code is public, which helps. But the central derivation does not hold up on close reading, and the experimental results are weaker than the text claims.

What's new and what works: unlike Tsirtsis et al. and Amitai et al., which are discrete-action, this targets continuous action sequences directly, with a reasonable relative-distance metric (Eq. 1) and a sensible extension to constrained states via an augmented MDP (Section 4.2). The P2 formulation, where certain states must follow a fixed or baseline policy, is a useful practical twist. The code appears to be available, which is real credit.

The soft spot is load-bearing. Problem 1 asks for a trajectory satisfying the hard constraint G(τ') > G(τ) with minimal distance. Eq. (2) replaces that with a soft penalty, Eq. (4) moves to an essential supremum over policy rollouts, and Eq. (5) swaps the essential supremum for an expectation. The justification for that swap is one sentence citing concentration inequalities, with no bound and no argument that the optimizer of the expectation will, with high probability, produce trajectories that actually beat the observed trajectory. That is not a cosmetic gap; it changes the problem. A policy maximizing E[G − λD] can easily have low probability of exceeding the specific threshold G(τ), and the paper's own ρ+ numbers confirm this: for the diabetes P1 variant, 0.53 single-env and 0.44 multi-env mean that for roughly half the test trajectories, none of the 10 rollouts satisfies the defining constraint. A method that frequently outputs non-improving sequences does not solve Problem 1 as stated.

I also found the baseline too weak. Comparing against a single rollout of the PPO policy in evaluation mode is not a meaningful baseline for a method that can sample many candidates; a trivial noise-added rollout baseline would be fairer. The ρadv metric is clever but only evaluated on trajectories where both methods find a positive counterfactual, and P2-base showing ρadv = 0.0 in the single-env diabetes setting suggests the metric is not telling a clean story.

On the other hand, the paper is not circular and not incoherent. The training objective is clear, the evaluation separately checks the hard condition, and the authors do not misreport their own metrics — they just overclaim what the relaxation achieves. The derivation gap is central but fixable: post-filtering candidates on G(τ') > G(τ), using a proper constrained optimization approach, or explicitly reporting the probability of improvement would substantially strengthen the work.

This paper deserves a serious referee: the problem setting is relevant and the empirical setup, while flawed, is real. I would send it to review, with the expectation that the authors need to address the relaxation gap and improve the baseline before acceptance. For my own reading group, it is a maybe — the gap discussion is instructive even if the method is not yet convincing.

Referee Report

3 major / 3 minor

Summary. The paper proposes a method for generating counterfactual explanations in continuous-action reinforcement learning. The authors formulate Problems 1 and 2 as constrained optimization problems seeking an alternative action sequence that increases cumulative reward while minimizing a relative action distance, possibly subject to state-dependent policy constraints. They relax the hard reward constraint into a soft penalty (Eq. 2), approximate the trajectory-level optimization by a policy optimization over an essential supremum (Eq. 4), and then replace the essential supremum with an expectation (Eq. 5). The resulting objective is optimized with a modified TD3 algorithm (Algorithm 1). The approach is evaluated on diabetes control (UVA/Padova simulator) and Lunar Lander, reporting positive counterfactual percentage (ρ+) and advantage counterfactual percentage (ρadv).

Significance. If the method worked as claimed, it would fill a real gap: counterfactual explanations for continuous-action RL are underexplored, and the paper provides a concrete algorithm with released code and experiments in two domains. However, the significance is undermined by a major derivation gap in the relaxation from the constrained problem to the learned objective, and by experimental results that show the method frequently fails to produce any positive counterfactual. The central claim of 'efficiently generating counterfactual trajectories that improve outcomes while minimizing deviations' is therefore not supported by the evidence presented.

major comments (3)

[Section 4.1, Eqs. (2)-(5)] The replacement of the hard constraint G(τ')>G(τ) in Problem 1 with the soft penalty in Eq. (2), and the subsequent substitution of the essential supremum in Eq. (4) by the expectation in Eq. (5), is not a valid relaxation. The authors appeal to concentration inequalities and large deviation theory but provide no bound, theorem, or argument that a policy maximizing E[G(τ'(μ))-λD] will produce trajectories satisfying G(τ')>G(τ) with any guaranteed probability. This is load-bearing because Algorithm 1 optimizes the expected objective, while the evaluation tests the hard constraint.
[Section 5.1, Table 1] The reported ρ+ values for the diabetes domain, such as P1: 0.53±0.01 in the single-environment setting and 0.44±0.0 in the multi-environment setting, are only marginally above the baseline (0.44±0.0 and 0.39±0.0) and are far from 1.0. This means that for roughly half of the test trajectories, none of the 10 generated rollouts satisfies G(τ')>G(τ), so the method does not reliably solve Problem 1 under the paper's own definition of a positive counterfactual. The conclusion that the approach 'efficiently generates counterfactual trajectories' is not supported by these results.
[Section 5, Metrics] The evaluation never directly reports the action distances D(α(τ), α(τ')) of the generated counterfactual trajectories. The ρadv metric uses a ratio of distances to define advantage, but no absolute distances are presented, so the 'minimizing deviations' component of the central claim in Sections 1 and 6 is not empirically verified. The paper provides no evidence that the generated counterfactuals are proximal to the original actions.

minor comments (3)

[Section 4.1, Eq. (2)] Equation (2) uses λ⁻¹ as the penalty weight, while Eq. (3) and Eq. (5) use λ; the relationship between the two parameters is not clearly explained, and the text should clarify that Eq. (2) and Eq. (3) are equivalent up to scaling.
[Section 4.1, Eq. (4)] The notation in Eq. (4) is imprecise: the essential supremum should be written with respect to the random variable τ'(μ) (e.g., ess sup over realizations of τ'(μ)), and the text in the following paragraph should make the domain of the essential supremum explicit.
[Section 5, Metrics and Baselines] The baseline method, which rolls out the baseline policy without additional training, is a weak comparator for counterfactual generation, and the paper would benefit from a discussion of why this is an appropriate reference point given that no existing method targets continuous-action counterfactual RL.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the counterfactual policy is trained on an explicit soft-penalty objective and then independently evaluated against the hard constraint G(τ')>G(τ).

full rationale

The paper's derivation chain is not circular. Problem 1 defines a constrained optimization (minimize action distance subject to G(τ')>G(τ)), and Section 4.1 explicitly relaxes the hard constraint into a soft penalty in Eq. (2), equivalently maximizing G(τ')−λD in Eq. (3). The subsequent move from the essential supremum in Eq. (4) to the expectation in Eq. (5) is an approximation justified informally by concentration inequalities, not a definitional equivalence. Algorithm 1 trains a TD3 policy to optimize that expected objective, and the evaluation metrics are independent: ρ+ is measured by rolling out the trained policy and checking whether any rollout actually satisfies G(τ')>G(τ), and ρadv compares reward gains against action distance. No parameter is fitted to the evaluation data and then reported as a prediction; λ is a user-set hyperparameter. The paper also does not rely on a load-bearing self-citation: its references to TD3, PPO, and the UVA/PADOVA simulator are standard external sources. The main weakness—that the soft-penalty relaxation and expectation swap do not guarantee satisfaction of the hard constraint, as evidenced by ρ+ values well below 1—is a correctness and soundness concern, not a circularity. The central claim is therefore supported by independent evaluation of an explicitly stated optimization objective rather than by construction or self-reference.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central method rests on two ad hoc assumptions: the soft-penalty relaxation preserving the hard constraint, and the expectation-to-essential-supremum approximation. Both are stated without proof. The two free parameters λ and δ are not tuned or justified. No new entities are introduced.

free parameters (2)

λ (distance reward weight) = 1 (both domains)
Controls the tradeoff between reward gain and action deviation in the soft-penalty objective (Eq. 2 and Eq. 5). Set to 1 in all experiments with no sensitivity analysis, so the behavior of the method outside this single value is unknown.
δ (distance metric stability constant) = unspecified (only stated δ > 0)
Appears in the denominator of Eq. (1) to prevent division by zero when a_t = 0. Its value changes the relative penalty for deviating from zero actions, yet no numerical value or sensitivity study is reported.

assumptions (3)

ad hoc to paper The soft-penalty relaxation in Eq. (2) with a finite λ produces the same solution set as the hard constraint G(τ') > G(τ) in Problem 1.
This is a core unproven assumption: the paper replaces the hard constraint with a weighted penalty but gives no argument that the minimizer of the penalized objective satisfies the original constraint. The empirical ρ+ results show this is often violated.
ad hoc to paper Maximizing the expectation E[G(τ'(μ)) - λD(α(τ), α(τ'(μ)))] is a faithful proxy for maximizing the essential supremum of the trajectory objective in Eq. (4).
Stated in Section 4.1 after Eq. (4) with a reference to concentration inequalities and large deviation theory, but no explicit bound or proof is given. This step is necessary for the reduction to a standard RL problem.
domain assumption The transition function of the environment is available for rolling out the counterfactual policy during training and evaluation.
Algorithm 1 requires sampling transitions (s', r) for counterfactual trajectories. This is true in the simulators used, but in real high-stakes domains such as patient care such rollouts are not possible, limiting the applicability of the method.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Counterfactual Explanations for Continuous Action Reinforcement Learning." pith.science (2026). https://pith.science/paper/AORIYVFZ

@misc{pith2026250512701,
  author       = {Pith},
  title        = {Pith review of: Counterfactual Explanations for Continuous Action Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AORIYVFZ}},
  note         = {Machine review of arXiv:2505.12701}
}

read the original abstract

Reinforcement Learning (RL) has shown great promise in domains like healthcare and robotics but often struggles with adoption due to its lack of interpretability. Counterfactual explanations, which address "what if" scenarios, provide a promising avenue for understanding RL decisions but remain underexplored for continuous action spaces. We propose a novel approach for generating counterfactual explanations in continuous action RL by computing alternative action sequences that improve outcomes while minimizing deviations from the original sequence. Our approach leverages a distance metric for continuous actions and accounts for constraints such as adhering to predefined policies in specific states. Evaluations in two RL domains, Diabetes Control and Lunar Lander, demonstrate the effectiveness, efficiency, and generalization of our approach, enabling more interpretable and trustworthy RL applications.

Figures

Figures reproduced from arXiv: 2505.12701 by the authors.

**Figure 2.** Learning curves of Positive Counterfactual Percentage [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 5.** Learning curves of Advantage Counterfactual Percentage [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Interpret Policies in Deep Reinforcement Learning using SILVER with RL-Guided Labeling: A Model-level Approach to High-dimensional and Multi-action Environments
cs.LG 2025-10 reject novelty 4.0 of 10

SILVER with RL-guided labeling: SHAP plus clustering plus policy-query labels plus decision trees or regression to interpret multi-action Atari policies.

Reference graph

Works this paper leans on

20 extracted references · 16 canonical work pages · cited by 1 Pith paper

[1]

Explaining reinforcement learning agents through counterfactual action outcomes

[Amitaiet al., 2024 ] Yotam Amitai, Yael Septon, and Ofra Amir. Explaining reinforcement learning agents through counterfactual action outcomes. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10003–10011,

work page 2024
[5]

Explaining reinforcement learning policies through counterfactual trajectories.ICML 2021 Workshop on Human in the Loop Learning,

[Frostet al., 2021 ] Julius Frost, Olivia Watkins, Eric Weiner, Pieter Abbeel, Trevor Darrell, Bryan Plummer, and Kate Saenko. Explaining reinforcement learning policies through counterfactual trajectories.ICML 2021 Workshop on Human in the Loop Learning,

work page 2021
[6]

Addressing function approximation error in actor-critic methods

[Fujimotoet al., 2018 ] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. InInternational Conference on Machine Learning, pages 1587–1596. PMLR,

2018
[8]

Counterfactual explana- tions and how to find them: literature review and bench- marking.Data Mining and Knowledge Discovery, pages 1–55,

[Guidotti, 2022] Riccardo Guidotti. Counterfactual explana- tions and how to find them: literature review and bench- marking.Data Mining and Knowledge Discovery, pages 1–55,

work page 2022
[12]

Stable-baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8,

[Raffinet al., 2021 ] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8,

2021
[13]

Prox- imal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

[Schulmanet al., 2017 ] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

arXiv 2017
[15]

Reinforcement learn- ing application in diabetes blood glucose control: A systematic review.Artificial intelligence in medicine, 104:101836,

[Tejedoret al., 2020 ] Miguel Tejedor, Ashenafi Zebene Woldaregay, and Fred Godtliebsen. Reinforcement learn- ing application in diabetes blood glucose control: A systematic review.Artificial intelligence in medicine, 104:101836,

work page 2020
[17]

Counterfactual explanations and algorithmic recourses for machine learning: A review.ACM Computing Surveys, 56(12):1–42,

[Vermaet al., 2024 ] Sahil Verma, Varich Boonsanong, Minh Hoang, Keegan Hines, John Dickerson, and Chirag Shah. Counterfactual explanations and algorithmic recourses for machine learning: A review.ACM Computing Surveys, 56(12):1–42,

work page 2024

Show all 20 references

[18]

Counterfactual explanations without opening the black box: automated decisions and the gdpr.Harvard Journal of Law and Technology, 31(2),

[Wachteret al., 2018 ] S Wachter, B Mittelstadt, and C Rus- sell. Counterfactual explanations without opening the black box: automated decisions and the gdpr.Harvard Journal of Law and Technology, 31(2),

2018
[19]

Reinforcement learning in healthcare: A survey.ACM Computing Surveys (CSUR), 55(1):1–36,

[Yuet al., 2021 ] Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey.ACM Computing Surveys (CSUR), 55(1):1–36,

2021
[20]

Basal glucose control in type 1 dia- betes using deep reinforcement learning: An in silico vali- dation.IEEE Journal of Biomedical and Health Informat- ics, 25(4):1223–1232, 2020

[Zhuet al., 2020 ] Taiyu Zhu, Kezhi Li, Pau Herrero, and Pantelis Georgiou. Basal glucose control in type 1 dia- betes using deep reinforcement learning: An in silico vali- dation.IEEE Journal of Biomedical and Health Informat- ics, 25(4):1223–1232, 2020

2020
[2014]

Explainable reinforcement learning: A survey and comparative review.ACM Com- puting Surveys,

[Milaniet al., 2023 ] Stephanie Milani, Nicholay Topin, Manuela Veloso, and Fei Fang. Explainable reinforcement learning: A survey and comparative review.ACM Com- puting Surveys,

2023
[2016]

Explain the explainer: Interpreting model-agnostic counterfactual explanations of a deep reinforcement learning agent.IEEE Transactions on Artificial Intelligence, 5(04):1443–1457,

[Chenet al., 2024 ] Ziheng Chen, Fabrizio Silvestri, Gabriele Tolomei, Jia Wang, He Zhu, and Hongshik Ahn. Explain the explainer: Interpreting model-agnostic counterfactual explanations of a deep reinforcement learning agent.IEEE Transactions on Artificial Intelligence, 5(04):...

2024
[2017]

Deep reinforcement learning for robotics: A sur- vey of real-world successes.Annual Review of Control, Robotics, and Autonomous Systems, 8,

[Tanget al., 2024 ] Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Mart ´ın-Mart´ın, and Peter Stone. Deep reinforcement learning for robotics: A sur- vey of real-world successes.Annual Review of Control, Robotics, and Autonomous Systems, 8,

2024
[2018]

Redefining counterfactual explanations for rein- forcement learning: Overview, challenges and opportuni- ties.ACM Computing Surveys, 56(9):1–33,

[Gajcin and Dusparic, 2024] Jasmina Gajcin and Ivana Dus- paric. Redefining counterfactual explanations for rein- forcement learning: Overview, challenges and opportuni- ties.ACM Computing Surveys, 56(9):1–33,

2024
[2020]

Counterfactual explanations in sequential de- cision making under uncertainty.Advances in Neural In- formation Processing Systems, 34:30127–30139,

[Tsirtsiset al., 2021 ] Stratis Tsirtsis, Abir De, and Manuel Rodriguez. Counterfactual explanations in sequential de- cision making under uncertainty.Advances in Neural In- formation Processing Systems, 34:30127–30139,

2021
[2021]

Openai gym.arXiv preprint arXiv:1606.01540,

[Brockman, 2016] G Brockman. Openai gym.arXiv preprint arXiv:1606.01540,

2016 arXiv
[2022]

The uva/padova type 1 diabetes simulator: new features.Journal of Diabetes Science and Technology, 8(1):26–34,

[Manet al., 2014 ] Chiara Dalla Man, Francesco Micheletto, Dayu Lv, Marc Breton, Boris Kovatchev, and Claudio Co- belli. The uva/padova type 1 diabetes simulator: new features.Journal of Diabetes Science and Technology, 8(1):26–34,

2014
[2023]

Coun- terfactual state explanations for reinforcement learning agents via generative deep learning.Artificial Intelligence, 295:103455,

[Olsonet al., 2021 ] Matthew L Olson, Roli Khanna, Lawrence Neal, Fuxin Li, and Weng-Keen Wong. Coun- terfactual state explanations for reinforcement learning agents via generative deep learning.Artificial Intelligence, 295:103455,

2021
[2024]

Learning “what-if” expla- nations for sequential decision-making

[Bicaet al., 2021 ] Ioana Bica, Daniel Jarrett, Alihan Huyuk, and Mihaela van der Schaar. Learning “what-if” expla- nations for sequential decision-making. InInternational Conference on Learning Representations,

2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Explaining reinforcement learning agents through counterfactual action outcomes

[Amitaiet al., 2024 ] Yotam Amitai, Yael Septon, and Ofra Amir. Explaining reinforcement learning agents through counterfactual action outcomes. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10003–10011,

work page 2024

[2] [5]

Explaining reinforcement learning policies through counterfactual trajectories.ICML 2021 Workshop on Human in the Loop Learning,

[Frostet al., 2021 ] Julius Frost, Olivia Watkins, Eric Weiner, Pieter Abbeel, Trevor Darrell, Bryan Plummer, and Kate Saenko. Explaining reinforcement learning policies through counterfactual trajectories.ICML 2021 Workshop on Human in the Loop Learning,

work page 2021

[3] [6]

Addressing function approximation error in actor-critic methods

[Fujimotoet al., 2018 ] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. InInternational Conference on Machine Learning, pages 1587–1596. PMLR,

2018

[4] [8]

Counterfactual explana- tions and how to find them: literature review and bench- marking.Data Mining and Knowledge Discovery, pages 1–55,

[Guidotti, 2022] Riccardo Guidotti. Counterfactual explana- tions and how to find them: literature review and bench- marking.Data Mining and Knowledge Discovery, pages 1–55,

work page 2022

[5] [12]

Stable-baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8,

[Raffinet al., 2021 ] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8,

2021

[6] [13]

Prox- imal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

[Schulmanet al., 2017 ] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

arXiv 2017

[7] [15]

Reinforcement learn- ing application in diabetes blood glucose control: A systematic review.Artificial intelligence in medicine, 104:101836,

[Tejedoret al., 2020 ] Miguel Tejedor, Ashenafi Zebene Woldaregay, and Fred Godtliebsen. Reinforcement learn- ing application in diabetes blood glucose control: A systematic review.Artificial intelligence in medicine, 104:101836,

work page 2020

[8] [17]

Counterfactual explanations and algorithmic recourses for machine learning: A review.ACM Computing Surveys, 56(12):1–42,

[Vermaet al., 2024 ] Sahil Verma, Varich Boonsanong, Minh Hoang, Keegan Hines, John Dickerson, and Chirag Shah. Counterfactual explanations and algorithmic recourses for machine learning: A review.ACM Computing Surveys, 56(12):1–42,

work page 2024