From Explainability to Interpretability: Interpretable Policies in Reinforcement Learning Via Model Explanation

Peilang Li; Umer Siddique; Yongcan Cao

REVIEW 4 major objections 5 minor 1 cited by

From Explainability to Interpretability: Interpretable Policies in Reinforcement Learning Via Model Explanation

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a deep RL policy can be distilled into a transparent, linear decision rule without retraining by locating the action boundary in Shapley value space and mapping it back to the state space.

desk verdict The core idea is appealing but the load-bearing geometry is broken: the boundary-point step is ill-posed and the proof of Property 1 contradicts itself. read the letter →

arxiv 2501.09858 v1 pith:IDA34M3M submitted 2025-01-16 cs.LG cs.AIcs.SYeess.SY

Peilang Li , Umer Siddique , Yongcan Cao This is my paper · ORCID

classification cs.LGcs.AIcs.SYeess.SY

keywords interpretablereinforcementlearningShapleyvaluespolicyextractionmodel-agnosticexplanationdecisionboundarylinearCartPoleMountainCar

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that local Shapley-value explanations of a deep reinforcement learning policy contain enough global structure to reconstruct the policy itself. Its proposed pipeline translates each state into a Shapley vector, clusters these vectors by action, finds the boundary surface between clusters in Shapley space, and maps that surface back to the original state variables with linear regression. The load-bearing claim is Property 1: for a stationary deterministic policy there is a unique boundary surface in Shapley space, and every state's Shapley vector lies at a constant Euclidean distance from it. If this property holds, the recovered boundary yields a simple, fully transparent policy that reproduces—and in the reported experiments sometimes improves on—the original agent's performance while showing lower variance. The method is tested on DQN, PPO, and A2C in CartPole and MountainCar.

What carries the argument

The central object is the Shapley vector $\Phi_s = (\phi_1,\dots,\phi_n)$, the vector of per-feature Shapley contributions to the policy's action choice at state $s$, computed with the on-manifold characteristic function $v_\pi(C) = \sum_{s'} p^\pi(s'|s_C)\pi(s')$. The argument runs through four steps: action k-means clustering of the Shapley vectors into $k$ action groups; boundary-point identification, where a point $X$ between clusters $A_i,A_j$ minimizes $\|X-\mu_i\|_2 - \|X-\mu_j\|_2$; the inverse Shapley value problem, which maps each boundary point back to a state by storing states together with their Shapley vectors; and decision-boundary linear regression, which fits a line $f_{ij}(s)$ in the original state space separating the actions. Property 1 is the load-bearing mechanism: it guarantees the boundary surface exists, is unique, and sits at a constant distance from every Shapley vector, which is what lets the boundary found in Shapley space be trusted as a genuine decision boundary in state space.

What would settle it

Take a trained discrete-action policy, sample a large batch of states, compute each state's Shapley vector, and measure the signed distance from each vector to the plane $\sum_i v_i = \bar{a} + \Delta a/2$; if states sharing the same action do not sit on the same side at a constant distance, or if the linear boundary fitted in the original state space disagrees with the original policy's action on held-out states, the central claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that explanations and policies are not separate: the Shapley vectors of a trained policy form well-separated clouds, one per discrete action, with a unique boundary plane between them. Property 1 states that for a stationary deterministic policy, the efficiency property of Shapley values makes the summed contributions $\sum_i \phi_i = \pi(s) - \mathbb{E}_S[\pi(S)]$, so states choosing different actions differ by a fixed gap $\Delta a$; the boundary surface $B = \{v \in \mathbb{R}^n : \sum_i v_i = \bar{a} + \Delta a/2\}$ therefore separates the action clouds and every Shapley vector has the same Euclidean distance to it. Since the Shapley vectors encode how each state feature drives the action choice, this boundary can be mapped back to a linear decision boundary in the original state space. The paper reports that the resulting interpretable policies match the original returns in CartPole, exceed them for PPO and A2C in MountainCar, and show lower standard deviation than the original policies.

Load-bearing premise

The method assumes that the decision boundary between actions, mapped back from Shapley space, is well approximated by a straight line in the original state variables, and that each boundary point in Shapley space can be matched to a real state; if either assumption fails, the extracted policy will not reproduce the original behavior.

Editorial extensions

If this is right

A practitioner can take an already trained discrete-action deep RL policy and, without any retraining or environment interaction beyond sampling trajectories, replace it with a linear decision rule that a human can read and verify.
Because the extraction is model-agnostic, the same pipeline applies to off-policy learners such as DQN and on-policy learners such as PPO and A2C, so a single interpretability procedure can serve many deployed agents.
The reported reduction in standard deviation for MountainCar suggests that the distilled linear policy is not only faithful but more consistent, which matters for safety-critical deployment where erratic behavior is worse than mild suboptimality.
The explicit boundary equations produced, such as $f_{01} = -0.5x - 0.687\dot{x} - 1.09\theta - \dot{\theta} - 0.018$ for DQN on CartPole, give a direct, checkable specification of the agent's behavior that can be audited before deployment.
If the linear separation in Shapley space holds generally, the approach offers a practical bridge between post-hoc explanation and fully transparent policy design for high-stakes control tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The constant-distance boundary property, if it holds beyond the two tested environments, may extend to any MDP with discrete actions and a stationary policy: the only structural requirement is that the summed Shapley contributions separate cleanly by action, which is testable before applying the method.
The authors list continuous action spaces as future work; a natural extension is to discretize the action space into macro-actions, apply the same boundary extraction, and verify that the resulting piecewise-linear policy is as stable as in the discrete cases studied here.
One could push the regression step beyond linear fits—for example, to sparse or piecewise-linear boundaries—to preserve interpretability while relaxing the linearity assumption, at the cost of slightly more complex extracted rules.
A direct stress test of Property 1 would be to construct a policy whose action regions in state space are deliberately nonlinear, then measure whether the Shapley-space clusters are still separated by a constant-distance plane; failure would show where the method's range ends.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper tries to turn local Shapley explanations into a global interpretable policy by clustering Shapley vectors, locating decision boundaries between clusters, and regressing those boundaries back to state space. The pipeline itself is novel in the cited literature, and the writing is clear. Credit is due for testing on both off-policy and on-policy algorithms and for reporting the linear policies in full.

The problem is that the construction is not actually well-defined. Algorithm 1's boundary point is the minimizer of ||X−μ_i||−||X−μ_j||. That expression has no finite minimizer (the infimum is approached at infinity), and if you add an absolute value, every point on the perpendicular bisector is a minimizer. So X_ij is not unique, and the subsequent 'inverse Shapley' step has no well-defined input. The paper says states are 'systematically stored' but never explains how to choose among the minimizers or map a Shapley-space point back to a state. That alone makes the method unreproducible.

The theory is worse. Property 1's proof is internally inconsistent: equation (10) says the Shapley-sum gap between states with different actions is Δa, so their distances to the boundary surface B in (11) differ by Δa/√n. Equation (13) says the distances are constant for all states. Both cannot be true.

The experiments do not help: 10 episodes, no seeds, no error bars, no baselines beyond the original policies, and no code. The performance and stability claims are unsupported. On MountainCar, the interpretable policy from PPO/A2C 'outperforms' the original, which should have raised concerns rather than being reported as a success.

The salvageable part is the idea of using Shapley vectors as a representation for policy extraction. But as it stands, the central mechanism is ill-posed and the validation is too weak. I would not send this to a serious journal in its current form. A workshop could take it as a position paper, but for peer review it needs a corrected theory and a proper empirical study.

Recommendation: reject, but tell the authors the idea has merit and point them to the specific mathematical issues.

Referee Report

4 major / 5 minor

Summary. The paper proposes a model-agnostic framework that converts deep RL policies into interpretable policies using Shapley values. The method computes Shapley vectors for sampled states, applies k-means clustering to group vectors by action, identifies boundary points between cluster centroids, maps those boundary points back to states via an 'inverse Shapley value' lookup, and fits linear decision boundaries by regression. The authors claim the resulting interpretable policies preserve original performance and improve stability, and they demonstrate this on CartPole and MountainCar using DQN, PPO, and A2C. The paper's central theoretical guarantee is Property 1, which asserts the existence of a unique boundary surface in Shapley vector space with constant distance to all state vectors.

Significance. If the method worked as stated, it would be a useful contribution to interpretable RL: it is model-agnostic, applies to both off-policy and on-policy algorithms, and yields closed-form linear policies that are straightforward to deploy in high-stakes settings. The authors also provide explicit boundary equations in Tables 1 and 2, which supports reproducibility. However, the paper's load-bearing theoretical result (Property 1) is not proven as written, and the key inversion step in Algorithm 1 is mathematically ill-posed. The experimental evaluation is too thin to establish the headline claims of performance preservation and stability. The significance is therefore conditional on a substantial revision of the method's foundations.

major comments (4)

[Section 4, Property 1 proof (Eqs. 7-13)] Equation (13) contradicts Eq. (10). For states s_p and s_q with different actions, Eq. (10) states |Σ_i φ_i,s_p − Σ_i φ_i,s_q| = Δa. With the boundary B defined in Eq. (11) as {v : Σ_i v_i = ā + Δa/2}, the distance formula in Eq. (12) gives dist(Φ_s, B) = |Σ_i φ_i − (ā + Δa/2)| / √n. Therefore the distances for s_p and s_q differ by Δa/√n, not remain constant as claimed in Eq. (13). Thus Property 1 is unproven and the claimed existence/uniqueness of the decision boundary is not established.
[Algorithm 1, line 8 and Eq. (6)] The boundary-point optimization is ill-posed. The objective in line 8 is argmin_X (||X−μ_i||_2 − ||X−μ_j||_2). Without an absolute value, this expression is unbounded below along the ray X = μ_i + t(μ_i−μ_j) as t→+∞, so no minimizer exists. If the intended objective is |||X−μ_i||_2 − ||X−μ_j||_2|, then every point on the perpendicular bisector hyperplane {X : ||X−μ_i||_2 = ||X−μ_j||_2} attains the minimum, so X_ij is not unique. In either reading, the subsequent inverse map φ^{−1}(X_ij) has no well-defined input, and the 'systematically storing' description does not specify which of the infinitely many minimizers is selected. The extracted boundary f_ij is therefore not well-defined.
[Section 4, Inverse Shapley Values] The inverse Shapley value problem is not mathematically well-posed. The map from states to Shapley vectors is not injective in general; the authors themselves note that states contributing equally to a decision yield identical Shapley values, which is the symmetry property of Shapley values. Even if the map were injective on the sampled states, a boundary point X_ij in Shapley space will generically not coincide with any stored state's Shapley vector, and no interpolation, nearest-neighbor, or other selection rule is specified. Consequently, the mapping from boundary points in Shapley space to boundary states in state space is undefined, and the regression step in the 'Decision Boundary Regression' paragraph depends on an unspecified choice.
[Experiments (Figures 2 and 4)] The experimental evidence does not support the strong claim that the interpretable policy 'preserves the original models' performance.' Figures 2 and 4 show only 10 episodes per algorithm with no error bars across independent training runs, no comparison with existing interpretable-policy baselines (e.g., VIPER or PIRL), and no statistical significance tests. In MountainCar, the interpretable policies from PPO and A2C outperform the original deep RL policies, a result that is unexplained and suggests either the base policies are suboptimally trained or the evaluation is too short to be representative. These issues matter because performance preservation is the paper's headline empirical claim.

minor comments (5)

[MountainCar section] The text refers to 'the first row of Figure 4' for the Shapley vector visualization, but the visualization is in Figure 3; Figure 4 is the performance bar chart. This cross-reference should be corrected.
[Background, Shapley Values in RL] Equations (2) and (3) use pπ(s'|sc) with a subscript 'sc' that is not defined; the authors define s_C only for C ⊂ N. The notation should be unified and the conditional distribution over states given a partial observation should be stated more carefully.
[Action K-Means Clustering] The clustering step assumes that k-means clusters correspond to action regions, but the cluster labels produced by k-means are arbitrary; the paper does not explain how cluster indices are aligned with actual action labels before assigning the decision rule (e.g., 'if f01 > 0, select action 0').
[Shapley Values in Reinforcement Learning] The statement that a positive φ_i indicates feature i encourages selection of action a2 while a negative value suggests a preference for a1 is not generally valid: it presupposes a monotonic encoding of discrete actions (e.g., 0 and 1) as numerical values, which is an arbitrary modeling choice.
[Tables 1 and 2] The boundary equations are presented without specifying the preprocessing or scaling of state features; without this information the reported coefficients are not reproducible across different environment wrappers or initial seeds.

Circularity Check

1 steps flagged · score 3.0 of 10

The interpretable policy is fitted to Shapley-vector boundary points derived from the original policy, so the reported performance preservation is a fitted outcome rather than an independent prediction; no self-citation-based circularity is present.

fitted input called prediction [Algorithm 1 (lines 8-14); Method, 'Inverse Shapley Values' and 'Decision Boundary Regression'; performance claims in the CartPole experiments]
"We address this problem by systematically storing the original states with their corresponding Shapley value vectors ... After the boundary state points sij are discovered using Shapley values, the decision can be drawn accordingly. ... we use linear regression ... The results, shown in Figure 2, demonstrate that the interpretable policies consistently achieved the maximum reward of 500 across all algorithms. This indicates that our method preserves the performance of the original deep RL algorithms while providing interpretability."

The boundary state points sij are obtained by applying the stored-state inverse map φ^{-1} to Shapley-space points Xij, which are computed from Shapley vectors of the original policy. Linear regression on sij then produces fij, the interpretable policy. Thus the policy parameters are fitted to the original policy's own decision regions, and the reported performance preservation is a measure of that fit (with linear-boundary approximation error) on the same environment, not an independent first-principles prediction. Up to regression error, the result is constructed from the original policy's Shapley-vector data.

full rationale

The paper does not lean on self-citations for its central mechanism: the author self-citations (Siddique et al. 2020; Wu et al. 2024) are motivational, and the Shapley-value framework is cited to external work (Beechey et al.; Frye et al.; Lundberg and Lee). Property 1, although its proof appears internally inconsistent and disconnected from Algorithm 1, is a mathematical claim rather than a circular step. The main circularity is the fitted-input pattern: the interpretable policy is produced by fitting a linear boundary to boundary points recovered from the original policy's Shapley vectors, so the claim that it 'preserves the original models' performance' is largely an evaluation of that fitted approximation. There is still independent empirical content in whether a linear boundary in the original state space generalizes over 10 test episodes, which keeps the circularity moderate rather than complete.

Assumptions & free parameters 2 free parameters · 6 assumptions · 1 invented entities

The method relies on several unproven heuristics: the correspondence between k-means clusters and action regions, the invertibility of the Shapley-to-state mapping, and the linearity of the state-space decision boundary. The theoretical proof offered is invalid, so these remain assumptions rather than established facts.

free parameters (2)

Cluster centroids μ_i (k-means) = not reported (illustrated in Figure 1)
Computed by action k-means on Shapley vectors; the boundary points are derived from these centroids, so the resulting policy depends on them.
Linear decision boundary regression coefficients = Tables 1 and 2, e.g., CartPole DQN f01 = -0.5x - 0.687 xdot - 1.09 theta - thetadot - 0.018
The final interpretable policy is defined by these fitted coefficients; they are fit to the boundary points and directly determine action selection.

assumptions (6)

standard math Shapley value efficiency property: sum of feature contributions equals policy output minus expected policy output (Eq. 7).
Invoked in the proof of Property 1; however, the stated form is not the standard efficiency property for the on-manifold characteristic function, so the justification is flawed.
domain assumption On-manifold characteristic function vπ(C) = sum over states pπ(s'|s_C) π(s') as in Frye et al. and Beechey et al.
Used to compute Shapley values that respect feature correlations; the paper relies on this without further validation in these environments.
domain assumption The state distribution under the trained policy is stationary and the expected action value converges to a fixed scalar (Eq. 9).
Used in the proof of Property 1; the assumption is unverified in the experiments.
ad hoc to paper K-means clustering with k equal to the number of discrete actions groups Shapley vectors by action region.
No proof or analysis is given that clusters correspond to actions; it is a central heuristic of Algorithm 1.
ad hoc to paper The mapping from Shapley vectors back to states is practically invertible via a stored lookup table.
The 'Inverse Shapley Values' section asserts this without specifying the inverse operation or analyzing its error.
ad hoc to paper The decision boundary in the original state space is approximately linear.
Linear regression is chosen 'due to its simplicity and interpretability'; nonlinear boundaries would break the extracted policy.

invented entities (1)

Decision boundary surface B in Shapley vector space
purpose: Defines the threshold separating action regions and is used to derive the state-space decision boundary.
The existence and uniqueness of B are claimed in Property 1, but the proof is internally inconsistent, so B is a postulated construct without verified foundation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of From Explainability to Interpretability: Interpretable Policies in Reinforcement Learning Via Model Explanation." pith.science (2026). https://pith.science/paper/IDA34M3M

@misc{pith2026250109858,
  author       = {Pith},
  title        = {Pith review of: From Explainability to Interpretability: Interpretable Policies in Reinforcement Learning Via Model Explanation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IDA34M3M}},
  note         = {Machine review of arXiv:2501.09858}
}

read the original abstract

Deep reinforcement learning (RL) has shown remarkable success in complex domains, however, the inherent black box nature of deep neural network policies raises significant challenges in understanding and trusting the decision-making processes. While existing explainable RL methods provide local insights, they fail to deliver a global understanding of the model, particularly in high-stakes applications. To overcome this limitation, we propose a novel model-agnostic approach that bridges the gap between explainability and interpretability by leveraging Shapley values to transform complex deep RL policies into transparent representations. The proposed approach offers two key contributions: a novel approach employing Shapley values to policy interpretation beyond local explanations and a general framework applicable to off-policy and on-policy algorithms. We evaluate our approach with three existing deep RL algorithms and validate its performance in two classic control environments. The results demonstrate that our approach not only preserves the original models' performance but also generates more stable interpretable policies.

Figures

Figures reproduced from arXiv: 2501.09858 by the authors.

**Figure 1.** Visualization of Shapley values and interpretable policy formulation in the CartPole. The first row depicts the Shapley [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

**Figure 2.** Performances of the interpretable policy with orig [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Visualization of Shapley values and interpretable policy formulation in the MountainCar. The first row depicts the [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Performances of the interpretable policy with orig [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Interpret Policies in Deep Reinforcement Learning using SILVER with RL-Guided Labeling: A Model-level Approach to High-dimensional and Multi-action Environments
cs.LG 2025-10 reject novelty 4.0 of 10

SILVER with RL-guided labeling: SHAP plus clustering plus policy-query labels plus decision trees or regression to interpret multi-action Atari policies.

Reference graph

Works this paper leans on

32 extracted references · 22 canonical work pages · cited by 1 Pith paper

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Amir, D.; and Amir, O. 2018. HIGHLIGHTS: Summarizing Agent Behavior to People. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '18, 1168–1176. Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems

work page 2018
[4]

Bastani, O.; Pu, Y.; and Solar-Lezama, A. 2018. Verifiable reinforcement learning via policy extraction. Advances in neural information processing systems, 31

work page 2018
[5]

M.; and S im s ek, \"O

Beechey, D.; Smith, T. M.; and S im s ek, \"O . 2023. Explaining reinforcement learning with shapley values. In International Conference on Machine Learning, 2003--2014. PMLR

work page 2023
[6]

Frye, C.; de Mijolla, D.; Begley, T.; Cowton, L.; Stanley, M.; and Feige, I. 2021. Shapley explainability on the data manifold. In International Conference on Learning Representations

work page 2021
[7]

Glanois, C.; Weng, P.; Zimmer, M.; Li, D.; Yang, T.; Hao, J.; and Liu, W. 2024. A survey on interpretable reinforcement learning. Machine Learning, 1--44

work page 2024
[8]

Gu, S.; Holly, E.; Lillicrap, T.; and Levine, S. 2017. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference on robotics and automation, 3389--3396. IEEE

work page 2017

Show all 32 references

[9]

Hein, D.; Udluft, S.; and Runkler, T. A. 2018. Interpretable policies for reinforcement learning by genetic programming. Engineering Applications of Artificial Intelligence, 76: 158--169

2018
[10]

Henderson, P.; Islam, R.; Bachman, P.; Pineau, J.; Precup, D.; and Meger, D. 2018. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32

2018
[11]

H.; Bhatia, K.; Abbeel, P.; and Dragan, A

Huang, S. H.; Bhatia, K.; Abbeel, P.; and Dragan, A. D. 2018. Establishing appropriate trust via critical states. In 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), 3929--3936. IEEE

2018
[12]

Juozapaitis, Z.; Koul, A.; Fern, A.; Erwig, M.; and Doshi-Velez, F. 2019. Explainable reinforcement learning via reward decomposition. In IJCAI/ECAI Workshop on explainable artificial intelligence

2019
[13]

M.; and Lee, S.-I

Lundberg, S. M.; and Lee, S.-I. 2017. A Unified Approach to Interpreting Model Predictions. In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., Advances in Neural Information Processing Systems 30, 4765--4774. Curran Asso...

2017
[14]

P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; and Kavukcuoglu, K

Mnih, V.; Badia, A. P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; and Kavukcuoglu, K. 2016. Asynchronous Methods for Deep Reinforcement Learning. In Balcan, M. F.; and Weinberger, K. Q., eds., Proceedings of The 33rd International Conference on Machine Lear...

2016
[15]

A.; Veness, J.; Bellemare, M

Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; Petersen, S.; Beattie, C.; Sadik, A.; Antonoglou, I.; King, H.; Kumaran, D.; Wierstra, D.; Legg, S.; and Hassabis, D. 2015 a . Human-l...

2015
[16]

A.; Veness, J.; Bellemare, M

Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; et al. 2015 b . Human-level control through deep reinforcement learning. nature, 518(7540): 529--533

2015
[17]

L.; Khanna, R.; Neal, L.; Li, F.; and Wong, W.-K

Olson, M. L.; Khanna, R.; Neal, L.; Li, F.; and Wong, W.-K. 2021. Counterfactual state explanations for reinforcement learning agents via generative deep learning. Artificial Intelligence, 295: 103455

2021
[18]

Why Should I Trust You?

Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, 1135–1144. New York, NY, USA: Association ...

2016
[19]

Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

2017 arXiv
[20]

Shapley, L. S. 1953. A value for n-person games. Contribution to the Theory of Games, 2

1953
[21]

Siddique, U.; Weng, P.; and Zimmer, M. 2020. Learning fair policies in multi-objective (deep) reinforcement learning with average and discounted rewards. In International Conference on Machine Learning, 8905--8915. PMLR

2020
[22]

Silva, A.; Gombolay, M.; Killian, T.; Jimenez, I.; and Son, S.-H. 2020 a . Optimization Methods for Interpretable Differentiable Decision Trees Applied to Reinforcement Learning. In Chiappa, S.; and Calandra, R., eds., Proceedings of the Twenty Third International Conference o...

2020
[23]

Silva, A.; Gombolay, M.; Killian, T.; Jimenez, I.; and Son, S.-H. 2020 b . Optimization methods for interpretable differentiable decision trees applied to reinforcement learning. In International conference on artificial intelligence and statistics, 1855--1865. PMLR

2020
[24]

Silver, D.; Schrittwieser, J.; Simonyan, K.; Antonoglou, I.; Huang, A.; Guez, A.; Hubert, T.; Baker, L.; Lai, M.; Bolton, A.; Chen, Y.; Lillicrap, T.; Hui, F.; Sifre, L.; van den Driessche, G.; Graepel, T.; and Hassabis, D. 2017. Mastering the game of Go without human knowledg...

2017
[25]

S trumbelj, E.; and Kononenko, I. 2010. An Efficient Explanation of Individual Classifications using Game Theory. Journal of Machine Learning Research, 11(1): 1--18

2010
[26]

S trumbelj, E.; and Kononenko, I. 2014. Explaining prediction models and individual predictions with feature contributions. Knowledge and information systems, 41: 647--665

2014
[27]

S.; and Barto, A

Sutton, R. S.; and Barto, A. G. 2018. Reinforcement Learning: An Introduction. The MIT Press, second edition

2018
[28]

U.; De Cola, G.; Deleu, T.; Goulao, M.; Kallinteris, A.; Krimmel, M.; KG, A.; et al

Towers, M.; Kwiatkowski, A.; Terry, J.; Balis, J. U.; De Cola, G.; Deleu, T.; Goulao, M.; Kallinteris, A.; Krimmel, M.; KG, A.; et al. 2024. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032

2024 arXiv
[29]

Verma, A.; Murali, V.; Singh, R.; Kohli, P.; and Chaudhuri, S. 2018. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning, 5045--5054. PMLR

2018
[30]

Wachter, S.; Mittelstadt, B.; and Russell, C. 2017. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harv. JL & Tech., 31: 841

2017
[31]

Wu, M.; Siddique, U.; Sinha, A.; and Cao, Y. 2024. Offline Reinforcement Learning with Failure Under Sparse Reward Environments. In 2024 IEEE 3rd International Conference on Computing and Machine Intelligence (ICMI), 1--5. IEEE

2024
[32]

Zahavy, T.; Ben-Zrihem, N.; and Mannor, S. 2016. Graying the black box: Understanding dqns. In International conference on machine learning, 1899--1908. PMLR

2016

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Amir, D.; and Amir, O. 2018. HIGHLIGHTS: Summarizing Agent Behavior to People. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '18, 1168–1176. Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems

work page 2018

[4] [4]

Bastani, O.; Pu, Y.; and Solar-Lezama, A. 2018. Verifiable reinforcement learning via policy extraction. Advances in neural information processing systems, 31

work page 2018

[5] [5]

M.; and S im s ek, \"O

Beechey, D.; Smith, T. M.; and S im s ek, \"O . 2023. Explaining reinforcement learning with shapley values. In International Conference on Machine Learning, 2003--2014. PMLR

work page 2023

[6] [6]

Frye, C.; de Mijolla, D.; Begley, T.; Cowton, L.; Stanley, M.; and Feige, I. 2021. Shapley explainability on the data manifold. In International Conference on Learning Representations

work page 2021

[7] [7]

Glanois, C.; Weng, P.; Zimmer, M.; Li, D.; Yang, T.; Hao, J.; and Liu, W. 2024. A survey on interpretable reinforcement learning. Machine Learning, 1--44

work page 2024

[8] [8]

Gu, S.; Holly, E.; Lillicrap, T.; and Levine, S. 2017. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference on robotics and automation, 3389--3396. IEEE

work page 2017