Pith. sign in

REVIEW 3 major objections 8 minor 78 references

ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies

T0 review · 3 major / 8 minor · reviewed 2026-07-09 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Oblique trees with local linear models distill deep RL policies

desk verdict Solid engineering contribution to XRL with continuous actions; the linear-in-leaf assumption is the right thing to worry about, and Half Cheetah is where it shows. read the letter →

arxiv 2607.07235 v1 pith:FTHMFQTO submitted 2026-07-08 cs.LG cs.AIcs.SE

Ignacio D. Lopez-Miguel , Ezio Bartocci , Thomas Eiter , Martin Tappler This is my paper · ORCID
classification cs.LGcs.AIcs.SE
keywords deepinterpretableorcaidpoliciespolicyrule-basedactioncontinuous
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Deep reinforcement learning policies are powerful but opaque: they map sensory inputs to continuous control actions through neural networks that no human can inspect or verify. ORCAID is a method for converting such a policy into a small set of human-readable rules. Each rule says: when the system's state falls in a certain region (defined by a slanted hyperplane rather than a single feature threshold), apply a simple linear formula to compute the action. The method builds an oblique decision tree by searching for good hyperplane splits in three stages—random initialization, local optimization, and backward elimination of unnecessary features—then merges adjacent regions that can share the same linear model. The result is a compact surrogate that closely mimics the original neural network's behavior, retains strong task performance across nine control benchmarks, and uses far fewer parameters than standard axis-aligned decision trees or competing rule-learning methods. The paper also shows that the extracted rules can be mirrored along symmetry axes to identify and fix weaknesses in the original policy, yielding measurable performance improvements.

What carries the argument

The load-bearing mechanism is the oblique split condition: a single inequality w · x ≤ b where w is a learned weight vector over all state features, not just one. This lets one split boundary separate states that an axis-aligned tree would need many nested thresholds to distinguish. Inside each leaf, ordinary least-squares regression maps state features to continuous action values, with backward elimination pruning irrelevant coefficients. The QM (Quine-McCluskey) method then simplifies the Boolean logic of merged regions into minimal disjunctive normal form. A DAgger loop iteratively collects states where the surrogate and original policy disagree, retraining to close the gap.

What would settle it

Find a continuous-action RL environment where the optimal policy's action function is highly nonlinear within any convex region of the state space (e.g., a policy that requires sinusoidal or oscillatory action modulation within a single operating mode). If ORCAID's linear leaf models cannot approximate such functions without an impractical number of splits, the method would fail to produce a compact and faithful surrogate, and model drift would degrade task performance below acceptable thresholds.

Watch

Extended reading notes

Core claim

The central technical claim is that partitioning the state space with oblique (linear-combination) hyperplanes and fitting a separate pruned linear regression in each leaf produces a faithful, compact surrogate of a continuous-action deep RL policy. The three-stage split search—random hyperplane initialization via solving Aw = -1, differential evolution refinement, and backward feature elimination—makes this tractable. Region merging via DNF simplification then reduces the rule count without sacrificing fidelity. Across nine environments, ORCAID achieves reward ratios at or near the original policy while using model sizes consistently smaller than axis-aligned decision trees, Cubist, and en.

Load-bearing premise

The method assumes that within each region carved out by hyperplane splits, the original neural network's action function is well-approximated by a single linear model. If the policy's action surface curves sharply within a region—not just across region boundaries—the linear fit introduces errors that accumulate as the surrogate executes, causing trajectories to diverge from the original agent's behavior. The DAgger loop mitigates this by collecting new data where the two dev

Editorial extensions

If this is right

  • Safety-critical RL deployments (robotics, autonomous driving, industrial control) could replace opaque neural network policies with verifiable rule sets that regulators can inspect, satisfying transparency requirements like the EU AI Act's Article 13.3.
  • The rule-mirroring approach to policy improvement suggests a general methodology: extract interpretable structure from a learned policy, apply domain symmetries to generate counterfactual rules, and use enforcement testing to find regions where the original policy is suboptimal.
  • The oblique-tree training algorithm (random init + differential evolution + backward elimination) could be applied beyond RL policy distillation to any regression or classification problem where compact oblique partitions are preferred over deep axis-aligned trees.
  • The DAgger integration with oblique trees provides a template for iterative surrogate refinement that could extend to stochastic policies if leaf models are replaced with linear-Gaussian distributions, as the authors note.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper introduces ORCAID, a method for extracting interpretable rule-based surrogate models from deep RL policies with continuous action spaces. The approach learns oblique decision trees (hyperplane-based splits) with local linear regression models in each leaf, then merges adjacent regions and prunes features to produce compact DNF rules. The method is evaluated on nine Gymnasium control tasks against decision trees (CART), Cubist, and RuleFit, with all methods wrapped in a DAgger loop. The paper also demonstrates that ORCAID rules can be mirrored to identify and exploit policy weaknesses (adapted LEGIBLE approach), and uses an LLM-as-a-Judge protocol to assess interpretability. The core idea—oblique splits plus per-leaf linear models for continuous-action policy distillation—is sound and the experimental coverage is broad. The main concerns center on the Half Cheetah underperformance, the absence of per-region diagnostics, and the validity of the LLM-based interpretability evaluation.

Significance. The paper addresses a genuine gap: most XRL work on policy extraction targets discrete actions, and oblique trees with linear leaf models for continuous actions are not well explored. The method is practical, the code is publicly available, and the evaluation across nine environments with multiple baselines is substantial. The policy-improvement application via rule mirroring is a creative and actionable contribution. The LLM-as-a-Judge protocol for interpretability is novel in this context, though its reliability is debatable. The work is a solid contribution to the XRL literature.

major comments (3)
  1. §5.1, Performance paragraph and Figure 3: The paper states that Cubist's advantage in Half Cheetah (HC) is 'non-significant, by the confidence intervals.' From Table 25, ORCAID achieves 78±2% and Cubist 85±1% reward ratio. Using the paper's own CI formula (t_{0.025,8} ≈ 2.3, N=9), ORCAID's CI is approximately [73.4, 82.6] and Cubist's is [82.7, 87.3]. These intervals do not overlap, contradicting the 'non-significant' characterization. This matters because HC is the highest-dimensional environment (17-D state, 6-D action) and is the primary case where the linear-in-leaf assumption appears to strain. The paper should either correct the significance claim or provide a proper statistical test (e.g., paired t-test or bootstrap) to support it.
  2. §4.3 and §5.1: The paper's central methodological assumption is that local linear models in oblique-tree leaves adequately approximate the policy within each region. The Half Cheetah underperformance and the ablation results (Table 64: removing DAgger drops Swimmer from 90% to 72%) suggest that DAgger is doing substantial compensatory work, but the paper provides no per-region MSE breakdowns or diagnostics to distinguish 'linear models are adequate here' from 'DAgger is compensating for within-region nonlinearity.' Adding per-region MSE statistics (or at least a histogram of leaf-level MSEs for HC vs. a well-performing environment like Pendulum) would substantially strengthen the paper's claims and help readers understand when the method is expected to work.
  3. §5.1, Table 2, and Appendix D: The LLM-as-a-Judge evaluation uses six free-tier LLMs (Appendix D.1.3) to rate interpretability on a 1–5 Likert scale across 15 questions. The axiom that LLMs can reliably assess interpretability of rule-based models is load-bearing for RQ4, yet the paper provides no validation against human judgments. The paper acknowledges this limitation in §6 ('establishing agreement with human judgments remains important future work'), but RQ4 is still answered affirmatively in the main text. At minimum, the paper should (a) report inter-LLM agreement (e.g., Krippendorff's alpha or ICC across the six judges), and (b) temper the RQ4 conclusion to reflect that this is an exploratory evaluation, not a validated measure of interpretability.
minor comments (8)
  1. Table 1, IP row: The entry '1720' for ORCAID's Rate (%) appears to be a formatting error (likely '17' or '20'). Please fix the table formatting.
  2. Figure 3: The x-axis label 'Model Size (log)' is ambiguous—it is unclear whether the axis is log-scaled or whether the values themselves are logarithms. The tick labels (10^0 through 10^4) suggest log-scaling, but this should be stated explicitly.
  3. §4.1, step 3 (Sampling): The noise standard deviation σ is set to 0.15 for most environments (Table 24) but 0.01–0.05 for others. The paper states 'small noise was enough to obtain good results' but does not explain why HC and Hopper require much smaller σ. A brief discussion of how σ was selected and why it varies would help reproducibility.
  4. Definition 7 (ORCAID model size): The distinction between M_conds (non-zero coefficients in conditions) and M_Bools (count of conditions) is slightly confusing. If a condition has 3 non-zero coefficients, it contributes 3 to M_conds and 1 to M_Bools? Clarifying with a small worked example beyond Example 3 would help.
  5. §4.4: The merging criteria reference τ = 2·max_mse, but max_mse is a per-environment hyperparameter (Table 24). This means the merging threshold varies across environments in a way that is not transparent from the main text. A note directing readers to Table 24 for the actual τ values would help.
  6. Appendix D.1.3: The list of LLMs includes 'openai/gpt-oss-120b' which does not appear to be a standard model name. Please verify and correct.
  7. §3, Example 1: The Mountain Car description says the car 'applies continuous left/right acceleration,' but the standard Gymnasium MountainCar has a discrete action space. The continuous version (MountainCarContinuous) should be specified.
  8. References: The citation to Tappler et al. [43] (LEGIBLE) is described as 'adapted' for continuous environments. Since this adaptation is non-trivial and central to RQ3, a brief note on what changed beyond 'mirroring along axes' would help readers understand the contribution boundary.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; one minor self-citation for the LEGIBLE policy-improvement framework that is not load-bearing for the paper's central claims

full rationale

The paper's core derivation chain is: (1) collect state-action data from a deep RL policy, (2) learn an oblique decision tree with linear leaf models via random initialization + differential evolution + backward elimination, (3) merge adjacent regions and simplify conditions, (4) evaluate the surrogate against two external benchmarks: fidelity (MSE between surrogate and original policy actions) and performance (cumulative reward in the environment). None of these steps reduce to their inputs by construction. The surrogate is not defined in terms of the evaluation metrics; the metrics (MSE, reward ratio, model size per Def. 7) are computed independently. The DAgger loop (from [34], no author overlap) iteratively adds data where the surrogate deviates from the policy, but the retraining still optimizes against the original policy's actions — an external target, not a self-referential one. The one self-citation is [43] (Tappler, Lopez-Miguel — two of four present authors), which provides the LEGIBLE framework for rule mirroring used in RQ3 (policy improvement). However, this self-citation is not load-bearing for the paper's central claims (RQ1: small model size + strong performance; RQ2: low MSE/fidelity; RQ4: interpretability). The RQ3 results are externally falsifiable: mirrored rules are enforced in the environment and reward is measured. The self-citation provides the transformation methodology, not the empirical result. No uniqueness theorem is invoked, no ansatz is smuggled, and no prediction reduces to a fitted parameter by construction. The derivation is self-contained against external benchmarks.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities or mathematical objects beyond standard decision tree components (hyperplanes, linear models, regions). The free parameters are hyperparameters tuned per environment, which is standard for ML methods. The key axioms are domain assumptions about the adequacy of linear approximations and the DAgger loop, plus the ad-hoc use of LLM judges for interpretability evaluation.

free parameters (8)
  • max_depth = varies per environment (1-6)
    Maximum tree depth, tuned per environment to maximize performance before test MSE stops improving.
  • max_mse = varies (0.001-0.45)
    MSE threshold below which a leaf stops growing; tuned per environment (Table 24).
  • sigma (noise std) = 0.01-0.15
    Standard deviation of Gaussian noise added to cluster centroids during sampling; tuned per environment.
  • k (number of clusters) = 4-6000
    Number of k-means clusters; chosen by elbow method but actual values vary widely per environment.
  • epsilon (merge threshold) = 0.05
    Relative improvement threshold for region merging; set globally.
  • tau (global MSE merge bound) = 2*max_mse
    Global error bound for merging adjacent regions; derived from max_mse.
  • delta (backward elimination threshold) = not specified
    MSE increase threshold for feature removal in backward elimination; value not given in paper.
  • pca_dim = not specified
    Number of PCA components retained for split search; mentioned in Algorithm 1 but value not specified.
assumptions (5)
  • domain assumption Local linear models can adequately approximate the policy's action function within each oblique-tree leaf region.
    This is the fundamental modeling assumption. If the action surface is highly nonlinear within regions, the linear models introduce errors. The paper does not provide formal conditions under which this holds.
  • domain assumption The DAgger loop with disagreement filtering suffices to mitigate model drift.
    The iterative data aggregation is assumed to handle the compounding of small errors during execution. The paper acknowledges model drift but does not prove convergence or bounded drift.
  • domain assumption Oblique hyperplanes found via random initialization and differential evolution in PCA space yield good partitions of the original state space.
    The back-mapping from PCA space to original space (line 21 of Alg. 1) assumes that good splits in reduced space correspond to good splits in original space. No formal guarantee is provided.
  • standard math Standard OLS regression is appropriate for fitting action models in each leaf.
    OLS is a standard method; the assumption is that the linear model class is appropriate, which is a domain assumption about the smoothness of the policy.
  • ad hoc to paper LLM judges can reliably assess interpretability of rule-based models.
    The LLM-as-a-Judge evaluation (Table 2) assumes that LLM ratings correlate with human interpretability judgments. The authors acknowledge this is unvalidated and defer human evaluation to future work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies." pith.science (2026). https://pith.science/paper/FTHMFQTO

@misc{pith2026260707235,
  author       = {Pith},
  title        = {Pith review of: ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FTHMFQTO}},
  note         = {Machine review of arXiv:2607.07235}
}
read the original abstract

Explainability remains a key issue in reinforcement learning (RL). Distilling an interpretable policy from an agent trained in a complex environment is particularly challenging when the action space is continuous. We introduce ORCAID, a novel method for extracting interpretable rule-based policies from RL agents operating in mixed continuous-discrete environments with continuous action spaces. Our main contribution is an efficient oblique decision tree training algorithm that partitions the state space by hyperplanes and fits local linear models. The key idea lies in a three-stage split search: efficient random initialization, local refinement, and backward elimination. Finally, adjacent leaves are merged to yield a concise set of interpretable rules describing a given deep RL policy. We evaluate ORCAID across multiple RL environments, demonstrating that the extracted rule-based policies maintain strong performance with a low number of parameters and can even be used to improve the performance of the original deep RL policy.

Figures

Figures reproduced from arXiv: 2607.07235 by the authors.

Figure 1
Figure 1. DT vs. Oblique DT. Despite this success, the application of deep RL is hampered by the opacity of learned control policies, which are difficult to verify and understand. This has led regulators to require greater transparency in AI systems used in critical domains, e.g. the [48] and the [11]. For example, the EU AI Act specifies in Article 13.3 “The instructions for use shall contain [...] technical capabilities and… view at source ↗
Figure 2
Figure 2. Overview of our proposed method. policy π : S → A maximizing the expected discounted return E P∞ t=0 γ t r(st, at) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Model size & surrogate/RL reward ratio per envi [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Test MSE scaled (best:0) according to the output [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Swimmer in Region 0 from the ORCAID learned model with depth 1 [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 7
Figure 7. Figure 7: Hopper environment whose state has been set manually [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 78 canonical work pages

  1. [1]

    Python package for fitting quinlan’s cubist v2.07 regression model, 2021

    Aselin et al. Python package for fitting quinlan’s cubist v2.07 regression model, 2021

    work page 2021

  2. [2]

    Verifiable reinforcement learning via policy extraction

    Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors,NeurIPS 2018, pages 2499–2509, 2018

    work page 2018

  3. [3]

    Leo Breiman, J. H. Friedman, Richard A. Olshen, and C. J. Stone.Classification and Regression Trees. Wadsworth, 1984

    work page 1984

  4. [4]

    Human-friendly explanations checklist for reinforcement learning: XRL H-F-E checklist

    Daniel Adrián Contreras Olivas and Lourdes Martinez-Villaseñor. Human-friendly explanations checklist for reinforcement learning: XRL H-F-E checklist. InArtificial Intelligence – COMIA 2025, pages 268–279, 2025

    work page 2025

  5. [5]

    Distilling deep reinforce- ment learning policies in soft decision trees

    Youri Coppens, Kyriakos Efthymiadis, Tom Lenaerts, and Ann Nowe. Distilling deep reinforce- ment learning policies in soft decision trees. InIJCAI 2019 Workshop on Explainable Artificial Intelligence, pages 1–6, August 2019

    work page 2019

  6. [6]

    Yuxin Daiet al.Enhanced oblique decision tree enabled policy extraction for deep reinforcement learning in power system emergency control.Electric Power Systems Research, 209:107932, 2022

    work page 2022

  7. [7]

    Danesh, Anurag Koul, Alan Fern, and Saeed Khorram

    Mohamad H. Danesh, Anurag Koul, Alan Fern, and Saeed Khorram. Re-understanding finite- state representations of recurrent policy networks. InICML 2021, volume 139 ofProceedings of Machine Learning Research, pages 2388–2397. PMLR, 2021

    work page 2021

  8. [8]

    Nat., 602(7897):414–419, 2022

    Jonas Degraveet al.Magnetic control of tokamak plasmas through deep reinforcement learning. Nat., 602(7897):414–419, 2022

    work page 2022

Show all 78 references
  1. [9]

    Dhebar, Kalyanmoy Deb, Subramanya Nageshrao, Ling Zhu, and Dimitar P

    Yashesh D. Dhebar, Kalyanmoy Deb, Subramanya Nageshrao, Ling Zhu, and Dimitar P. Filev. Toward interpretable-AI policies using evolutionary nonlinear decision trees for discrete-action systems.IEEE Trans. Cybern., 54(1):50–62, 2024

    2024

  2. [10]

    Charles W. Dunnett. A multiple comparison procedure for comparing several treatments with a control.Journal of the American Statistical Association, 50(272):1096–1121, 1955

    1955

  3. [11]

    Regulation (eu) 2024/1689 of the European parliament and of the council – the EU artificial intelligence act

    European Parliament. Regulation (eu) 2024/1689 of the European parliament and of the council – the EU artificial intelligence act. online https://eur-lex.europa.eu/legal-content/ EN/TXT/?uri=CELEX%3A32024R1689&qid=1753356665433, 6 2024

    2024

  4. [12]

    Friedman and Bogdan E

    Jerome H. Friedman and Bogdan E. Popescu. Predictive learning via rule ensembles.The Annals of Applied Statistics, 2(3):916–954, September 2008

    2008

  5. [13]

    A brief overview of rule learning

    Johannes Fürnkranz and Tomás Kliegr. A brief overview of rule learning. InRuleML 2015, volume 9202 ofLNCS, pages 54–69. Springer, 2015

    2015

  6. [14]

    Shruti Govinda, Bouziane Brik, and Saad Harous. A survey on deep reinforcement learning applications in autonomous systems: Applications, open challenges, and future directions.IEEE Transactions on Intelligent Transportation Systems, 26(7):11088–11113, 2025. 10

    2025

  7. [15]

    A survey on LLM-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. A survey on LLM-as-a-judge. CoRR, abs/2411.15594, 2024

    2024 arXiv

  8. [16]

    Daniel Hein, Steffen Udluft, and Thomas A. Runkler. Interpretable policies for reinforcement learning by genetic programming.Eng. Appl. Artif. Intell., 76:158–169, 2018

    2018

  9. [17]

    Learning finite state representations of recurrent policy networks

    Anurag Koul, Alan Fern, and Sam Greydanus. Learning finite state representations of recurrent policy networks. InICLR. OpenReview.net, 2019

    2019

  10. [18]

    Explainable classification by learning human-readable sentences in feature subsets.Information Sciences, 564:202–219, 2021

    Prashanth Krishnamurthy, Alireza Sarmadi, and Farshad Khorrami. Explainable classification by learning human-readable sentences in feature subsets.Information Sciences, 564:202–219, 2021

    2021

  11. [19]

    Lillicrapet al.Continuous control with deep reinforcement learning

    Timothy P. Lillicrapet al.Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun, editors,4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016

    2016

  12. [20]

    Alqahtani, and Dongwon Lee

    Joe McCalmon, Thai Le, Sarra M. Alqahtani, and Dongwon Lee. CAPS: comprehensible abstract policy summaries for explaining reinforcement learning agents. InAAMAS 2022, pages 889–897. IFAAMAS, 2022

    2022

  13. [21]

    E. J. McCluskey. Minimization of Boolean functions.The Bell System Technical Journal, 35(6):1417–1444, 1956

    1956

  14. [22]

    Explainable reinforcement learning: A survey and comparative review.ACM Comput

    Stephanie Milani, Nicholay Topin, Manuela Veloso, and Fei Fang. Explainable reinforcement learning: A survey and comparative review.ACM Comput. Surv., 56(7):168:1–168:36, 2024

    2024

  15. [23]

    Kamhoua, Evangelos E

    Stephanie Milani, Zhicheng Zhang, Nicholay Topin, Zheyuan Ryan Shi, Charles A. Kamhoua, Evangelos E. Papalexakis, and Fei Fang. MA VIPER: learning decision tree policies for inter- pretable multi-agent reinforcement learning. InECML PKDD 2022, volume 13716 ofLNCS, pages 251–26...

    2022

  16. [24]

    V olodymyr Mnihet al.Human-level control through deep reinforcement learning.Nat., 518(7540):529–533, 2015

    2015

  17. [25]

    3 edition, 2025

    Christoph Molnar.Interpretable Machine Learning. 3 edition, 2025

    2025

  18. [26]

    Efficient memory-based learning for robot control

    Andrew Moore. Efficient memory-based learning for robot control. Technical report, Pittsburgh, PA, November 1990

    1990

  19. [27]

    Murthy, Simon Kasif, and Steven Salzberg

    Sreerama K. Murthy, Simon Kasif, and Steven Salzberg. A system for induction of oblique decision trees.J. Artif. Int. Res., 2(1):1–32, August 1994

    1994

  20. [28]

    Interpretable approximation of a deep reinforcement learning agent as a set of if-then rules

    Subramanya Nageshrao, Bruno Costa, and Dimitar Filev. Interpretable approximation of a deep reinforcement learning agent as a set of if-then rules. InICMLA 2019, pages 216–221, 2019

    2019

  21. [29]

    Paleja, Yaru Niu, Andrew Silva, Chace Ritchie, Sugju Choi, and Matthew C

    Rohan R. Paleja, Yaru Niu, Andrew Silva, Chace Ritchie, Sugju Choi, and Matthew C. Gom- bolay. Learning interpretable, high-performing policies for autonomous driving. InRobotics: Science and Systems XVIII, 2022, 2022

    2022

  22. [30]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python.Journal of Machine Learnin...

    2011

  23. [31]

    W. V . Quine. The problem of simplifying truth functions.The American Mathematical Monthly, 59(8):521–531, 1952

    1952

  24. [32]

    Ross Quinlan

    J. Ross Quinlan. Learning with continuous classes. InProceedings of Australian Joint Confer- ence on Artificial Intelligence, 1992

    1992

  25. [33]

    Stable-baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8, 2021

    Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8, 2021. 11

    2021

  26. [34]

    A reduction of imitation learning and structured prediction to no-regret online learning

    Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. InAISTATS 2011, volume 15 ofProceedings of Machine Learning Research, pages 627–635. PMLR, 11–13 Apr 2011

    2011

  27. [35]

    Multiple comparisons of treatment against control under unequal variances using parametric bootstrap.J

    Guoyi Zhang Sarah Alver. Multiple comparisons of treatment against control under unequal variances using parametric bootstrap.J. Appl. Stat., 51(10):1861–1877, 2023

    2023

  28. [36]

    Safety verification of decision-tree policies in continuous time

    Christian Schilling, Anna Lukina, Emir Demirovic, and Kim Guldstrand Larsen. Safety verification of decision-tree policies in continuous time. InNeurIPS 2023, 2023

    2023

  29. [37]

    Neurolinear: From neural networks to oblique decision rules

    Rudy Setiono and Huan Liu. Neurolinear: From neural networks to oblique decision rules. Neurocomputing, 17(1):1–24, 1997

    1997

  30. [38]

    Gombolay, Taylor W

    Andrew Silva, Matthew C. Gombolay, Taylor W. Killian, Ivan Dario Jimenez Jimenez, and Sung-Hyun Son. Optimization methods for interpretable differentiable decision trees applied to reinforcement learning. InAISTATS 2020, volume 108 ofProceedings of Machine Learning Research, p...

    2020

  31. [39]

    Ried- miller

    David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin A. Ried- miller. Deterministic policy gradient algorithms. InICML 2014, volume 32 ofJMLR Workshop and Conference Proceedings, pages 387–395. JMLR.org, 2014

    2014

  32. [40]

    David Silveret al.Mastering the game of Go with deep neural networks and tree search.Nat., 529(7587):484–489, 2016

    2016

  33. [41]

    TLdR: Policy summa- rization for factored SSP problems using temporal abstractions

    Sarath Sreedharan, Siddharth Srivastava, and Subbarao Kambhampati. TLdR: Policy summa- rization for factored SSP problems using temporal abstractions. InICAPS 2020, pages 272–280. AAAI Press, 2020

    2020

  34. [42]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018

    2018

  35. [43]

    Lopez-Miguel, Sebastian Tschiatschek, and Ezio Bartocci

    Martin Tappler, Ignacio D. Lopez-Miguel, Sebastian Tschiatschek, and Ezio Bartocci. Rule- guided reinforcement learning policy evaluation and improvement. InIJCAI 2025, pages 6254–6262. ijcai.org, 2025

    2025

  36. [44]

    Thorndike

    Robert L. Thorndike. Who belongs in the family?Psychometrika, 18(4):267–276, 1953

    1953

  37. [45]

    Generation of policy-level explanations for reinforcement learning

    Nicholay Topin and Manuela Veloso. Generation of policy-level explanations for reinforcement learning. InAAAI 2019, pages 2514–2521. AAAI Press, 2019

    2019

  38. [46]

    Llama 2: Open foundation and fine-tuned chat models.CoRR, abs/2307.09288, 2023

    Touvron et al. Llama 2: Open foundation and fine-tuned chat models.CoRR, abs/2307.09288, 2023

    2023 arXiv

  39. [47]

    CoRR, abs/2407.17032, 2024

    Mark Towerset al.Gymnasium: A standard interface for reinforcement learning environments. CoRR, abs/2407.17032, 2024

    2024 arXiv

  40. [48]

    Executive Office of the President

    U.S. Executive Office of the President. Safe, secure, and trustworthy development and use of artificial intelligence, 11 2023. U.S. Executive Order 14110

    2023

  41. [49]

    Self-interpretable reinforcement learning via rule ensembles

    Yue Yang, Fan Yang, Yu Bai, and Hao Wang. Self-interpretable reinforcement learning via rule ensembles. InAAMAS 2025, page 2235–2243. International Foundation for Autonomous Agents and Multiagent Systems, 2025. 12 Algorithm 2Dagger-like Rule Learning Loop 1:M j ←FitModel(M j−1...

    2025

  42. [50]

    The actions from this model are compared to the ones that the original RL agent would have taken in those states

    Episode collection and labeling: New environment trajectories are generated using the current model Mj. The actions from this model are compared to the ones that the original RL agent would have taken in those states. The actions from the RL agent are considered expert demonstrations

  43. [51]

    This refinement step is repeated until performance degrades or no more useful data can be collected

    Model refinement: The model is re-trained on an augmented dataset consisting of the original data plus the newly collected samples, where the action of the learned model deviates from the pre-trained RL agent. This refinement step is repeated until performance degrades or no m...

  44. [52]

    This places the focus on the states where the RL policy and the learned model differ the most

    Disagreement filtering:For each point in the new trajectory, the squared Euclidean distance between the learned action and the expert action is computed: d(ot) =∥a π t −a Mj t ∥2 2 Only observations where this distance is significant ( d >|A|/2 ) are kept, where |A| is the abs...

  45. [53]

    Clustering:The filtered points are clustered to reduce redundancy and represent different regions of the state space

  46. [54]

    comb-as & random-rs-nm

    Noise augmentation:Each cluster is used as a sample, to which Gaussian noise is added. This improves model robustness and potential model drift. A.1.3 Model fitting After adding the new data points to the training dataset, the model is fitted to this new dataset. The retrainin...

  47. [55]

    A user could enforce the mirrored rule that improves the RL policy

  48. [56]

    The following section shows how to understand the results with an example

    A user could generate demonstrations that satisfy the rule and retrain the RL model. The following section shows how to understand the results with an example. B.3 Explanation example for Swimmer Table 15 shows that weaknesses were often found with an ORCAIDmodel of depth 1. O...

  49. [57]

    Providing a graphical interface to set any valid state of the environment (Fig. 8)

  50. [58]

    Instantaneously determining the ORCAIDregion for the current state (Fig. 8)

  51. [59]

    Displaying the linear model that defines the policy within that region (Fig. 8)

  52. [60]

    Visually updating the environment’s rendering to reflect the new state (Fig. 7). C.2 Workflow and User Interaction The typical workflow for a user of this tool is as follows:

  53. [61]

    Initialization:The user runs the script selecting an ORCAIDmodel, which creates the environment and loads the model

  54. [62]

    Exploration:The user sees a window with the environment rendering and a separate control panel

  55. [63]

    What is the policy at the edge of this region?

    Manipulation:By moving the sliders forqposandqvel, the user can set the environment to any state. As they move a slider: (a) The environment updates. (b) The corresponding region of the current state for the ORCAIDupdates, showing the conditions and the linear model for that r...

  56. [64]

    3: Slight perturbations cause noticeable changes in the explanation or in the operating modes it captures, while preserving core decision logic

    Robustness: Is the explanation consistent and reliable across different input conditions, avoiding significant variations or failures when the model is subjected to slight perturbations in the environ- ment? Rubrics: 5: Explanation remains unchanged under slight perturbations....

  57. [65]

    3: The explanation reveals sources of bias, but fails to capture causes, e.g., resulting from the 21 environment

    Bias: Does the explanation account for and reveal potential biases in the RL agent’s decision- making, ensuring that it does not systematically favor or disadvantage certain actions? Rubrics: 5: The explanation reveals any bias in the RL agent’s decisions and provides insight ...

  58. [66]

    3: The explanation supports reasoning about either policies with deviating behavior or slight environment variations, but not both

    Transferability: Can the insights derived from the explanation be applied to similar RL models, different environments, or other tasks, ensuring generalizability beyond a single use case? Rubrics: 5: The explanation supports reasoning about policies with partially deviating be...

  59. [67]

    Human Comprehensibility: Is the explanation structured in a way that aligns with human cognitive abilities, avoiding excessive complexity while maintaining interpretability? Rubrics: 5: A user could trace what factors led to an action decision through a low number of intuitive...

  60. [68]

    3: Some factors leading to a decision are omitted and verification and validation requires substantial engineering

    Transparency: Does the explanation offer a clear and verifiable account of the RL agent’s decision-making process? Rubrics: 5: The explanation captures all decision-relevant details, and its complexity admits verification and validation. 3: Some factors leading to a decision a...

  61. [69]

    3: The explanation contains relevant details, but also details that are not necessary

    Selective: Does the explanation focus on the most relevant and influential factors affecting the agent’s decision, avoiding unnecessary details that may overwhelm the user? Rubrics: 5: The explanation includes mostly relevant details. 3: The explanation contains relevant detai...

  62. [70]

    3: The explanation provides some insight, but modifications require familiarity with the control policy

    Facilitating Actions: Does the explanation provide actionable insights that enable the user to modify inputs, adjust policies, or intervene effectively based on the RL system’s decision-making process? 5: The explanation provides sufficient insight for a user familiar with the...

  63. [71]

    3: Some partitions reflect expected operating modes, while some are arbitrary

    Physical coherence: Does the model’s partitioning of the state space align with the system’s meaningful physical regimes or modes? Or are the regions arbitrary, non-intuitive, and too small from a physics perspective? Rubrics: 5: The partitions correspond to operating modes ex...

  64. [72]

    3: Some regions have partial semantic meaning, but others are small, fragmented, or unrelated to the environment’s dynamics

    Granularity and partitioning: How well does the model structure the state space into meaningful regions? Are these regions too large or too small? Are there any arbitrary or poorly defined 22 regions? Is there a risk of overfitting due to excessive partitioning? Rubrics: 5: Re...

  65. [73]

    3: Users need to consider various regions and alternative decisions to trace failures, and failure causes cannot be unambiguously deduced

    Debugging: After a failure, how easy would it be to diagnose the problem? Does the model’s structure help to identify a certain physical regime, or does it give you a rule that failed in a particular small region of the state space? Rubrics: 5: Failures can be traced to decisi...

  66. [74]

    3: Users knowing the environment understand the aspects of system dynamics that prompted the definition of several regions, while others do not correspond to system dynamics

    Interpretability of Regions: Do regions correspond to physical behaviors? Is it easy to interpret what each region represents in terms of system dynamics? Rubrics: 5: Users knowing the environment understand the aspects of system dynamics that prompted the definition of most r...

  67. [75]

    They balance granularity and interpretability while maintaining clear semantic links to the envi- ronment’s dynamics

    Global Interpretability: Is the high-level decision-making intuitive, or is it obscured by com- plexity? Rubrics: 5: Regions are appropriately sized to capture complex, meaningful behaviors without overfitting. They balance granularity and interpretability while maintaining cl...

  68. [76]

    Regions capture essential dynamics without redundancy, balancing simplicity and completeness

    Minimal Granularity: Is the number of regions justified by the system’s behavior and the environment’s complexity? Are there too many regions, leading to unnecessary complexity, excessive noise, and a failure to capture the essential dynamics? Rubrics: 5: The number of regions...

  69. [77]

    Parameters are interpretable and contribute to clear, concise explanations

    Simplicity: Is the number of parameters justified, or are there too many, making it unnecessarily harder to understand? 23 Rubrics: 5: The number of parameters is minimal and well justified, directly aligning with the environment’s state-space structure and action prediction n...

  70. [78]

    rel. improvement

    Verdict: Declare the winning model. Base your conclusion on the previous answers and on the principles of control theory and robotics. An explainable controller should be understandable, trustworthy, and verifiable. Coherence, stability, and adaptability of the global policy a...

    2019

Pith tools

Reviewed July 9, 2026 · model on record in the stance chip above.