Pith. sign in

REVIEW 4 major objections 5 minor 111 references

High-level robotic manipulation policies can be expressed as decision trees over human-interpretable concept values — object states and spatial relations — so every skill choice is traceable to a short path of semantic predicates, and a sin

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

ConceptTree trains a decision tree over VLM-supervised, human-interpretable visual concepts to select robotic manipulation skills, enabling traceable and intervenable high-level decisions.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection Useful interpretable-policy assembly, but the semantic transparency claim rests on an unmeasured label-propagation shortcut; worth reviewing, needs fixes. the 4 major comments →

arxiv 2607.17861 v1 pith:WFRL7PE5 submitted 2026-07-20 cs.RO cs.AI

ConceptTree: Bringing Semantic Transparency to Black-Box Decision Making for Robotic Manipulation

Yongyan Wen , Feifan Liu , Jinyi Chen , Bo An , Peng Liu , Siyuan Li This is my paper
classification cs.RO cs.AI MSC 68T40
keywords concept bottleneck modelsinterpretable decision makingrobotic manipulationdecision treeslong-horizon tasksvision-language modelsconcept intervention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ConceptTree claims that the high-level part of robotic manipulation — choosing which skill to execute next — does not have to be a black box. It learns a small set of human-understandable concepts, such as 'bread inside microwave' or 'fridge door open', from visual observations, then selects skills using a decision tree over those concept values. This makes each decision an inspectable chain of predicates and allows errors to be corrected by editing one concept value rather than retraining. The paper reports that this approach outperforms existing concept-based baselines and black-box VLM policies on four real long-horizon tasks, with the largest gains on the most complex tasks.

Core claim

On the paper's own terms, the central claim is that representing a high-level policy as a sequence of concept-level predicates over visual observations — learned with VLM-generated binary labels and organized into a decision tree — yields decisions that are both transparent and reliably better than latent or directly-queried black-box alternatives. The learned continuous concept values act as a semantic interface: the decision tree's splits are thresholds on single concepts, so each prediction reduces to a conjunction of a few scene predicates. The paper demonstrates that an error can be located by finding where the decision path diverges from the nearest correct path, and that flipping one

What carries the argument

A two-stage policy. Stage one is a concept layer: a fixed visual encoder followed by a learned projection with per-dimension sigmoid calibration, trained by binary cross-entropy against VLM-generated concept labels (complemented by a skill-aware loss that keeps the concept space aligned with decisions). Stage two is an axis-aligned decision tree whose internal nodes test whether a single concept value falls below a threshold; the tree is trained on the learned concept vectors (optionally concatenated with the previous step's vector). The concept space is the load-bearing object: it is the semantic interface that makes the tree's decisions human-readable and intervenable, and it transfers VLM

Load-bearing premise

The load-bearing premise is that one VLM binary judgment on the first frame of each 50-frame skill window remains valid for the entire window; if a concept flips mid-skill, the learned concept values average over a wrong label and the decision tree inherits the error.

What would settle it

Take the Heat-Bread task and annotate every frame with the VLM instead of only the first frame of each window. Compare completion rate and concept accuracy. If, for episodes where 'bread inside microwave' changes during the move skill, the per-frame-trained ConceptTree recovers the correct close(microwave) decision that first-frame-trained ConceptTree misses, the central mechanism is supported; if not, the error lies elsewhere.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Each skill prediction becomes a short chain of concept predicates (typically 3-5), so failure diagnosis reduces to finding the first false predicate.
  • Wrong decisions can be corrected in the field by editing the identified concept value instead of collecting new data and retraining.
  • The VLM is needed only at dataset-annotation time, so the deployed policy is cheap and fast while inheriting controllable semantics.
  • Tree-structured reasoning over concepts beats a sparse linear head on the same concept representation, especially on temporally dependent and visually ambiguous tasks.
  • Stronger concept supervision yields smaller trees and shorter decision paths, so investing in accurate semantics pays off in both accuracy and interpretability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the policy is a literal decision tree over bounded concept values, it is a candidate for formal verification: safety properties (e.g., 'never pour when cup is empty') could be checked by exhaustive path inspection.
  • The method suggests a debugging protocol for robot failures that is close to code: read the path, edit the value, re-run — an implicit roadmap for explainable error correction in embodied systems.
  • The first-frame annotation shortcut opens a testable extension: generate per-frame VLM labels at low frequency but use interpolation or change-point detection to see if mid-skill concept drift explains the residual errors on the hardest task.
  • The fixed concept set bottleneck implies the next problem is automatic concept-set expansion from task descriptions; a failure mode to watch is concepts that are visually ambiguous by design.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ConceptTree, a framework for interpretable high-level skill selection in robotic manipulation. The approach trains a concept bottleneck layer with VLM-generated binary concept labels, then fits an axis-aligned decision tree over the learned concept values to predict skills. The authors evaluate on four real Franka Emika Panda manipulation tasks, comparing against a VLM policy, two concept-bottleneck baselines, and a sparse-linear variant (ConceptSLC) that shares the same concept layer. They report completion rates, ablation studies on supervision quality, temporal context, and tree depth, and qualitative case studies showing that changing a single concept value can redirect an incorrect tree prediction. The main claims are that ConceptTree consistently outperforms concept-based baselines, especially on harder tasks, and that its decisions are transparent and intervenable at the concept level.

Significance. If the central claims hold, ConceptTree would be a meaningful step toward interpretable high-level policies for long-horizon manipulation: it combines a concept bottleneck with a fully inspectable decision structure, and the intervention mechanism could support practical debugging without retraining. The real-robot evaluation against several baselines is a strength, as is the ablation isolating the decision module (ConceptSLC comparison). However, the paper's two central claims — consistent empirical superiority and semantic transparency — are not yet adequately supported. The largest gap is that concept-level correctness is never measured, and the VLM supervision itself is propagated from the first frame of each 50-frame skill window, a premise the paper acknowledges may be violated. The empirical 'consistently outperforms' claim is also weakened by the Fruits-Snacks result and the high variance on Coffee. These issues are fixable with additional analysis and experiments, so the contribution is potentially valuable but requires revision.

major comments (4)
  1. [§A.1, Eq. (4), Fig. 3] The label-propagation assumption is load-bearing but unvalidated. The paper states that VLM binary judgments are obtained only for the first observation in each 50-frame skill window and then assigned to all remaining frames, and it concedes that 'some later observations may occasionally deviate from the first frame in concept semantics.' For dynamic concepts like 'bread inside microwave' or 'microwave door open,' this can produce stale training targets at exactly the decision-relevant moments. No concept-level accuracy is reported, so we cannot know whether the learned concept layer is faithful to the true scene predicates. The ablation in Fig. 3 shows that supervision quality is the dominant factor in downstream performance, making this more than a peripheral detail. To support the central transparency/traceability claim, the paper should either collect per-frame VLM labels on a held-o
  2. [§4.2, Table 1] The abstract and Section 4.2 claim that ConceptTree 'consistently outperforms' existing concept-based baselines, but Table 1 does not support an unqualified statement. On Fruits-Snacks w/ history, ConceptSLC achieves 100.00±0.00 while ConceptTree is 94.92±6.22; on Coffee, ConceptTree's 37.09±17.36 substantially overlaps ConceptSLC's 18.18±6.08 given the reported standard deviations. The authors acknowledge the Fruits-Snacks exception but still use 'consistently' in the abstract. The paper should report statistical significance tests (e.g., paired tests across seeds/episodes) and soften the claim to reflect the actual pattern: strong gains on Heat-Bread and Cola, mixed on Fruits-Snacks, and high-variance on Coffee. Additionally, Appendix C shows that the skill-aware loss weight λ is tuned per task (λ=0.1 for three tasks, λ=0.5 for Coffee), but it is not stated whether the baselines receiv
  3. [§4.4, Appendix G] The intervention case studies demonstrate that changing a concept value to the correct side of a threshold redirects the decision-tree prediction. This is a tautological property of axis-aligned trees: if you set the value so that the branch changes, the prediction changes accordingly. The paper's actual contribution is the path-based diagnosis procedure in Appendix G (finding the deepest node shared with a correct path), and this is only illustrated on a few hand-picked examples. The claim that errors can be 'corrected by modifying individual concepts without retraining' needs quantitative support: how often does the identified concept correspond to the true error source, and how often does the intervention actually fix the decision? A systematic evaluation across all test errors (or a larger sample) is needed; otherwise the intervention claim rests on anecdotal evidence.
  4. [§4.3, Fig. 3] The 'w/ CLIP target' ablation is used to argue that concept supervision quality is critical, and the reader is invited to infer that the paper's first-frame propagation is therefore adequate. But replacing VLM supervision with CLIP similarity changes many factors at once; it does not isolate the effect of label propagation. A direct ablation would compare training with first-frame-only propagated labels versus per-frame VLM labels (or at least labels sampled from more frames per window). Without such an experiment, the paper's statement in Appendix A.1 that 'this approximation to work well in practice' is an unsupported assertion. The correlation between better supervision and simpler trees (Fig. 4) is also reported descriptively; the causal direction is unclear.
minor comments (5)
  1. [Abstract ] The phrase 'consistently outperforms' is contradicted by Table 1 itself; suggest 'generally outperforms' or a similar qualified statement.
  2. [Eq. (1)] The piecewise definition of c̃_t is rendered awkwardly on the page; use a clear piecewise/cases format.
  3. [Table 4 (Appendix C) vs Section 4.2] Table 4 reports the 'best depth' for each λ, while Section 4.2 says a fixed maximum depth of 7 is used for all tasks. Clarify which depth values are used in the main results of Table 1, and whether the best-depth values in Appendix C are used or only reported.
  4. [Fig. 1] The block labeled 'CBL' is not defined in text; either define it or expand the label to 'Concept Projection Layer'.
  5. [§4.3, Fig. 3] The x-axis label says 'Depth' but the caption notes it is the maximum allowed depth during training. Consider relabeling to 'Maximum depth' to avoid ambiguity.

Circularity Check

1 steps flagged

Empirical comparisons are independent, but the intervention-correction demonstration is tautological: editing a concept to the correct side of a split forces the tree's prediction by construction.

specific steps
  1. self definitional [Section 3.3 (Eq. 7) and Appendix G; Fig. 5 case study]
    "Given a modified concept vector ˜c′_t, the decision outcome can be directly recomputed as a′_t = g(˜c′_t), allowing one to analyze how changes in specific concepts affect the final decision. [...] We then edit only this concept value, move it to the semantically correct side of the split, and recompute the tree prediction without changing model parameters."

    The corrected skill is selected, not discovered. Appendix G chooses the first predicate where the erroneous path diverges from a path predicting the ground-truth skill and edits the concept to 'the semantically correct side of the split.' In an axis-aligned decision tree (Eq. 7: d_n(˜c_t)=I[˜c_t^(k)≤τ_n]), crossing any threshold flips the branch deterministically, so the recomputed prediction is forced by the construction. The demo therefore shows decision-tree determinism, not independent evidence that the concept is semantically grounded or that this concept caused the error.

full rationale

The main empirical derivation is not circular. Concept supervision comes from VLM binary judgments (Eq. 2), the concept layer is trained with a BCE loss (Eq. 4) plus an auxiliary skill loss (Eq. 5), and the decision tree is trained on the learned concept vectors against ground-truth skill labels (Eqs. 7-8). Completion rates are measured on held-out physical episodes using a first-error protocol (Eq. 9), so the central performance claim is not obtained by renaming fitted inputs as predictions. The self-citation to SkillTree [39] is contextual only and carries no load-bearing uniqueness or ansatz. The one tautological element is the intervention case study: because the paper explicitly identifies the divergent predicate and moves the concept to the side of the split that leads to the correct path, the resulting 'correction' is guaranteed by the tree's construction rather than independently validated. Appendix A.1's admitted first-frame label propagation and the Conclusion's caveat that concept quality depends on VLM supervision are genuine limitations on semantic fidelity, but they are data-quality concerns, not circular reductions. Overall, the framework has independent content, with a minor self-definitional demonstration in the intervention claim.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claims rest on the validity of VLM-generated concept labels, the expressiveness of hand-selected binary concepts, the 50-frame label propagation, and the reliability of low-level skills. There are no new physical entities. Free parameters: per-concept calibration (p_k, q_k), skill-loss weight λ (per task), tree depth d, and the concept sets themselves.

free parameters (4)
  • per-concept calibration (p_k, q_k) = learned
    Eq. 3: per-dimension sigmoid scale and shift fitted by the BCE loss; concept values used by the tree depend on these parameters.
  • skill-aware loss weight λ = 0.1 for Fruits-Snacks, Heat-Bread, Cola; 0.5 for Coffee
    Chosen per task from {0, 0.05, 0.1, 0.2, 0.5, 1.0} based on validation completion rate (Appendix C, Table 4). It changes the concept representation and downstream tree.
  • maximum tree depth d = 7 (all tasks)
    Selected after ablation as a balance between performance and interpretability (§4.3). Coffee's best validation depth is 10 (Table 4), so the global choice is consequential.
  • task-specific concept set C = M = 17, 11, 14, 21 for the four tasks
    Generated by LLM prompt and manually filtered (Appendix A.1); concept sets are hand-selected and shared with concept-based baselines.
axioms (5)
  • domain assumption VLM binary judgments y_t(c^(k)) are reliable and consistent proxies for concept presence.
    Used throughout §3.1 and Appendix A.2; the Conclusion acknowledges VLM supervision may be unreliable for subtle visual states.
  • ad hoc to paper First-frame VLM labels can be propagated across the remaining 49 frames of each skill window.
    Appendix A.1: "we request VLM annotations only for the first observation in each 50-frame window and assign the resulting concept labels to the remaining observations"; paper admits occasional deviation.
  • domain assumption Task-relevant concepts are expressible as binary predicates observable from a single side-camera image.
    Appendix A concept construction prompt requires concepts to be "visually observable from a single image."
  • domain assumption Low-level skills succeed with high probability and do not confound high-level evaluation.
    Appendix H.3 reports >90% skill success with HIL-SERL; the completion-rate metric credits high-level decisions, so low-level failures could be misattributed.
  • domain assumption The fixed expert skill sequences are the correct ground truth for all evaluation episodes.
    §4.1 and Appendix H.2; all methods are evaluated against the same fixed sequences.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of ConceptTree: Bringing Semantic Transparency to Black-Box Decision Making for Robotic Manipulation." pith.science (2026). https://pith.science/paper/WFRL7PE5

@misc{pith2026260717861,
  author       = {Pith},
  title        = {Pith review of: ConceptTree: Bringing Semantic Transparency to Black-Box Decision Making for Robotic Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WFRL7PE5}},
  note         = {Machine review of arXiv:2607.17861}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Establishing interpretable decision-making processes in long-horizon robotic manipulation is critical for enabling reliable human oversight and intervention. However, existing approaches to robotic manipulation largely treat skill selection as opaque mappings from observations to actions, offering limited transparency into how decisions are formed. In this work, we propose ConceptTree, a framework that reframes high-level manipulation skill selection as reasoning over human-interpretable concepts, representing high-level policies as a sequence of concept-level predicates over visual observations. Rather than relying on implicit latent representations, our method learns a normalized concept space grounded in visual inputs, over which a decision tree is trained to predict high-level skills. This formulation yields a transparent decision process that is both traceable and intervenable, enabling direct inspection and modification of policy behavior. We evaluate our approach on a set of real-world robotic manipulation tasks with increasing complexity. Experimental results show that ConceptTree consistently outperforms existing concept-based baselines, particularly in complex, long-horizon scenarios. Furthermore, we provide qualitative case studies showing that our model supports fine-grained intervention by modifying individual concepts, enabling targeted correction of decision errors without retraining.

Figures

Figures reproduced from arXiv: 2607.17861 by Bo An, Feifan Liu, Jinyi Chen, Peng Liu, Siyuan Li, Yongyan Wen.

Figure 1
Figure 1. Figure 1: Overview of the ConceptTree framework. The model consists of three main components: (i) a visual encoder that extracts features from observations, (ii) a concept projection layer that maps features to a structured concept representation, and (iii) a decision tree that predicts high-level skills based on the concept vector. During training, a VLM provides binary concept judgments for supervision, while at i… view at source ↗
Figure 2
Figure 2. Figure 2: Tasks evaluated in our study, including storing fruits and snacks, heating and preparing the bread, preparing a cup of ice cola and storing the milk, manual coffee brewing, with example observations. Interpretability and Intervention. In the formulation, each prediction can be traced to a sequence of concept-level predicates, providing a transparent explanation of the decision. Furthermore, the explicit st… view at source ↗
Figure 3
Figure 3. Figure 3: Ablation study across four tasks. Bars indicate standard deviation. Depth denotes the maximum allowed tree depth during training, not the actual depth. 5 10 15 Used Concepts Fruits-Snacks 5 10 Heat-Bread 5 10 Cola 10 20 Coffee 0 200 Internal Nodes 0 200 400 0 200 400 0 200 4 6 8 10 Depth 5.0 7.5 Path Length 4 6 8 10 Depth 5.0 7.5 4 6 8 10 Depth 5.0 7.5 10.0 4 6 8 10 Depth 4 6 8 w/ CLIP target w/o skill los… view at source ↗
Figure 4
Figure 4. Figure 4 [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Decision-path diagnosis and concept intervention. In this Heat-Bread example, the original path predicts open(microwave) because the value of bread inside microwave falls below the split threshold. This is inconsistent with the observation, where the bread is already inside the microwave, so the concept is falsely treated as negative. Manually correcting this single concept value redirects the tree travers… view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of concept importance in decision paths across tasks. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Visualization of decision tree for the Fruits-Snacks task. microwave_door_open (previous)@0.26 object_inside_microwave@0.00 microwave_door_open@1.00 microwave_door_open (previous)@1.00 object_on_counter@1.00 bread_inside_microwave@0.58 microwave_door_open@0.98 microwave_door_open@0.26 object_inside_microwave@0.65 object_inside_microwave@0.02 object_inside_microwave (previous)@0.33 ≤ > [PITH_FULL_IMAGE:fig… view at source ↗
Figure 8
Figure 8. Figure 8: Visualization of decision tree for the Heat-Bread task. F Decision Tree vs. Sparse Linear Model We further compare the interpretability of a single decision tree and a sparse linear classifier [40] when both operate on the same task-specific concept representation for a fair comparison. For the tree model, we use the corresponding ConceptTree model and report complexity statistics at maximum depth d = 7. F… view at source ↗
Figure 9
Figure 9. Figure 9: Visualization of decision tree for the Cola task [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Visualization of decision tree for the Coffee task. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Visualization of top-8 concept weights in sparse linear models across tasks. G Additional Intervention Case Study This section describes how we identify the concept to intervene on when ConceptTree predicts an incorrect skill. Rather than inspecting all concepts, we use the tree structure to find the first predicate where the erroneous decision path diverges from a nearby path that predicts the correct sk… view at source ↗
Figure 12
Figure 12. Figure 12: Visualization of the intervention case study on Fruits-Snacks. Previous Observation Current Observation Ground Truth move(cola, bin) 0.0 0.5 1.0 Value Concept fridge open 0.00 can on surface 0.15 -> 1.00 0.15 cup contains liquid 1.00 Original Decision Path 1 fridge open: 0.001 <= 0.453 | left branch 2 fridge open (previous): 0.001 <= 0.538 | left branch 3 cup contains liquid (previous): 0.981 > 0.814 | ri… view at source ↗
Figure 13
Figure 13. Figure 13: Visualization of the intervention case study on Cola. Intervention target. Suppose the nearest divergent parent node tests dimension k of c˜t with threshold τ . If ρerr and ρnear follow different branches at this node, then dimension k is the first semantic variable that prevents the current prediction from reaching the nearest correct path. We use the corresponding concept as the intervention target. In … view at source ↗
Figure 14
Figure 14. Figure 14: Visualization of the intervention case study on Coffee. H.1 Robot Platform and Low-level Skills All experiments are conducted on a physical Franka Emika Panda 7-DOF robot arm with a stationary side camera that provides a global view of the workspace. The high-level policy selects from a predefined library of discrete skills, and each selected skill is executed by a pretrained low-level controller. These l… view at source ↗
Figure 15
Figure 15. Figure 15: Illustrations of the tasks in the experiments and skill sequences. window captures the visual conditions under which the high-level skill choice is made, while avoiding later frames dominated by low-level execution transients, intermediate object motions, and post-action states that can deviate from the original decision context. In practice, this selective sampling increases visual diversity across skill… view at source ↗
Figure 16
Figure 16. Figure 16: Visualization of a test episode for the Fruits-Snacks. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Visualization of a test episode for the Heat-Bread with history input. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Visualization of a test episode for the Cola with history input. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_18.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

111 extracted references · 1 linked inside Pith

  1. [1]

    Towards a unified understanding of robot manipulation: A comprehensive survey.arXiv preprint arXiv:2510.10903, 2025

    Shuanghao Bai, Wenxuan Song, Jiayi Chen, Yuheng Ji, Zhide Zhong, Jin Yang, Han Zhao, Wanqi Zhou, Wei Zhao, Zhe Li, et al. Towards a unified understanding of robot manipulation: A comprehensive survey.arXiv preprint arXiv:2510.10903, 2025

    arXiv 2025

  2. [2]

    Verifiable reinforcement learning via policy extraction.Advances in Neural Information Processing Systems, 31, 2018

    Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction.Advances in Neural Information Processing Systems, 31, 2018

    2018

  3. [3]

    Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Manuel Y . Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren,...

    2025

  4. [4]

    Do as i can, not as i say: Grounding language in robotic affordances

    Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. InConference on robot learning, pages 287–318. PMLR, 2023

    2023

  5. [5]

    Imagenet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  6. [6]

    Yashesh Dhebar and Kalyanmoy Deb. Interpretable rule discovery through bilevel optimization of split-rules of nonlinear decision trees for classification problems.IEEE Transactions on Cybernetics, 51(11):5573–5584, 2020

    2020

  7. [7]

    Toward interpretable-ai policies using evolutionary nonlinear decision trees for discrete-action systems

    Yashesh Dhebar, Kalyanmoy Deb, Subramanya Nageshrao, Ling Zhu, and Dimitar Filev. Toward interpretable-ai policies using evolutionary nonlinear decision trees for discrete-action systems. IEEE Transactions on Cybernetics, 54(1):50–62, 2022

    2022

  8. [8]

    Task and motion planning with large language models for object rearrangement

    Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for object rearrangement. In2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2086–2092. IEEE, 2023

    2086

  9. [9]

    Video language planning

    Yilun Du, Sherry Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B Tenenbaum, Leslie Pack Kaelbling, et al. Video language planning. In The Twelfth International Conference on Learning Representations, 2024

    2024

  10. [10]

    A tale of two explanations: Enhancing human trust by explaining robot behavior.Science Robotics, 4(37):eaay4663, 2019

    Mark Edmonds, Feng Gao, Hangxin Liu, Xu Xie, Siyuan Qi, Brandon Rothrock, Yixin Zhu, Ying Nian Wu, Hongjing Lu, and Song-Chun Zhu. A tale of two explanations: Enhancing human trust by explaining robot behavior.Science Robotics, 4(37):eaay4663, 2019

    2019

  11. [11]

    A survey on interpretable reinforcement learning.Machine Learning, 113(8):5847–5890, 2024

    Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang, Jianye Hao, and Wulong Liu. A survey on interpretable reinforcement learning.Machine Learning, 113(8):5847–5890, 2024

    2024

  12. [12]

    Visualizing and understand- ing atari agents

    Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understand- ing atari agents. InInternational Conference on Machine Learning, pages 1792–1801. PMLR, 2018

    2018

  13. [13]

    Interpreting black-box models: a review on explainable artificial intelligence.Cognitive Computation, 16 (1):45–74, 2024

    Vikas Hassija, Vinay Chamola, Atmesh Mahapatra, Abhinandan Singal, Divyansh Goel, Kaizhu Huang, Simone Scardapane, Indro Spinelli, Mufti Mahmud, and Amir Hussain. Interpreting black-box models: a review on explainable artificial intelligence.Cognitive Computation, 16 (1):45–74, 2024. 10

    2024

  14. [14]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  15. [15]

    Semi-supervised concept bottleneck models

    Lijie Hu, Tianhao Huang, Huanyi Xie, Xilin Gong, Chenyang Ren, Zhengyu Hu, Lu Yu, Ping Ma, and Di Wang. Semi-supervised concept bottleneck models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2110–2119, 2025

    2025

  16. [16]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, et al. Openvla: An open-source vision-language-action model. InConference on Robot Learning, pages 2679–2713. PMLR, 2025

    2025

  17. [17]

    Concept bottleneck models

    Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. InInternational Conference on Machine Learning, pages 5338–5348. PMLR, 2020

    2020

  18. [18]

    Inter- pretable and editable programmatic tree policies for reinforcement learning

    Hector Kohler, Quentin Delfosse, Riad Akrour, Kristian Kersting, and Philippe Preux. Inter- pretable and editable programmatic tree policies for reinforcement learning. InWorkshop on Interpretable Policies in Reinforcement Learning@ RLC-2024, 2024

    2024

  19. [19]

    A review of robot learning for manipula- tion: Challenges, representations, and algorithms.Journal of Machine Learning Research, 22 (30):1–82, 2021

    Oliver Kroemer, Scott Niekum, and George Konidaris. A review of robot learning for manipula- tion: Challenges, representations, and algorithms.Journal of Machine Learning Research, 22 (30):1–82, 2021

    2021

  20. [20]

    Concept-based explanations in computer vision: Where are we and where could we go? In European Conference on Computer Vision, pages 266–287

    Jae Hee Lee, Georgii Mikriukov, Gesina Schwalbe, Stefan Wermter, and Diedrich Wolter. Concept-based explanations in computer vision: Where are we and where could we go? In European Conference on Computer Vision, pages 266–287. Springer, 2024

    2024

  21. [21]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuropean Conference on Computer Vision, pages 38–55. Springer, 2024

    2024

  22. [22]

    Hybrid concept bottleneck models

    Yang Liu, Tianwei Zhang, and Shi Gu. Hybrid concept bottleneck models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 20179–20189, 2025

    2025

  23. [23]

    Zhenyu Lu, Ning Wang, and Chenguang Yang. A dynamic movement primitives-based tool use skill learning and transfer framework for robot manipulation.IEEE Transactions on Automation Science and Engineering, 22:1748–1763, 2024

    2024

  24. [24]

    Precise and dexterous robotic manip- ulation via human-in-the-loop reinforcement learning.Science Robotics, 10(105):eads5033, 2025

    Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manip- ulation via human-in-the-loop reinforcement learning.Science Robotics, 10(105):eads5033, 2025

    2025

  25. [25]

    End-to-end neuro-symbolic reinforcement learning with textual explanations

    Lirui Luo, Guoxi Zhang, Hongming Xu, Yaodong Yang, Cong Fang, and Qing Li. End-to-end neuro-symbolic reinforcement learning with textual explanations. InInternational Conference on Machine Learning, pages 33533–33557. PMLR, 2024

    2024

  26. [26]

    Mitigating information loss in tree-based reinforcement learning via direct optimization

    Sascha Marton, Tim Grams, Florian V ogt, Stefan Lüdtke, Christian Bartelt, and Heiner Stucken- schmidt. Mitigating information loss in tree-based reinforcement learning via direct optimization. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  27. [27]

    Explainable reinforcement learning: A survey and comparative review.ACM Computing Surveys, 56(7):1–36, 2024

    Stephanie Milani, Nicholay Topin, Manuela Veloso, and Fei Fang. Explainable reinforcement learning: A survey and comparative review.ACM Computing Surveys, 56(7):1–36, 2024

    2024

  28. [28]

    Label-free concept bottleneck models

    Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  29. [29]

    Learning interpretable, high-performing policies for autonomous driving

    Rohan Paleja, Yaru Niu, Andrew Silva, Chace Ritchie, Sugju Choi, and Matthew Gombolay. Learning interpretable, high-performing policies for autonomous driving. InRobotics science and systems, 2022. 11

    2022

  30. [30]

    Scikit- learn: Machine learning in python.the Journal of machine Learning research, 12:2825–2830, 2011

    Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python.the Journal of machine Learning research, 12:2825–2830, 2011

    2011

  31. [31]

    Explain your move: Understanding agent actions using specific and relevant feature attribution

    Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krish- namurthy, and Sameer Singh. Explain your move: Understanding agent actions using specific and relevant feature attribution. InInternational Conference on Learning Representations, 2020

    2020

  32. [32]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  33. [33]

    Gen- erating explanations for autonomous robots: a systematic review.IEEE Access, 2025

    David Sobrín-Hidalgo, Ángel Manuel Guerrero-Higueras, and Vicente Matellán-Olivera. Gen- erating explanations for autonomous robots: a systematic review.IEEE Access, 2025

    2025

  34. [34]

    Llm-planner: Few-shot grounded planning for embodied agents with large language models

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2998–3009, 2023

    2023

  35. [35]

    Vlg-cbm: Training concept bottleneck models with vision-language guidance.Advances in Neural Information Processing Systems, 37:79057–79094, 2024

    Divyansh Srivastava, Ge Yan, and Tsui-Wei Weng. Vlg-cbm: Training concept bottleneck models with vision-language guidance.Advances in Neural Information Processing Systems, 37:79057–79094, 2024

    2024

  36. [36]

    Adaptive neural trees

    Ryutaro Tanno, Kai Arulkumaran, Daniel Alexander, Antonio Criminisi, and Aditya Nori. Adaptive neural trees. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors,Proceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 6166–6175. PMLR, 09–15 Jun 2019

    2019

  37. [37]

    Piecewise linear parametrization of policies: Towards interpretable deep reinforcement learning

    Maxime Wabartha and Joelle Pineau. Piecewise linear parametrization of policies: Towards interpretable deep reinforcement learning. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  38. [38]

    Nbdt: Neural-backed decision tree

    Alvin Wan, Lisa Dunlap, Daniel Ho, Jihan Yin, Scott Lee, Suzanne Petryk, Sarah Adel Bargal, and Joseph E Gonzalez. Nbdt: Neural-backed decision tree. InInternational Conference on Learning Representations, 2021

    2021

  39. [39]

    Skilltree: Explainable skill-based deep reinforcement learning for long-horizon control tasks

    Yongyan Wen, Siyuan Li, Rongchang Zuo, Lei Yuan, Hangyu Mao, and Peng Liu. Skilltree: Explainable skill-based deep reinforcement learning for long-horizon control tasks. InPro- ceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 21491–21500, 2025

    2025

  40. [40]

    Leveraging sparse linear layers for debuggable deep networks

    Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. InInternational Conference on Machine Learning, pages 11205– 11216. PMLR, 2021

    2021

  41. [41]

    Embodied task planning with large language models.arXiv preprint arXiv:2307.01848, 2023

    Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan. Embodied task planning with large language models.arXiv preprint arXiv:2307.01848, 2023

    Pith/arXiv arXiv 2023

  42. [42]

    Post-hoc concept bottleneck models

    Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  43. [43]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. InConference on Robot Learning, pages 2165–2183. PMLR, 2023. 12 A Concept Set Construction To construct a task-specific concept set, we query an LL...

    2023

  44. [44]

    Visibility Format: <object>_visible

  45. [45]

    Spatial Relation (object-object or object-location) Format: <object>_on_surface, <object>_on_counter, <object>_on_plate, <object>_inside_< container>, <object>_above_<object> Examples: cup_on_surface, funnel_above_pot, milk_bottle_inside_fridge

  46. [46]

    Container State (binary content state) Format: <object>_empty, <object>_contains_liquid, <object>_contains_dark_liquid Examples: cup_empty, coffee_cup_contains_liquid, pot_contains_liquid

  47. [47]

    Object State / Configuration Format: <object>_upright, <object>_tilted Examples: gooseneck_kettle_upright, gooseneck_kettle_tilted

  48. [48]

    Interaction Proxy (only if visually obvious) Format: <object>_pouring_to_<object> Example: gooseneck_kettle_pouring_to_funnel

  49. [49]

    Appliance State Format: <appliance>_door_open Examples: fridge_door_open, cabinet_door_open, microwave_door_open

  50. [50]

    Generic Object Group (category abstraction) Format: any_object_on_counter, any_object_inside_fridge, any_object_inside_cabinet, object_inside_<container>, object_on_<location>

  51. [51]

    Shape / Appearance Category (only predefined categories) Format: round_<color>_object_on_counter, elongated_<color>_object_on_counter, small_< color>_box_on_counter 13 Examples: round_orange_object_on_counter, elongated_red_object_on_counter, small_light_box_on_counter Output Requirements: - Return ALL concepts following the formats above - Do NOT include...

  52. [52]

    on_surface

    A list of concepts. Your task: For EACH concept, determine whether it is TRUE (1) or FALSE (0) based ONLY on what is clearly visible in the image. IMPORTANT RULES: - Only use visual evidence from the image. - Do NOT guess or assume anything not clearly visible. - If uncertain, output 0. - Be strict and conservative. - Each concept must be evaluated indepe...

  53. [53]

    fridge door open = 0.003 <= 0.504

  54. [54]

    small light box on counter = 1.000 > 0.514

  55. [55]

    fridge door open = 0.994 > 0.504

    round orange object on counter = 0.999 > 0.510 Predict: open(fridge) Decision Path Skill 0 Observation Ground Truth: open(fridge) 0.0 0.5 1.0 Value Concept any object inside fridge 0.01 cabinet door open 0.67 fridge door open 0.99 round yellow object on counter 1.00 round orange object on counter 1.00 small light box on counter 1.00 1. fridge door open = ...

  56. [56]

    round yellow object on counter = 0.999 > 0.459

  57. [57]

    fridge door open = 0.999 > 0.504

    any object inside fridge = 0.007 <= 0.495 Predict: move(mango, fridge) Skill 1 Observation Ground Truth: move(mango, fridge) 0.0 0.5 1.0 Value Concept cabinet door open 0.42 any object inside fridge 0.95 round yellow object on counter 0.99 fridge door open 1.00 round orange object on counter 1.00 small light box on counter 1.00 1. fridge door open = 0.999 > 0.504

  58. [58]

    round yellow object on counter = 0.990 > 0.459

  59. [59]

    fridge door open = 0.999 > 0.504

    any object inside fridge = 0.952 > 0.495 Predict: move(lemon, fridge) Skill 2 Observation Ground Truth: move(lemon, fridge) 0.0 0.5 1.0 Value Concept round yellow object on counter 0.02 cabinet door open 0.37 round orange object on counter 0.95 any object inside fridge 1.00 fridge door open 1.00 small light box on counter 1.00 1. fridge door open = 0.999 > 0.504

  60. [60]

    round yellow object on counter = 0.022 <= 0.459

  61. [61]

    fridge door open = 0.999 > 0.504

    round orange object on counter = 0.953 > 0.511 Predict: move(orange, fridge) Skill 3 Observation Ground Truth: move(orange, fridge) 0.0 0.5 1.0 Value Concept round yellow object on counter 0.00 round orange object on counter 0.01 cabinet door open 0.73 any object inside fridge 0.99 fridge door open 1.00 small light box on counter 1.00 1. fridge door open ...

  62. [62]

    round yellow object on counter = 0.004 <= 0.459

  63. [63]

    fridge door open = 0.002 <= 0.504

    round orange object on counter = 0.006 <= 0.511 Predict: close(fridge) Skill 4 Observation Ground Truth: close(fridge) 0.0 0.5 1.0 Value Concept any object inside fridge 0.00 fridge door open 0.00 round orange object on counter 0.00 round yellow object on counter 0.01 cabinet door open 0.05 small light box on counter 0.99 1. fridge door open = 0.002 <= 0.504

  64. [64]

    small light box on counter = 0.991 > 0.514

  65. [65]

    round orange object on counter = 0.003 <= 0.510

  66. [66]

    fridge door open = 0.003 <= 0.504

    cabinet door open = 0.045 <= 0.294 Predict: open(cabinet) Skill 5 Observation Ground Truth: open(cabinet) 0.0 0.5 1.0 Value Concept round orange object on counter 0.00 any object inside fridge 0.00 round yellow object on counter 0.00 fridge door open 0.00 cabinet door open 0.33 small light box on counter 0.99 1. fridge door open = 0.003 <= 0.504

  67. [67]

    small light box on counter = 0.994 > 0.514

  68. [68]

    round orange object on counter = 0.001 <= 0.510

  69. [69]

    fridge door open = 0.000 <= 0.504

    cabinet door open = 0.330 > 0.294 Predict: move(cheezit, cabinet) Skill 6 Observation Ground Truth: move(cheezit, cabinet) 0.0 0.5 1.0 Value Concept round yellow object on counter 0.00 round orange object on counter 0.00 fridge door open 0.00 any object inside fridge 0.00 small light box on counter 0.03 cabinet door open 0.34 1. fridge door open = 0.000 <= 0.504

  70. [70]

    small light box on counter = 0.030 <= 0.514

  71. [71]

    fridge door open = 0.000 <= 0.504

    cabinet door open = 0.343 <= 0.525 Predict: move(jello, cabinet) Skill 7 Observation Ground Truth: move(jello, cabinet) 0.0 0.5 1.0 Value Concept round orange object on counter 0.00 round yellow object on counter 0.00 fridge door open 0.00 any object inside fridge 0.00 small light box on counter 0.01 cabinet door open 0.70 1. fridge door open = 0.000 <= 0.504

  72. [72]

    small light box on counter = 0.013 <= 0.514

  73. [73]

    25 Table 6:Task settings used in the experiments.For each task, we list the predefined skill library and the expert execution sequence used for high-level decision evaluation

    cabinet door open = 0.697 > 0.525 Predict: close(cabinet) Skill 8 Observation Ground Truth: close(cabinet) Figure 16:Visualization of a test episode for theFruits-Snacks. 25 Table 6:Task settings used in the experiments.For each task, we list the predefined skill library and the expert execution sequence used for high-level decision evaluation. Task Skill...

  74. [74]

    object inside microwave (previous) = 0.000 <= 0.331

  75. [75]

    bread inside microwave = 0.000 <= 0.578

  76. [76]

    microwave door open = 0.985 <= 0.999

  77. [77]

    object inside microwave (previous) = 0.000 <= 0.331

    microwave door open (previous) = 0.985 > 0.262 Predict: open(microwave) Decision Path Skill 0 Previous Observation Current Observation Ground Truth: open(microwave) 0.0 0.5 1.0 Value Concept bread inside microwave 0.01 object inside microwave 0.04 microwave door open 1.00 object on counter 1.00 1. object inside microwave (previous) = 0.000 <= 0.331

  78. [78]

    bread inside microwave = 0.010 <= 0.578

  79. [79]

    microwave door open = 1.000 > 0.999

  80. [80]

    object inside microwave (previous) = 0.035 <= 0.331

    object inside microwave = 0.035 > 0.001 Predict: move(bread, microwave) Skill 1 Previous Observation Current Observation Ground Truth: move(bread, microwave) 0.0 0.5 1.0 Value Concept object inside microwave 0.97 bread inside microwave 0.99 microwave door open 0.99 object on counter 1.00 1. object inside microwave (previous) = 0.035 <= 0.331

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.