{"id":"730c0a56-7ac9-44cc-95e5-b14cf8fc2643","arxiv_id":"2607.17861","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ConceptTree trains a decision tree over VLM-supervised, human-interpretable visual concepts to select robotic manipulation skills, enabling traceable and intervenable high-level decisions.","lead":"ConceptTree makes a robot's high-level skill choices easier to understand by turning camera images into simple concepts, such as \"fridge door open\" or \"yellow object on counter,\" and then using a decision tree over those concepts to choose the next skill. The method reports better task completion than concept-based baselines on four real Franka Panda manipulation tasks, and demonstrates that editing one wrong concept value can correct a bad decision without retraining.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ConceptTree's transparency and performance claims rest on an unmeasured label-propagation shortcut: one VLM judgment per 50-frame skill window is reused for frames whose scene state may have changed, and no concept-level accuracy is reported.","rationale":"The reader's weakest assumption is exactly the label propagation; I agree. Why it is the load-bearing one: the framework's distinctive contribution is semantic transparency, so tree paths are meaningful only if concept values are accurate. The paper's own ablation (Fig. 3) says concept-supervision quality dominates, and the limitations section admits VLM supervision can be unreliable. There is no direct measurement of concept accuracy, and the first-frame propagation is an explicit design choice with an admitted failure mode. No code/data are released, so external checking is impossible. I do not think this warrants rejection: it is addressable by dense annotation and retraining, and the qualitative case studies provide some support. But it does warrant keeping the verdict CONDITIONAL rather than ACCEPT. Other issues (Table 1 inconsistency on Fruits-Snacks w/H, wide error bars, depth/lambda selected after validation) are real but less fundamental; they affect the strength of \"consistently outperforms\" rather than the core mechanism. The concrete test would settle whether the label-propagation shortcut actually compromises the concept layer.","tokens_in":24507,"tokens_out":4976,"duration_ms":50360,"concrete_test":"Sample frames stratified by position within the 50-frame windows (e.g., 50 per task at frames 1, 10, 20, 30, 40, 50), obtain fresh VLM binary labels for each frame, and compare with the propagated first-frame labels; have a human judge resolve disagreements. If agreement is below, say, 95%, retrain the concept layer with dense per-frame labels (or with only true first-frame labels) and rerun Table 1 and the intervention cases. If the completion-rate gap vs. baselines and the case-study tracing remain unchanged, the shortcut is benign; if they shift, the central claims must be conditioned on the quality of this supervision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To establish the central claim — a transparent high-level policy whose decisions are traceable to true concept-level scene predicates — the learned concept layer must be faithful to those predicates. The only supervision comes from VLM annotations, and Appendix A.1 propagates the first-frame binary judgment over all 50 frames of a skill window. But skills are physical actions: during move(bread, microwave), \"bread inside microwave\" changes from 0 to 1; during close(microwave), \"microwave door open\" changes from 1 to 0. The paper explicitly admits that \"some later observations may occasionally deviate from the first frame in concept semantics.\" Training the BCE concept loss (Eq. 4) on these stale labels teaches the concept layer to predict averaged or incorrect values at exactly the decision-relevant moments. The ablation (Fig. 3) shows supervision quality is the dominant factor, so this is not a peripheral detail. Moreover, no concept-level accuracy is reported; only downstream CR and hand-picked qualitative examples. A decision tree over an inaccurate concept layer is transparent in form but not semantically truthful: its paths cite concepts that do not correspond to the actual scene. This makes both empirical superiority and the core transparency/intervention claims conditional on an unvalidated data-quality premise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ConceptTree, a framework for interpretable high-level skill selection in robotic manipulation. The approach trains a concept bottleneck layer with VLM-generated binary concept labels, then fits an axis-aligned decision tree over the learned concept values to predict skills. The authors evaluate on four real Franka Emika Panda manipulation tasks, comparing against a VLM policy, two concept-bottleneck baselines, and a sparse-linear variant (ConceptSLC) that shares the same concept layer. They report completion rates, ablation studies on supervision quality, temporal context, and tree depth, and qualitative case studies showing that changing a single concept value can redirect an incorrect tree prediction. The main claims are that ConceptTree consistently outperforms concept-based baselines, especially on harder tasks, and that its decisions are transparent and intervenable at the concept level.","tokens_in":24779,"tokens_out":5813,"duration_ms":58935,"significance":"If the central claims hold, ConceptTree would be a meaningful step toward interpretable high-level policies for long-horizon manipulation: it combines a concept bottleneck with a fully inspectable decision structure, and the intervention mechanism could support practical debugging without retraining. The real-robot evaluation against several baselines is a strength, as is the ablation isolating the decision module (ConceptSLC comparison). However, the paper's two central claims — consistent empirical superiority and semantic transparency — are not yet adequately supported. The largest gap is that concept-level correctness is never measured, and the VLM supervision itself is propagated from the first frame of each 50-frame skill window, a premise the paper acknowledges may be violated. The empirical 'consistently outperforms' claim is also weakened by the Fruits-Snacks result and the high variance on Coffee. These issues are fixable with additional analysis and experiments, so the contribution is potentially valuable but requires revision.","major_comments":[{"comment":"The label-propagation assumption is load-bearing but unvalidated. The paper states that VLM binary judgments are obtained only for the first observation in each 50-frame skill window and then assigned to all remaining frames, and it concedes that 'some later observations may occasionally deviate from the first frame in concept semantics.' For dynamic concepts like 'bread inside microwave' or 'microwave door open,' this can produce stale training targets at exactly the decision-relevant moments. No concept-level accuracy is reported, so we cannot know whether the learned concept layer is faithful to the true scene predicates. The ablation in Fig. 3 shows that supervision quality is the dominant factor in downstream performance, making this more than a peripheral detail. To support the central transparency/traceability claim, the paper should either collect per-frame VLM labels on a held-o","section":"§A.1, Eq. (4), Fig. 3"},{"comment":"The abstract and Section 4.2 claim that ConceptTree 'consistently outperforms' existing concept-based baselines, but Table 1 does not support an unqualified statement. On Fruits-Snacks w/ history, ConceptSLC achieves 100.00±0.00 while ConceptTree is 94.92±6.22; on Coffee, ConceptTree's 37.09±17.36 substantially overlaps ConceptSLC's 18.18±6.08 given the reported standard deviations. The authors acknowledge the Fruits-Snacks exception but still use 'consistently' in the abstract. The paper should report statistical significance tests (e.g., paired tests across seeds/episodes) and soften the claim to reflect the actual pattern: strong gains on Heat-Bread and Cola, mixed on Fruits-Snacks, and high-variance on Coffee. Additionally, Appendix C shows that the skill-aware loss weight λ is tuned per task (λ=0.1 for three tasks, λ=0.5 for Coffee), but it is not stated whether the baselines receiv","section":"§4.2, Table 1"},{"comment":"The intervention case studies demonstrate that changing a concept value to the correct side of a threshold redirects the decision-tree prediction. This is a tautological property of axis-aligned trees: if you set the value so that the branch changes, the prediction changes accordingly. The paper's actual contribution is the path-based diagnosis procedure in Appendix G (finding the deepest node shared with a correct path), and this is only illustrated on a few hand-picked examples. The claim that errors can be 'corrected by modifying individual concepts without retraining' needs quantitative support: how often does the identified concept correspond to the true error source, and how often does the intervention actually fix the decision? A systematic evaluation across all test errors (or a larger sample) is needed; otherwise the intervention claim rests on anecdotal evidence.","section":"§4.4, Appendix G"},{"comment":"The 'w/ CLIP target' ablation is used to argue that concept supervision quality is critical, and the reader is invited to infer that the paper's first-frame propagation is therefore adequate. But replacing VLM supervision with CLIP similarity changes many factors at once; it does not isolate the effect of label propagation. A direct ablation would compare training with first-frame-only propagated labels versus per-frame VLM labels (or at least labels sampled from more frames per window). Without such an experiment, the paper's statement in Appendix A.1 that 'this approximation to work well in practice' is an unsupported assertion. The correlation between better supervision and simpler trees (Fig. 4) is also reported descriptively; the causal direction is unclear.","section":"§4.3, Fig. 3"}],"minor_comments":[{"comment":"The phrase 'consistently outperforms' is contradicted by Table 1 itself; suggest 'generally outperforms' or a similar qualified statement.","section":"Abstract "},{"comment":"The piecewise definition of c̃_t is rendered awkwardly on the page; use a clear piecewise/cases format.","section":"Eq. (1)"},{"comment":"Table 4 reports the 'best depth' for each λ, while Section 4.2 says a fixed maximum depth of 7 is used for all tasks. Clarify which depth values are used in the main results of Table 1, and whether the best-depth values in Appendix C are used or only reported.","section":"Table 4 (Appendix C) vs Section 4.2"},{"comment":"The block labeled 'CBL' is not defined in text; either define it or expand the label to 'Concept Projection Layer'.","section":"Fig. 1"},{"comment":"The x-axis label says 'Depth' but the caption notes it is the maximum allowed depth during training. Consider relabeling to 'Maximum depth' to avoid ambiguity.","section":"§4.3, Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising, and the real-robot evaluation is a plus, but the evidence falls short of the paper's claims. The label-propagation issue is the most serious concern: without measuring concept-level accuracy or demonstrating that propagated labels are harmless, the transparency/intervention story is conditional. The 'consistently outperforms' wording should also be corrected. These are addressable with additional experiments and analysis, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: ConceptTree is a sensible combination of VLM-verified concept supervision, a learned concept bottleneck, and an axis-aligned decision tree for high-level skill selection in long-horizon manipulation. The paper does real robot experiments, detailed ablations, and is honest about limitations. But there is a load-bearing concern about the concept supervision, and the headline claim \"consistently outperforms\" is not supported by the paper's own Table 1.\n\nWhat's new: the specific assembly is new, and the intervention-by-editing-concept-values is cleanly demonstrated. The ablation (Fig. 3) shows supervision quality is the dominant factor, and the tree is compact and uses few concepts. That is genuine.\n\nThe soft spot: Appendix A.1 says for each 50-frame skill window, they annotate only the first frame with the VLM and propagate that binary label to all 50 frames. They admit later frames can deviate. During move(bread, microwave), \"bread inside microwave\" flips 0->1 partway; during close(microwave), \"microwave door open\" flips 1->0. So the concept layer is trained on averaged or wrong labels at exactly the decision-relevant moments. And they do not report concept-level accuracy at all — only downstream completion rate. If the concept layer isn't faithful, the tree's transparency is formal but not semantic: the paths cite concepts that don't correspond to the scene. Fig. 3 tells us concept supervision quality is the dominant factor, so this isn't peripheral. I would want concept precision/recall per concept on held-out frames, ideally manually verified, before believing the transparency claims.\n\nOther issues: Table 1 shows ConceptSLC hits 100.00±0.00 on Fruits-Snacks w/H vs 94.92±6.22 for ConceptTree, so \"consistently outperforms\" is too strong. Coffee differences are within noise (37.09±17.36 vs 18.18±6.08, large overlap). λ and depth are selected on validation, which is fine but should be reported as such. No code or data release, which hurts reproducibility. The intervention demo is useful but is, to a large degree, a definitional property of an axis-aligned tree — you edit a concept and the path changes. That's a feature, not a flaw, but it doesn't by itself prove the concept is semantically correct.\n\nBottom line: this is a solid, limited-scope contribution with one load-bearing premise that is currently unvalidated. It deserves a serious referee, and with concept-level validation, significance tests, and a tempered claim, it would be a good paper. If code and data appear, I'd probably cite it.","headline":"Useful interpretable-policy assembly, but the semantic transparency claim rests on an unmeasured label-propagation shortcut; worth reviewing, needs fixes.","tokens_in":25332,"tokens_out":1885,"would_cite":true,"duration_ms":17359,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T40"],"pacs":[],"model":"deepseek-v4-flash","headline":"High-level robotic manipulation policies can be expressed as decision trees over human-interpretable concept values — object states and spatial relations — so every skill choice is traceable to a short path of semantic predicates, and a sin","keywords":["concept bottleneck models","interpretable decision making","robotic manipulation","decision trees","long-horizon tasks","vision-language models","concept intervention"],"falsifier":"Take the Heat-Bread task and annotate every frame with the VLM instead of only the first frame of each window. Compare completion rate and concept accuracy. If, for episodes where 'bread inside microwave' changes during the move skill, the per-frame-trained ConceptTree recovers the correct close(microwave) decision that first-frame-trained ConceptTree misses, the central mechanism is supported; if not, the error lies elsewhere.","tokens_in":24325,"feed_emoji":"🤖","tokens_out":4070,"duration_ms":40051,"temperature":0.7,"pith_summary":"ConceptTree claims that the high-level part of robotic manipulation — choosing which skill to execute next — does not have to be a black box. It learns a small set of human-understandable concepts, such as 'bread inside microwave' or 'fridge door open', from visual observations, then selects skills using a decision tree over those concept values. This makes each decision an inspectable chain of predicates and allows errors to be corrected by editing one concept value rather than retraining. The paper reports that this approach outperforms existing concept-based baselines and black-box VLM policies on four real long-horizon tasks, with the largest gains on the most complex tasks.","feed_headline":"Concept tree gives robot skill decisions an edit button","feed_subtitle":"Each skill choice becomes a short chain of object-state predicates that humans can inspect and correct without retraining.","key_machinery":"A two-stage policy. Stage one is a concept layer: a fixed visual encoder followed by a learned projection with per-dimension sigmoid calibration, trained by binary cross-entropy against VLM-generated concept labels (complemented by a skill-aware loss that keeps the concept space aligned with decisions). Stage two is an axis-aligned decision tree whose internal nodes test whether a single concept value falls below a threshold; the tree is trained on the learned concept vectors (optionally concatenated with the previous step's vector). The concept space is the load-bearing object: it is the semantic interface that makes the tree's decisions human-readable and intervenable, and it transfers VLM","core_discovery":"On the paper's own terms, the central claim is that representing a high-level policy as a sequence of concept-level predicates over visual observations — learned with VLM-generated binary labels and organized into a decision tree — yields decisions that are both transparent and reliably better than latent or directly-queried black-box alternatives. The learned continuous concept values act as a semantic interface: the decision tree's splits are thresholds on single concepts, so each prediction reduces to a conjunction of a few scene predicates. The paper demonstrates that an error can be located by finding where the decision path diverges from the nearest correct path, and that flipping one","pith_inferences":["Because the policy is a literal decision tree over bounded concept values, it is a candidate for formal verification: safety properties (e.g., 'never pour when cup is empty') could be checked by exhaustive path inspection.","The method suggests a debugging protocol for robot failures that is close to code: read the path, edit the value, re-run — an implicit roadmap for explainable error correction in embodied systems.","The first-frame annotation shortcut opens a testable extension: generate per-frame VLM labels at low frequency but use interpolation or change-point detection to see if mid-skill concept drift explains the residual errors on the hardest task.","The fixed concept set bottleneck implies the next problem is automatic concept-set expansion from task descriptions; a failure mode to watch is concepts that are visually ambiguous by design."],"forward_implications":["Each skill prediction becomes a short chain of concept predicates (typically 3-5), so failure diagnosis reduces to finding the first false predicate.","Wrong decisions can be corrected in the field by editing the identified concept value instead of collecting new data and retraining.","The VLM is needed only at dataset-annotation time, so the deployed policy is cheap and fast while inheriting controllable semantics.","Tree-structured reasoning over concepts beats a sparse linear head on the same concept representation, especially on temporally dependent and visually ambiguous tasks.","Stronger concept supervision yields smaller trees and shorter decision paths, so investing in accurate semantics pays off in both accuracy and interpretability."],"fun_headline_variants":["Robot decisions get human-editable concept tree","ConceptTree: see and edit why robots choose actions","Traceable robot policies: flip a concept, fix the error","Robots' skill choices explained as editable concept chains","Intervene in robot policy by editing single concepts"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that one VLM binary judgment on the first frame of each 50-frame skill window remains valid for the entire window; if a concept flips mid-skill, the learned concept values average over a wrong label and the decision tree inherits the error.","fun_headline_variants_meta":{"raw":{"variants":["Robot decisions get human-editable concept tree","ConceptTree: see and edit why robots choose actions","Traceable robot policies: flip a concept, fix the error","Robots' skill choices explained as editable concept chains","Intervene in robot policy by editing single concepts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1045,"prompt_tokens":707,"completion_tokens":338,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":263}},"tokens_in":451,"tokens_out":338,"duration_ms":4316,"temperature":1.0,"reasoning_tokens":263,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:45:26.571270+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the Heat-Bread task and annotate every frame with the VLM instead of only the first frame of each window. Compare completion rate and concept accuracy. If, for episodes where 'bread inside microwave' changes during the move skill, the per-frame-trained ConceptTree recovers the correct close(microwave) decision that first-frame-trained ConceptTree misses, the central mechanism is supported; if not, the error lies elsewhere.","supporting_citations":[],"review_version":1}