{"id":"d6caddd5-546a-4ab7-8db9-dcaf76dafe9b","arxiv_id":"2607.06256","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"VLA skills that score 77-100% in isolation stall from chained states in BEHAVIOR-1K; failures trace to next-skill readiness, target grounding, and control execution.","lead":"This paper shows that robot skills which work reliably on their own often fail when chained into long tasks, because each skill leaves the world in a state the next skill was never trained to start from. It introduces a diagnostic harness that labels these handoff failures, turning near-zero task success into a prioritized list of what a robot's skill library must learn.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The snapshot-vs-chained gap is not isolated from harness gating: without an oracle reset to chained terminal states, the claimed drop could stem from premature verifier advances or budget/recovery limits rather than the skill's sensitivity to chained initial conditions.","rationale":"The reader's weakest assumption targeted the VLM-derived failure-category labels, which affect the diagnostic payoff but not the core snapshot-vs-chained gap itself. I identify a more fundamental concern: the gap is measured across two protocols that differ in success metric and execution context, not just initial-state distribution. The missing oracle-reset baseline means the skill's competence from chained terminal states is never directly tested; the end-to-end drop could be produced by the harness's verifier gating, budgets, or recovery behavior. This is a load-bearing gap because the paper's headline contribution is precisely that the drop is due to chained initial conditions. The paper's own limitation note (Sec. V-B) and the absence of a no-recovery ablation (also noted) support this concern. The proposed test—restoring to chained terminal states and running the skill in isolation—would settle the issue cleanly. I agree with the reader's overall CONDITIONAL verdict because the claim is plausible and internally consistent, but the central evidentiary link is not yet closed. The arm-reach ablation (Table VI) is a good controlled comparison for the verifier criterion and lends partial support, but it does not substitute for the oracle-reset test. My verdict remains UNCHANGED relative to the reader's CONDITIONAL: the concern reinforces the condition rather than escalating or clearing the paper.","tokens_in":11977,"tokens_out":4545,"duration_ms":42904,"concrete_test":"For each failed skill attempt in a representative round, restore the simulator to the exact state saved at the moment the skill was dispatched (the chained terminal state produced by the previous skill). Run the same skill checkpoint in isolation from that state, using the same success criterion as Table I (VLM verdict plus human review). Compare the success rate from these chained states against the clean-snapshot rates (77–100%). If chained-state success is comparably high, the end-to-end drop is a harness artifact and the central claim is falsified. Additionally, have human annotators judge a random sample of positive verifier verdicts during end-to-end rollouts; if a substantial fraction (e.g., >10%) are false positives, premature advance confounds the gap. If the chained-state isolation test is infeasible, the claim should be softened to 'rollouts under this harness stall' rather th","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the same pi0.5 checkpoints are competent from clean skill-boundary snapshots (77–100% success, Table I) but fail from chained terminal states (19.5% mean progress, Table II). This requires holding the evaluation protocol fixed and varying only the initial-state distribution. The paper does not do that. The isolated protocol restores a clean snapshot and scores success with a human-reviewed VLM verdict; the end-to-end protocol changes the initial state, the success metric (BDDL predicate / human progress score), and the execution context (VLM verifier gating, step budgets, replanning, recovery). The authors explicitly acknowledge (Sec. V-B) that 'an oracle skill-boundary reset remains a future upper bound because our current live execution setup does not expose arbitrary skill-start state restoration.' Without this reset, the skill itself is never evaluated directly from the exact chained terminal states that produce the failures. A further confound: the verifier's positive decisions are not audited. Table VII validates only failure decisions (20/21 real, over-strict 0.05); false-positive 'success' verdicts would advance a skill prematurely, leaving a state that is not a legitimate terminal state. Subsequent failures would then be misattributed to chained-state distribution when the cause is premature advance. The paper states that 'the binding failure at most handoffs is not premature advance' but this is an assertion without positive-verdict auditing. Thus the central snapshot-vs-chained conclusion is not fully isolated from harness gating effects.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a semantic execution harness for BEHAVIOR-1K that orchestrates π_0.5-based VLA skill checkpoints through typed skill contracts, step budgets, and a multi-view VLM verifier. It evaluates the same checkpoints from two initial-state distributions: clean skill-boundary snapshots and chained terminal states produced by previous skills. Isolated skill success is reported at 77–100% for several skills, while composed rollouts reach only 19.5% mean progress, with failures attributed to next-skill readiness, target grounding, and control execution. The authors argue that snapshot-state competence does not carry over to chained initial conditions and propose future next-skill readiness predicates.","tokens_in":12267,"tokens_out":7903,"duration_ms":76438,"significance":"If the snapshot-vs-chained gap were properly isolated, the paper would offer a valuable diagnostic instrument and a concrete roadmap for VLA skill libraries: the need for chained-state data and handoff-aware verification. The paper has notable strengths: it explicitly routes the same checkpoints in both protocols, includes a controlled arm-reach verifier ablation, reports a blinded audit of binary failure decisions, and is unusually candid about limitations. However, the central claim is not yet established because the evaluation protocols differ in more than the initial-state distribution, and the failure-category labels—the main diagnostic payoff—are not independently validated. The diagnostic framing is timely and the paper is potentially acceptable after substantial revision.","major_comments":[{"comment":"The central claim in Contribution 3—that snapshot-state skill competence does not carry over to chained initial conditions—is not isolated by the evaluation. The isolated protocol restores a clean demonstration snapshot and scores by VLM verdict plus human review; the end-to-end protocol starts from chained states and scores by BDDL/human progress, under verifier gating, step budgets, and recovery. The comparison therefore varies initial-state distribution together with evaluation metric and execution context, not only the initial state. Section V-B explicitly states that an oracle skill-boundary reset is future work, confirming that the same skill is never evaluated directly from the exact chained terminal states. The observed drop could stem from premature verifier advances, budget/recovery limits, or metric differences rather than from skill sensitivity to chained starts. An oracle-re","section":"§IV-A/IV-C, Tables I–II"},{"comment":"The three-way failure attribution (next-skill readiness, target grounding, control execution) is the paper's main payoff, but the category labels are derived from the same VLM verifier's free-text reasons. The audit (Table VII) validates only the binary failure decision (20/21 confirmed), and the paper itself notes that the 'failure-category labels remain verifier-derived and may carry bias.' No independent human labeling of categories or per-category agreement is reported. If the labels are noisy, the counts in Table III are not established. In addition, positive verifier decisions are not audited; false-positive 'success' verdicts would advance skills prematurely and make subsequent failures appear to be chained-state failures. The statement in Sec. IV-D that 'the binding failure at most handoffs is not premature advance' is an assertion without positive-verdict auditing.","section":"§IV-D, Table III, Sec. V-B"},{"comment":"The arm-reach handoff clause hand-authored into the move_to postcondition is the operative readiness check: Table VI shows that tightening it changes the diagnostic counts (12 additional next-skill readiness failures, 25 more move_to attempts) and recovers the radio task. This wording is neither learned nor independently validated, so the readiness-failure counts are contingent on an arbitrary prompt choice. The 'controlled ablation' holds tasks, instances, and checkpoints fixed but does not validate the criterion itself. A sensitivity analysis with an alternative handoff criterion, or a human-judged set of handoff states, is needed before the readiness counts can be read as evidence about the VLA skills.","section":"§II, §IV-D, Table VI"},{"comment":"The quantitative evidence is thin relative to the strength of the headline. Isolated success rates are based on 4–35 attempts per skill (e.g., 6/6, 7/7, 4/6) with no confidence intervals; the '77–100%' claim overstates precision. Progress scores rely on n=3 instances per task, and the attribution diagnostics in Table III come from a single representative rollout per task. While the paper labels the results 'preliminary,' the abstract and conclusion present the gap as a finding. I would like to see confidence intervals for Table I and a sensitivity note about the single-round attribution; otherwise the reader cannot gauge how stable the snapshot-competence and failure-attribution numbers are.","section":"Tables I and II"}],"minor_comments":[{"comment":"The abstract says '77–100% success,' but Table I includes close_door at 66.7% (4/6). If close_door is excluded from 'door-opening skills,' the wording should clarify which skills are included.","section":"Abstract and Table I"},{"comment":"The progress score is scored by a single annotator and alternative valid plans are not credited; this is acknowledged in the caption but should be stated more prominently in the main text since it directly affects how the 19.5% figure should be interpreted.","section":"Table II"},{"comment":"The verifier confidence threshold of 0.6 is a free parameter with no sensitivity analysis. A brief report of how results change with threshold would strengthen the harness's robustness case.","section":"§III-B"},{"comment":"The table reports 'Mean task score q̄' of 0.01 and 0.10 under the two handoff criteria, but the main text uses 'progress score.' Clarify the relationship between q̄ and the progress score to avoid confusing two different metrics.","section":"Appendix, Table VI"},{"comment":"The manuscript spells 'BEHAVIOR-1K' inconsistently as 'BEHA VIOR-1K' in several places; use a consistent spelling.","section":"General"},{"comment":"The proposed readiness templates are not evaluated; the paper says this is future work. Fine for a diagnostic paper, but the conclusion should not overstate the benefits of readiness-aware verification.","section":"§V-C and Table IX"}],"recommendation":"major_revision","confidential_remarks":"The protocol confound is the main risk. If the authors can add an oracle-reset experiment or otherwise isolate the chained-state distribution, the paper becomes a solid diagnostic contribution; if not, the headline claim is unsupported and I would lean toward reject. The category-label validation (human labeling or per-category agreement) is also necessary. The authors are transparent about limitations, which is to their credit; the revision path is feasible within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper identifies a real and underappreciated failure mode—a VLA skill that succeeds from a clean snapshot can still stall when called from the state the previous skill left behind—and it documents that gap cleanly enough to be worth reading. The three-way attribution (readiness, grounding, control) is plausible but not yet established, and the chained-state explanation is entangled with the harness's own gating. Still, the directional finding is likely right.\n\nWhat it does well: the isolated-vs-composed comparison uses the same checkpoints, scenes, and instances, which is the correct design. The arm-reach verifier ablation is a single-variable change and it recovers the radio task, which lends credibility to the readiness signal. The authors are unusually honest—they flag the small samples, the verifier-derived labels, the absence of an oracle reset, and the single-annotator progress scoring. The binary verifier audit (20/21 real failures) is a legitimate check.\n\nSoft spots, in order of weight. First, the failure-category attribution, the paper's main payoff, rests entirely on VLM free-text labels that were never validated. The authors admit \"the failure-category labels remain verifier-derived and may carry bias.\" That's not a minor caveat; Table III could shift with a different verifier prompt. Second, the central claim—that snapshot competence doesn't carry over to chained initial conditions—is confounded. The end-to-end protocol changes not just the initial state but also the success metric and the execution context (verifier gating, step budgets, replanning). The paper says \"the binding failure is not premature advance,\" but it never audits positive verdicts. Without an oracle reset, a skeptic can argue the harness is the problem, not the skill. Third, the samples are tiny (as low as six trials per skill, three instances per task), and no code, data, or trace artifacts are released, despite the promise of \"replayable traces.\"\n\nThese are fixable gaps. An oracle-reset experiment, a validation of the category labels, and a release of the traces would answer the main objections. For a pilot, this is a solid, honest piece. I'd send it to peer review expecting heavy revision, and I'd cite the snapshot-vs-chained gap in my own work with the caveats attached.","headline":"A worth-reading diagnostic pilot with a real empirical finding, but the causal attribution is not yet substantiated.","tokens_in":12862,"tokens_out":4172,"would_cite":true,"duration_ms":40783,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Long-horizon robot tasks fail not because the skills are weak but because each skill leaves the next one a state it was never trained to start from.","keywords":["vision-language-action models","skill composition","semantic handoff","long-horizon robot tasks","VLM verification","imitation learning","distribution shift"],"falsifier":"Run the same chained checkpoints with an oracle skill-boundary reset: restore a clean snapshot before every skill instead of letting it start from the previous skill's terminal state. If the mean progress does not rise toward the 77–100% isolated success rates, the chained-state distribution is not the cause of the collapse; if it does, the central claim is confirmed.","tokens_in":11806,"feed_emoji":"🤖","tokens_out":8114,"duration_ms":75234,"temperature":0.7,"pith_summary":"This paper tries to establish that in long-horizon robot tasks composed from vision-language-action (VLA) skills, the bottleneck is rarely the individual skill and almost always the handoff between skills: a skill can satisfy its own postcondition yet leave the robot, objects, or camera views in a state from which the next skill cannot start. To test this, the authors build an execution harness that invokes the same skill checkpoints two ways — from clean skill-boundary snapshots and from the chained terminal states of previous skills — and find a large gap: navigation, grasping, placement, and door-opening skills reach 77–100% success in isolation, while composed rollouts average only 19.5% progress and nearly never satisfy the final task predicate. The harness then attributes the stalled attempts to three concrete causes — next-skill readiness, target grounding, and control execution — turning near-zero task success into a prioritized list of what VLA skill libraries must learn next. A sympathetic reader would care because the result suggests that the standard practice of benchmarking skills in isolation overstates what skill libraries can do in composition, and it points to chained-state data and readiness-aware verification as targeted fixes.","feed_headline":"Skills hit 77-100% alone, 19.5% in chains","feed_subtitle":"An agent harness traces the collapse to the messy states left between skills, and shows what robot skill libraries must fix.","key_machinery":"The load-bearing mechanism is the agent execution harness and its verification loop. Each skill call is a typed contract — name, arguments, language prompt, step budget, verifier interval, and a handoff-aware postcondition — and a multi-view vision-language-model (VLM) verifier judges head and wrist camera views against that postcondition to decide whether to advance, retry, or replan. The semantic handoff problem is the central object: a skill satisfies its own postcondition yet leaves the robot, objects, or camera views in a state from which the next skill cannot start. The operative readiness check is a hand-authored arm-reach clause in the navigation postcondition, which is what surfaces","core_discovery":"On its own terms, the paper claims to have diagnosed why composed long-horizon execution collapses. The central finding is that snapshot-state skill competence does not carry over to chained initial conditions: skills that succeed in 77–100% of isolated runs from curated boundaries stall from the messy terminal states produced by preceding skills, yielding an average 19.5% progress on the reference plan and near-zero task-predicate success. The same checkpoints are used in both evaluations, so the drop is due to the initial-state distribution, not to weaker policies. Trace-derived attribution classifies the failed attempts into next-skill readiness (navigation stopping short of a reachable p","pith_inferences":["A causal test the paper leaves open: hold the skill checkpoints fixed but restore a clean skill-boundary snapshot before every skill; if progress does not jump toward the 77–100% isolated rates, then chained-state distribution is not the whole explanation, and policy-level weakness shares the blame.","The failure-category counts should be re-estimated with human-labeled categories before being used to steer data collection, since the paper validates only the binary failure decision, not the labels.","The handoff diagnosis likely generalizes beyond household manipulation: any modular agent stack whose components are validated on curated inputs but composed without boundary validation should show similar composition collapses, so boundary-condition testing is a general engineering lesson.","The arm-reach clause could be studied as a tunable threshold: varying its strictness should trade off readiness failures against re-navigation cost, revealing an optimal handoff criterion per skill pair."],"forward_implications":["Isolated single-skill benchmarks cannot certify a VLA skill library for long-horizon use; composition adds a chained-state distribution that snapshot evaluation never samples, so progress must be measured end-to-end.","Scaling snapshot-only training data will not close the gap, because it adds density where skills are already competent; targeted chained-state data and readiness-aware verification are the likely fixes.","The verifier's handoff criterion is not a neutral observer: tightening the navigation postcondition from reached-area to arm-reach surfaced 12 additional readiness failures, triggered 25 more re-navigation attempts, and recovered one task.","Near-zero task-predicate success can be turned into a graded, actionable signal: the 19.5% mean progress score localizes the collapse to contact-rich manipulation and late multi-object steps, and each attributed failure maps to a concrete remedy."],"fun_headline_variants":["Robots ace skills solo, stall in long chains","Isolated skills 77-100% but composed tasks fail","Skill handoffs break: clean states pass, messy chains fail","Why robot skills fail in chains: boundary states","Agent harness traces skill-chain collapse to handoff"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The headline attribution rests on the assumption that the VLM verifier's free-text reasons are accurate failure-category labels — the paper validates only that a flagged failure is a real failure (20 of 21) and says the categories may carry bias; if the labels are noisy, the readiness-vs-grounding-vs-control counts are not established, even though the snapshot-vs-chained gap itself may still hold.","fun_headline_variants_meta":{"raw":{"variants":["Robots ace skills solo, stall in long chains","Isolated skills 77-100% but composed tasks fail","Skill handoffs break: clean states pass, messy chains fail","Why robot skills fail in chains: boundary states","Agent harness traces skill-chain collapse to handoff"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00053,"raw_usage":{"total_tokens":2407,"prompt_tokens":778,"completion_tokens":1629,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":1551}},"tokens_in":522,"tokens_out":1629,"duration_ms":10431,"temperature":1.0,"reasoning_tokens":1551,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T08:16:36.086166+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same chained checkpoints with an oracle skill-boundary reset: restore a clean snapshot before every skill instead of letting it start from the previous skill's terminal state. If the mean progress does not rise toward the 77–100% isolated success rates, the chained-state distribution is not the cause of the collapse; if it does, the central claim is confirmed.","supporting_citations":[],"review_version":2}