{"id":"4387255c-4846-4b2c-9683-2dcfd1332c67","arxiv_id":"2607.18060","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"RoboHarness combines VLAs, RL policies, and TAMP planners via an LLM router and a memory-bridge handoff, reporting 95.2% average success on long-horizon LIBERO-LoHo versus 64.8% for the best baseline.","lead":"RoboHarness is a framework that lets a language-model planner pick among different robot control systems — vision-language models, RL policies, and task-and-motion planners — for different subtasks, using stored execution memories to smooth handoffs. The paper reports large gains over single-policy baselines on long-horizon benchmarks and real-robot tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot evaluation is not established: memory bank and self-evolution updates retained across evaluation episodes may inject task-specific experience, so Table 2 may measure in-task adaptation rather than zero-shot orchestration.","rationale":"The reader's verdict is REJECT, and this stress-test identifies a concern that strengthens that rejection. The paper's central claim is that RoboHarness enables zero-shot long-horizon planning by composing heterogeneous policies without joint retraining or shared action representation. However, the evaluation protocol allows RoboHarness to keep memory and self-evolution updates across evaluation episodes, so its performance may be driven by task-specific experience accumulated during the evaluation itself. This is not a minor detail: the Memory Bridge's construction of R_conf,t and f_score,t depends on retrieved trajectories, and the paper never specifies how those trajectories exist at the start. The first-episode behavior is therefore either impossible as described or requires pre-seeded task-relevant memory. Both alternatives undermine the zero-shot interpretation of Table 2. This concern is externally verifiable but currently unresolved, and it directly affects the strongest empirical claim. The reader's weakest_assumption—memory initialization and coverage—captures the same issue, so agreement is 'agree.' The verdict remains REJECT because the empirical support for the central claim is not established as stated.","tokens_in":18328,"tokens_out":4241,"duration_ms":49200,"concrete_test":"Run LIBERO-LoHo under a strict zero-shot protocol: reset M to its initial contents before each task (or use the empty memory bank) and disable all four self-evolution mechanisms; report success/progress on the first episode only. Also require a written specification of M's initial contents—which tasks/rollouts were used to seed memory. If first-episode, memory-reset progress/success drops substantially below Table 2's 97.5/95.2, the zero-shot claim is unsupported. If no code or memory dumps are released to permit this check, the claim remains unverified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RoboHarness's headline claim is 'zero-shot long-horizon planning' (Sec. 3: no task-specific joint training of policies or orchestration). The evaluation protocol in Sec. 4.3 explicitly retains self-evolution updates 'across evaluation episodes, allowing RoboHarness to accumulate experience and progressively adapt to new tasks,' and Sec. 4.2.1 says memory is 'continuously updated after each rollout.' The Memory Bridge (Sec. 4.2.2) requires retrieved anchors from M to construct R_conf,t and f_score,t; with empty memory it cannot select a handoff target. Therefore either the first episode of each LIBERO-LoHo task runs without a functioning Memory Bridge, or M was pre-populated with trajectories—whose provenance is never disclosed. If those trajectories include successful rollouts of the same tasks or earlier evaluation episodes, then the reported 97.5 progress/95.2 success (Table 2) is not zero-shot: RoboHarness has task-specific execution experience that the frozen baselines lack. This is load-bearing because the central contribution is composition without task-specific adaptation; the paper does not rule out the alternative that most gains come from accumulated in-task memory/evolution rather than from the orchestration design itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RoboHarness is an orchestration framework that composes heterogeneous robot policies (a π0.5 VLA, an OpenVLA-OFT RL policy, and a TAMP planner) through an LLM-based coding agent, auxiliary understanding/memory/self-evolution skills, and a Memory Bridge that constructs handoff trajectories between policies. The paper claims strong performance on LIBERO, LIBERO-Plus, LIBERO-LoHo, 500 custom simulated tasks, and 135 real-robot trials, with the central claim being zero-shot long-horizon planning via capability-aware routing and stable inter-policy handoffs without joint retraining.","tokens_in":18635,"tokens_out":3923,"duration_ms":39345,"significance":"If the empirical claims are supported, the paper addresses a genuine and timely problem: composing independently developed robot policies with different interfaces and capability boundaries. The Memory Bridge is a conceptually interesting mechanism for handling inter-policy distribution shift, and the breadth of the experimental effort is commendable. However, the central zero-shot claim is not adequately supported by the current evaluation protocol, and several load-bearing details of the method remain unspecified. The framework's significance therefore depends on whether these issues can be resolved with additional experiments and analysis.","major_comments":[{"comment":"The zero-shot claim is contradicted by the evaluation protocol. Sec. 3 defines zero-shot as solving unseen compositions 'without task-specific joint training,' but Sec. 4.3 states that self-evolution updates are 'retained across evaluation episodes, allowing RoboHarness to accumulate experience and progressively adapt,' and Sec. 4.2.1 says memory is 'continuously updated after each rollout.' Thus the averages in Table 2 likely reflect within-evaluation adaptation rather than zero-shot performance, and the frozen baselines are not compared on equal footing. The paper must report first-episode performance with empty memory, or explicitly separate an adaptation phase from a zero-shot phase.","section":"Sec. 3, Sec. 4.2.1, Sec. 4.3, Table 2"},{"comment":"The Memory Bridge cannot operate from an empty memory: the progress estimator f_score,t and the support region R_conf,t are constructed entirely from retrieved anchor nodes (N_t = RetrieveM(gt+1, ot)). The paper never specifies how the first handoff is performed when M is empty, nor whether M is pre-populated and, if so, with what data. If the memory contains successful trajectories of the evaluation tasks or earlier episodes of the same tasks, the headline results in Table 2 are not evidence of zero-shot orchestration. A cold-start procedure or explicit memory initialization must be described.","section":"Sec. 4.2.2"},{"comment":"The central quantitative claims lack statistical support. No table or figure reports error bars, number of seeds, or number of evaluation episodes per task. Robot policies are stochastic, and differences such as 97.5 vs. 98.7 (Table 1) or 95.2% success (Table 2) may be within noise. The real-robot results (Sec. 8) report percentages from only 15 trials per condition without confidence intervals. The authors should provide variance estimates or, at minimum, full trial counts for every reported number.","section":"Tables 1, 2; Figure 4; Sec. 8"},{"comment":"Baseline comparisons are not controlled. The paper states that evaluation results were 'obtained from the benchmark evaluation of Huang et al. (2026a) and Zhang et al. (2026)' rather than from runs under the identical protocol used for RoboHarness. Differences in task splits, observation inputs, seeds, or evaluation code can materially affect success rates. The authors must either re-run all baselines under the same conditions or clearly state which numbers are copied and justify why the comparison is valid.","section":"Sec. 5.2"}],"minor_comments":[{"comment":"Typographical error: 'In addition, we designed In addition, we designed 500...' appears in the simulated benchmarks paragraph.","section":"Sec. 5.1"},{"comment":"Wording error: 'There are 10 total classes for of the tasks' should be corrected.","section":"Sec. 11.1.1"},{"comment":"Capitalization is inconsistent: 'Overall success Rate' should be 'Overall success rate.'","section":"Figure 5 caption"},{"comment":"The paper introduces several free parameters (epsilon, lambda_motion, l, Delta_t, K_text, K_vis, encoder fusion weights) but provides no sensitivity analysis. A brief study on at least the most sensitive parameters (e.g., support threshold epsilon and retrieval counts) would substantially strengthen the robustness claims.","section":"Sec. 4.2, equations"},{"comment":"The limitation that 'capability characterization and Memory Bridge construction rely on accumulated execution evidence and may be unreliable when a policy is newly introduced or relevant experience is sparse' is in direct tension with the zero-shot claim. This tension should be addressed explicitly in the main text, not only in the limitations section.","section":"Sec. 9"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim rests on a zero-shot evaluation, but the protocol allows memory and self-evolution to accumulate across evaluation episodes and the Memory Bridge cold-start is unspecified. These are fixable with additional experiments (first-episode evaluation, empty-memory baseline, seed counts, and controlled baseline re-runs). The framework is potentially interesting, but the current evidence is not sufficient to support the headline claim. I recommend major revision rather than rejection, as the issues are addressable within the paper's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth knowing two things. First, the framework idea — routing across genuinely heterogeneous policy families (VLA, RL, TAMP) with a memory-based handoff rather than joint training — is genuinely new, and the Memory Bridge is specified concretely enough to reimplement. Second, the headline zero-shot numbers should not be read as zero-shot, and the paper's own protocol is why. Section 4.3 says validated self-evolution updates are retained across evaluation episodes; Section 4.2.1 says memory is updated after each rollout. The bridge needs retrieved anchors to build its support region and progress estimator, so on the first episode either the bridge is inert or the memory was pre-populated from a source never disclosed. If that source includes trajectories from the evaluated tasks — or from earlier episodes, which the retention policy allows — Table 2 measures accumulated in-task experience against frozen baselines, not zero-shot composition. The stress-test note lands correctly; this is load-bearing because zero-shot is the paper's central claim.\n\nThe good parts: the framework is a real departure from the homogeneous skill-chaining work it cites (initiation sets, terminal-state regularizers). The Memory Bridge mechanism — hierarchical text-visual retrieval, support region, pairwise-ranking progress estimator, motion-cost-regularized target selection — is well specified. The ablations are informative: removing any of the three auxiliary modules drops success substantially, supporting the claim that all three contribute. Figure 3 is a reasonable attempt to show context-dependent routing, and Section 9 concedes the sparse-memory weakness, which is more honest than most systems papers.\n\nOther soft spots, in order of severity. Baselines are quoted from the authors' own earlier benchmark evaluations (Section 5.2) rather than re-run in a shared harness; there are no error bars or seed counts anywhere; no code or data are released; memory initialization and coverage are unspecified. These are not cosmetic — the magnitude of the reported margins (97.5 vs 84.9) is unverifiable as presented. Minor: a duplicated phrase in Section 5.1 ('we designed In addition, we designed') suggests hasty assembly. None of this makes the method incoherent; the framework is plausible on its own terms.\n\nWho it's for: anyone working on skill chaining, policy composition, or agentic harnesses for manipulation. The idea deserves engagement; the empirical claims need major revision. My recommendation: send to peer review, not desk reject, and instruct reviewers to require first-episode-only evaluation with cold memory, disclosed memory provenance, baselines re-run in one harness, and released artifacts. Fix those and this could be a solid systems paper.","headline":"Genuinely new orchestration idea with a concretely specified Memory Bridge, but the paper's own protocol contradicts its zero-shot claim — treat Table 2 as measuring in-task adaptation, not zero-shot composition.","tokens_in":19124,"tokens_out":5487,"would_cite":true,"duration_ms":58104,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RoboHarness claims that independently developed robot policies — a vision-language-action model, an RL-trained controller, and a task-and-motion planner — can be composed zero-shot by a memory-driven harness that routes subtasks and stabili","keywords":["heterogeneous robot policies","long-horizon planning","zero-shot orchestration","policy handoff","memory retrieval","capability boundaries","vision-language-action models","task-and-motion planning"],"falsifier":"Give each policy a memory bank built only from trajectories of a different LIBERO split (e.g., only LIBERO-Object trajectories for all policies) and run LIBERO-LoHo; if success drops to near single-policy baselines, the reported gains depend on in-distribution memory rather than the orchestration mechanism. A second direct check is to measure the fraction of bridge-accepted handoff states within R_conf,t that still fail under the next policy.","tokens_in":18214,"feed_emoji":"🤖","tokens_out":5354,"duration_ms":55163,"temperature":0.7,"pith_summary":"RoboHarness aims to establish a general answer to a scaling problem: long-horizon robotic tasks require capabilities no single policy has, and different policy families are strong in complementary regimes. The paper claims that these independently built policies can be chained without joint retraining or a shared action representation, provided a harness reasons about each policy's uncertain, context-dependent capability boundary and stabilizes inter-policy handoffs. The load-bearing mechanism is the Memory Bridge, which retrieves prior successful trajectories of the next policy, learns its local in-distribution state region, and steers the robot into that region before control transfer. Reported results include 95.2% success and 97.5 progress on LIBERO-LoHo long-horizon tasks, 98.7% on standard LIBERO, 93.2% average under LIBERO-Plus perturbations, and 86.7% on real-robot assembly trials, outperforming every constituent policy and baseline. If correct, the framework implies that the bottleneck in long-horizon robot execution is orchestrating capability boundaries rather than the policies themselves.","feed_headline":"95% long-horizon success by routing three robot policies","feed_subtitle":"Vision, RL, and symbolic planners cooperate without joint retraining — and beat every single policy alone.","key_machinery":"The Memory Bridge is a plug-and-play chaining technique that turns policy handoff into a retrieval-and-scoring problem. Given a handoff, it retrieves top-K anchor nodes from the next policy's memory bank, expands each anchor along its trajectory into robot-state/temporal-offset pairs, fits a lightweight progress estimator f_score,t, defines a support region R_conf,t = {s : distance to expanded memory ≤ ϵ}, and selects a handoff target that maximizes f_score,t minus motion cost inside that region, generating a bridge trajectory with an off-the-shelf motion planner. Supporting this are three auxiliary skill groups — understanding skills (uncertainty, visual/semantic context, state-policy compa","core_discovery":"RoboHarness claims that heterogeneous robot policies — independently developed VLAs, RL policies, and TAMP planners — can be composed zero-shot by treating them as agentic skills, and that the orchestration bottleneck is the uncertain capability boundary between them, not the policies themselves. The central mechanism is the Memory Bridge: it retrieves the next policy's successful trajectories, builds a local progress estimator f_score,t over robot states, declares a support region R_conf,t of states near retrieved memory, and plans a bridge trajectory that lands the robot in that region before handoff. On LIBERO-LoHo, reported progress/success is 97.5/95.2 versus 55.3/6.4 for the best singl","pith_inferences":["A decisive test of the framework is to seed each policy's memory bank with trajectories from unrelated tasks and then run LIBERO-LoHo: if success collapses to near single-policy baselines, the reported gains depend on distribution-matched memory rather than on the orchestration mechanism itself.","The 'zero-shot' label should be read as 'no joint training of policies' rather than 'no prior experience with the task'; the paper does not specify how memory is initialized or whether evaluation trajectories leak into it, so a fair reader should treat the claim as conditional on memory coverage.","The same orchestration principle should extend beyond tabletop manipulation to navigation, mobile manipulation, and MPC-vs-learned-policy settings, since the framework never touches policy internals — it only needs a state-space definition and per-policy memory banks.","If capability boundaries are learned reliably online, a natural next step is to have the harness propose new decomposed skills or flag subtasks outside all policies' support, a capability the paper gestures at in its limitations but does not develop."],"forward_implications":["Independently trained robot policies become plug-and-play components: a developer can add a new policy with a memory bank and a policy card, and the harness routes around it without retraining the stack.","Long-horizon performance shifts from the policy to the orchestration layer: the strongest single constituent policy improves from 6.4% to 95.2% success on LIBERO-LoHo under the harness, suggesting coordination, not single-policy capability, is the limiting factor.","The support-region constraint makes the system decline handoffs into states unsupported by memory, a testable property: bridge acceptance should correlate with downstream policy success.","Robustness behavior is adaptive rather than fixed: routing ratios shift with perturbation type, so capability characterization can be learned online from execution evidence instead of being hand-specified."],"fun_headline_variants":["Memory bridge orchestrates robot policies to 95% success","Zero-shot robot policy routing hits 95% on long-horizon tasks","RoboHarness: memory-driven robot policy team-up hits 95%","Heterogeneous robot policies cooperate via memory bridge","From 6% to 95%: memory bridge for robot policy handoffs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The memory bank for each policy must already contain enough successful trajectories from that policy's execution regime for retrieval to characterize its in-distribution state region; sparse, unrepresentative, or task-contaminated memory would break both routing and every handoff.","fun_headline_variants_meta":{"raw":{"variants":["Memory bridge orchestrates robot policies to 95% success","Zero-shot robot policy routing hits 95% on long-horizon tasks","RoboHarness: memory-driven robot policy team-up hits 95%","Heterogeneous robot policies cooperate via memory bridge","From 6% to 95%: memory bridge for robot policy handoffs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000708,"raw_usage":{"total_tokens":3034,"prompt_tokens":763,"completion_tokens":2271,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":2179}},"tokens_in":507,"tokens_out":2271,"duration_ms":17613,"temperature":1.0,"reasoning_tokens":2179,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:12:51.292886+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give each policy a memory bank built only from trajectories of a different LIBERO split (e.g., only LIBERO-Object trajectories for all policies) and run LIBERO-LoHo; if success drops to near single-policy baselines, the reported gains depend on in-distribution memory rather than the orchestration mechanism. A second direct check is to measure the fraction of bridge-accepted handoff states within R_conf,t that still fail under the next policy.","supporting_citations":[],"review_version":1}