{"id":"d8bc0b78-313f-4df7-a2b3-a52725157956","arxiv_id":"2608.09516","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"HarnessWAM wraps a world-action model with an external VLM task manager, task graph, capability projection, and recovery loop, reaching 59.6% full-task success on RoboMemArena and 23.7% SR on RoboCerebra Ideal.","lead":"The paper adds a planning and memory layer around a robot's predictive action model so the robot can remember what it has seen, check whether subtasks worked, and retry after failures. A generalist should care because this extra layer roughly doubles full-task success on a challenging memory-heavy robot benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is at risk because external baselines are not protocol-matched and the margin over the best non-harness baseline is small; the central design claim is independently supported.","rationale":"The reader's verdict is CONDITIONAL with the same weakest assumption: the published baseline numbers are not protocol-matched to HarnessWAM's evaluation. My stress test examined the internal logic of the paper and found no additional load-bearing concern that would overturn the central design claim. The architecture is coherent, the same-WAM baselines isolate the harness contribution, the ablations in Table 3 show monotonic and plausible degradations, and the plan-level diagnosis in Table 4 provides a quantitative mechanism for the largest ablation effect (executable-space projection). The termination argument in Section 3.5 (finite task graph plus bounded budget guarantees termination) is sound. The most defensible reading of the paper is that the harness substantially improves WAM reliability in these benchmarks; the weaker, less certain part is the claim to state-of-the-art status against externally reported numbers. Since the reader already conditioned on precisely this, and since the internal evidence supports the framework's value, I do not recommend moving the verdict. The honest non-finding on the central design claim, combined with the confirmed external-comparison concern, leaves the CONDITIONAL verdict unchanged.","tokens_in":14383,"tokens_out":2077,"duration_ms":17169,"concrete_test":"Re-run PrediMem, GPT-4o Planner + OpenVLA, and the HPE Framework under the HarnessWAM evaluation protocol that produced Tables 1 and 2 (same initial states, seeds, observation interfaces, 20-rollout budget, and task-level budgets), or obtain the benchmark authors' per-episode logs and recompute their metrics under that protocol. If the re-run baselines remain below HarnessWAM by margins consistent with the reported tables, the SOTA claim holds; if the baselines move by more than ~2 points or exceed HarnessWAM, the SOTA claim should be downgraded to a controlled-comparison claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"HarnessWAM's headline 'state-of-the-art' claim depends on comparisons to external baselines whose evaluation protocols are not matched to HarnessWAM's. Section 4.1 states HarnessWAM uses 20 rollouts per task with matched initial states, seeds, and observation interfaces, but the published baselines in Tables 1 and 2 are 'taken from the corresponding benchmark evaluations.' If those baselines used different numbers of rollouts, different initial-state sampling, different observation interfaces, or different task-level budgets, the reported gains of +21.1/+14.7 over PrediMem and +1.78 over GPT-4o Planner + OpenVLA could reflect protocol differences rather than capability. This is especially acute on RoboCerebra Ideal, where the gain over the next-best non-harness method is only 1.78 points and no error bars are reported, so sampling noise alone could invert the ranking. The concern is external to the central design claim: the same-WAM ablations (WAM + Whole Task, WAM + Static Plan) and the internal ablations in Table 3 support the statement that the harness, not the WAM weights, is what enables reliable long-horizon execution. But the subset of the abstract that asserts state-of-the-art performance is not verifiable from the manuscript as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"HarnessWAM proposes a model-external agentic framework that wraps a World Action Model (WAM) with an evidence-grounded scene belief, a structured task graph, a capability-conditioned executable-space projection, and an event-driven dual-timescale execution loop. The Task Manager (a VLM) maintains task state and decides when to advance, observe, replan, recover, or terminate, while a lightweight progress estimator supplies continuous execution evidence. Experiments on RoboMemArena and RoboCerebra Ideal report state-of-the-art full-task and subtask success rates, and same-WAM ablations (WAM + Whole Task, WAM + Static Plan) together with ablation Table 3 indicate that the harness components, especially executable-space projection and progress-conditioned event control, are responsible for large parts of the improvement over the raw WAM.","tokens_in":14581,"tokens_out":6669,"duration_ms":59302,"significance":"If the results hold, the paper makes a valuable contribution: it demonstrates that a finite-horizon WAM can be extended into a persistent embodied agent through a structured external runtime, without modifying the WAM architecture. The internal evidence is strong: the same-WAM baselines isolate the effect of the harness from the underlying policy, and the plan-level diagnostics in Table 4 directly quantify how projection improves syntax, dependency satisfaction, binding, and executability. The deterministic projection and the history-invariance constraint are clean, falsifiable design elements. The main weakness is that the abstract's state-of-the-art claim is not fully verifiable as written because the external baselines in Tables 1 and 2 are taken from benchmark evaluations rather than a matched protocol, and no uncertainty estimates are reported.","major_comments":[{"comment":"The evaluation protocol is not matched to the external baselines. Section 4.1 states that HarnessWAM and the same-WAM diagnostic variants use 20 rollouts per task with matched initial states, seeds, observation interfaces, and task-level execution budgets, while published baseline numbers are 'taken from the corresponding benchmark evaluations.' If those baselines used different rollout counts, initial-state distributions, observation interfaces, or budgets, the reported gains over PrediMem (+21.1/+14.7 on RoboMemArena) and over GPT-4o Planner + OpenVLA (+1.78 on RoboCerebra Ideal) could reflect protocol differences rather than capability. This is particularly acute on RoboCerebra Ideal, where the margin is small and no error bars are provided. The state-of-the-art claim in the abstract is therefore not verifiable from the manuscript as written.","section":"Section 4.1, Tables 1 and 2"},{"comment":"No confidence intervals, standard errors, or significance tests are reported for any of the success rates in Tables 1-3, despite the 20-rollout evaluation protocol. This matters for the small-margin RoboCerebra Ideal comparison (23.70 vs 21.92, a 1.78-point edge) and for ablation differences such as progress-only switching versus full HarnessWAM on full-task success (55.4 vs 59.6). With 20 rollouts per task, binomial sampling noise can easily exceed these margins, so the reader cannot assess whether the reported ordering is stable. The paper should report per-task variance, confidence intervals, or paired statistical tests for at least the same-WAM comparisons and the main RoboCerebra comparison.","section":"Tables 1-3"},{"comment":"The executable primitive set P_W is defined as primitives that 'have been empirically validated' under the WAM, but the validation procedure is never described, nor is the actual set of validated primitives used for LingBot-VA reported. Because the executable-space projection is central to the paper's contribution and its ablation (Table 3) shows a 41.1-point drop when removed, the reader cannot assess whether the benefit comes from the projection mechanism itself or from the particular choice of supported primitives. Please specify how skills are validated, list the resulting P_W, and state whether the same set is used across benchmarks and ablations.","section":"Section 3.3, Eq. (11)-(12)"}],"minor_comments":[{"comment":"The notation 'WAM' and 'W AM' is used inconsistently (e.g., the abstract and Section 1 contain both forms). Please standardize the spelling.","section":"Throughout"},{"comment":"The sentence 'We report macro-averages across tasks and rollouts' is ambiguous; clarify whether the macro-average is taken over task-level success rates first and then averaged over tasks, or over all task-rollout pairs.","section":"Section 4.1"},{"comment":"The six loss weights (lambda_r, lambda_b, lambda_rank, lambda_e, lambda_s, lambda_m) in the progress-estimator objective are introduced but their values or tuning procedure are not reported. Please state the settings or note that they are held fixed, and report how sensitive the event-triggering behavior is to these weights.","section":"Section 3.4, Eq. (15)"},{"comment":"The keyframes would benefit from explicit event labels (e.g., 'evidence binding', 'recovery', 'task advance') so that the qualitative rollout directly illustrates the evidence-retention and recovery claims rather than requiring the reader to infer them from the narrative.","section":"Figure 3"},{"comment":"No code, checkpoints, or data-release statement is provided. Adding an availability statement would support reproducibility, especially for the plan-level diagnostics in Table 4.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The internal ablation evidence is credible and the contribution is substantial, but the state-of-the-art claim rests on protocol-incomparable baselines and small margins without uncertainty quantification. I recommend requesting protocol-matched baseline comparisons (or at least a clear statement of protocol differences) and confidence intervals for the main comparisons, and possibly tempering the abstract's SOTA claim if the baselines cannot be re-run."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid systems paper that deserves referee time, but the headline SOTA numbers rest on unverified baseline comparability, and the RoboCerebra margin is within noise.\n\nWhat's actually new: the prediction-deliberation gap framing, and the capability-conditioned executable-space projection that compiles VLM plans into WAM-supported skills with dependency and embodiment checks. The architecture is clearly specified, and the event-driven dual-timescale loop is a sensible way to keep VLM calls sparse. The same-WAM ablations are the strongest part: WAM + Whole Task, WAM + Static Plan, and the five ablated variants in Table 3 convincingly show that the harness, not the WAM weights, drives the RoboMemArena gains. The plan-level diagnostics (Table 4) also do real work, distinguishing lexical normalization from actual executable projection.\n\nWhere it gets soft: the paper states that HarnessWAM runs 20 rollouts with matched initial states, seeds, observation interfaces, and budgets, but external baselines are 'taken from the corresponding benchmark evaluations.' If those baselines used different protocol details, the +21.1-point gain over PrediMem and the +1.78-point gain on RoboCerebra could be artifacts. That matters especially on RoboCerebra Ideal, where the margin over GPT-4o Planner + OpenVLA is small and there are no error bars. No code, prompts, or training details are released, so independent verification isn't possible right now. Those are fixable weaknesses, not fatal ones.\n\nThe free parameters listed are just hyperparameters; they aren't fitted to inflate success. The progress estimator is a trained module, but it only triggers events; the Task Manager decides transitions, so the circularity burden is low.\n\nBottom line: this is a paper for people building agentic wrappers around predictive policies or WAMs. The central design claim -- that a model-external structured harness improves long-horizon memory-heavy manipulation -- is supported by the internal comparisons. The SOTA claim needs matched baselines, confidence intervals, and at least partial artifacts before it's verifiable. I'd send it to peer review, with a request to harden the empirical comparison. If I worked on WAM planning, I'd cite it for the projection idea and the ablation methodology.","headline":"A well-specified harness for WAMs with strong internal ablations; the SOTA claim is real only if the external baselines were protocol-matched.","tokens_in":15214,"tokens_out":1787,"would_cite":true,"duration_ms":14830,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A model-external harness turns finite-horizon world-action models into reliable long-horizon robot agents.","keywords":["world action models","prediction-deliberation gap","embodied task planning","robot manipulation","closed-loop control","scene belief","failure recovery","vision-language model"],"falsifier":"Re-run PrediMem and GPT-4o Planner + OpenVLA under HarnessWAM's exact protocol (20 rollouts, same initial states, seeds, observation interfaces, and task-level budgets) and check whether HarnessWAM's 59.6% and 23.7% margins persist; if they shrink or reverse, the state-of-the-art claim fails.","tokens_in":14119,"feed_emoji":"🤖","tokens_out":6399,"duration_ms":49025,"temperature":0.7,"pith_summary":"The paper argues that World Action Models (WAMs), which jointly predict physical dynamics and generate robot actions, fail at long-horizon embodied tasks because their predictions are finite-horizon and local; the authors call this the prediction–deliberation gap. To close it, HarnessWAM adds a model-external agentic runtime that keeps an evidence-grounded scene belief and task graph, compiles open-ended language plans into WAM-executable skill sequences, and runs an event-driven dual-timescale loop of progress monitoring and semantic deliberation. On two benchmarks, RoboMemArena and RoboCerebra Ideal, the framework reports state-of-the-art full-task and subtask success rates of 59.6% and 69.9%, and an SR of 23.7%. If these results hold, the implication is that reliable multi-stage manipulation can come from orchestrating a WAM from the outside, rather than from changes to the model's weights.","feed_headline":"External harness lifts world-action models to top robot scores","feed_subtitle":"Memory, planning, and recovery close the gap between local prediction and long-horizon manipulation.","key_machinery":"The load-bearing mechanism is the separation of continuous WAM control from discrete task-level deliberation, organized as a harness with four components. First, an evidence-grounded scene belief represents facts as (subject, predicate, object, value, epistemic status, confidence, visual evidence), so occluded objects are not treated as absent. Second, a structured task graph contains motor and cognitive nodes with preconditions, expected effects, termination conditions, and recovery strategies, keeping unresolved entities as symbolic variables until evidence binds them. Third, capability-conditioned executable-space projection maps open-ended VLM plans onto the validated primitive set of the WAM, checking argument types, dependencies, preconditions, single-arm holding state, and acyclicity; semantic nodes that admit a composition of supported primitives are expanded, and infeasible projections return ⊥ to trigger replanning. Fourth, an event-driven, dual-timescale loop lets a lightweight progress estimator provide high-frequency progress and completion cues while the VLM Task Manager deliberates only at milestones, with saved joint and gripper states enabling embodiment-only recovery after local failures.","core_discovery":"The central claim is that finite-horizon prediction and action generation are insufficient for tasks that need global planning, cross-stage state maintenance, execution verification, and failure recovery, and that this gap can be closed by a structured runtime external to the WAM. HarnessWAM maintains a scene belief whose facts carry epistemic status (observed, inferred, unknown) and a task graph with symbolic variables bound only when visual evidence supports them. A deterministic projection compiles VLM-proposed semantic plans into the subset of primitives the WAM can actually execute, enforcing dependencies, argument types, and embodiment constraints. During execution, a lightweight progress estimator feeds a fast loop while a VLM Task Manager deliberates at milestones, deciding to advance, observe, replan, recover, or terminate; recovery restores only the robot embodiment, keeping scene knowledge and task memory. The paper reports that this design achieves the best scores on RoboMemArena and RoboCerebra Ideal, and its ablations attribute the largest gain to executable-space projection.","pith_inferences":["This harness design could generalize to other finite-horizon visuomotor policies beyond WAMs, though the paper only tests it with one underlying WAM, so the generality is an open implication rather than a demonstrated result.","Capability-conditioned executable-space projection could serve as a pre-execution vetting step for VLM planners in robotics generally; the paper's plan-level diagnostics show raw VLM plans are only 13.8% executable, suggesting many wasted rollouts could be prevented before execution.","The dual-timescale loop may yield compute savings in addition to reliability, since the VLM Task Manager is invoked only at milestones rather than at every step; the paper does not report compute cost, making this a testable extension.","The epistemic-status representation (observed vs inferred vs unknown) could transfer to other partially observable settings such as household robots or autonomous inspection, where distinguishing 'not observed' from 'observed to be false' is critical."],"forward_implications":["Static decomposition of a task at initialization is not enough: WAM + Static Plan beats WAM + Whole Task by 3.5 points full-task, but HarnessWAM adds another 11.7 points, so persistent task state and closed-loop management are necessary.","Executable-space projection is the largest single contributor; removing it drops full-task success from 59.6% to 18.5%, and plan-level diagnostics show it raises plan executability from 42.3% to 72.9%.","Progress-only switching, without semantic verification of outcomes, keeps subtask success similar but lowers full-task success from 59.6% to 55.4%, with a notable drop on sequential execution.","Local recovery that preserves scene knowledge and task memory improves average full-task success by 5.4 points, with the largest effect on sequential tasks.","The framework works on a fully observable benchmark (RoboCerebra Ideal) as well as a memory-heavy one, indicating benefits beyond explicit memory recovery."],"supporting_citations":[{"why":"Supplies the underlying WAM (LingBot-VA) whose skills HarnessWAM projects onto and which is fine-tuned per benchmark, isolating the harness's contribution.","marker":"Li et al., 2026"},{"why":"Defines the RoboMemArena benchmark and provides the PrediMem baseline that HarnessWAM outperforms by 21.1 full-task points.","marker":"Lei et al., 2026"},{"why":"Defines the RoboCerebra Ideal benchmark and the HPE Framework baseline, establishing the fully observable testbed and comparison.","marker":"Han et al., 2026"},{"why":"Establishes that external harnesses shape foundation-model capability in language agents, motivating the harness concept for embodied settings.","marker":"Yao et al., 2022"},{"why":"Provides the reflexion-style feedback loop that HarnessWAM adapts to continuous embodiment with physical recovery.","marker":"Shinn et al., 2023"},{"why":"Supplies the pi0.5 baseline scores on both RoboMemArena and RoboCerebra Ideal.","marker":"Intelligence et al., 2025"},{"why":"Supplies the OpenVLA baseline used in the GPT-4o Planner + OpenVLA comparison on RoboCerebra Ideal.","marker":"Kim et al., 2024"},{"why":"Provides empirical evidence that a small set of grasp types dominates natural manipulation, supporting the compact primitive ontology.","marker":"Bullock et al., 2013"}],"fun_headline_variants":["HarnessWAM closes robot planning gap with structured memory","World-action models get memory boost, top robot benchmarks","Agentic harness makes world-action models plan and recover","External loop turns world-action models into planners","Structured harness lifts world-action models to top scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The published baseline numbers are taken from the benchmarks' own evaluations and are assumed comparable to HarnessWAM's 20-rollout protocol with matched initial states, seeds, observation interfaces, and task-level budgets.","fun_headline_variants_meta":{"raw":{"variants":["HarnessWAM closes robot planning gap with structured memory","World-action models get memory boost, top robot benchmarks","Agentic harness makes world-action models plan and recover","External loop turns world-action models into planners","Structured harness lifts world-action models to top scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1570,"prompt_tokens":1071,"completion_tokens":499,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":687,"completion_tokens_details":{"reasoning_tokens":424}},"tokens_in":687,"tokens_out":499,"duration_ms":4511,"temperature":1.0,"reasoning_tokens":424,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:47:52.574416+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run PrediMem and GPT-4o Planner + OpenVLA under HarnessWAM's exact protocol (20 rollouts, same initial states, seeds, observation interfaces, and task-level budgets) and check whether HarnessWAM's 59.6% and 23.7% margins persist; if they shrink or reverse, the state-of-the-art claim fails.","supporting_citations":[{"cited_title":"Memoryvla: Perceptual-cognitive memory in vision-language- action models for robotic manipulation","cited_arxiv_id":null,"evidence_quote":"Defines the RoboCerebra Ideal benchmark and the HPE Framework baseline, establishing the fully observable testbed and comparison."},{"cited_title":"React: Synergizing reasoning and acting in language models","cited_arxiv_id":null,"evidence_quote":"Establishes that external harnesses shape foundation-model capability in language agents, motivating the harness concept for embodied settings."}],"review_version":1}