{"id":"03da5b17-f683-4e03-95da-c69f7a94a56a","arxiv_id":"2608.06756","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An execution-centric training recipe that specializes a vision-language model into four capability experts and consolidates them via weight merging and distillation yields a single embodied model with broad benchmark gains.","lead":"The paper trains four specialist vision-language models, one for each of spatial, temporal, action-guidance, and state-verification skills, then merges them into one model that runs embodied tasks. It reports that the combined model beats its starting checkpoint on most benchmarks and improves simulated robot task success.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Capek-StateBench may overlap with the BEHAVIOR-1K trajectories used for State Verification training, so the reported StateBench gains could reflect memorization rather than a genuine capability.","rationale":"After reviewing the manuscript, the most load-bearing concern is exactly the one identified by the reader: Capek-StateBench is constructed from the same BEHAVIOR-1K source as the State Verification training data, and the paper provides no evidence of disjointness. This is central because the paper's claim to 'retain all four specialized capabilities in one checkpoint' rests on the StateBench results. Notably, the VIGIL state-verification diagnostic (Table 6) shows Capek 0.5 at 96.0/84.8 W/B versus Qwen's 96.0/85.6, so Capek-StateBench is the only place where a State Verification improvement appears. If the benchmark overlaps with training trajectories, the gains are a measure of memorization, not capability acquisition. The benchmark scoring function (Eq. 9-10) is also nearly identical to the training reward (Section 4.1.2), further tightening the self-referential loop. This does not collapse the entire paper: the large, consistent Action Guidance improvements on external benchmarks, the controlled consolidation ablation, and the overall closed-loop gains on EmbodiedBench and VIGIL are independent evidence that the specialist-to-unified recipe works. However, the specific capability claim for State Verification is unsupported without a disjoint evaluation. I would therefore keep the CONDITIONAL verdict, requiring the authors to demonstrate disjointness or provide external state-verification evaluation. The leak concern is concrete and checkable; other issues (lack of error bars, no code release) are real but secondary. The stress-test confirms the reader's weakest-assumption analysis, so no verdict change is needed.","tokens_in":35078,"tokens_out":8283,"duration_ms":73770,"concrete_test":"Compute the intersection of BEHAVIOR-1K task-instance IDs and checkpoint timestamps or frame hashes between the State Verification training set (Section 3.4) and the 500 Capek-StateBench-P and 500 Capek-StateBench-T items. If any overlap is found, rerun the StateBench evaluation on a fully disjoint held-out set (e.g., held-out BEHAVIOR-1K activities or a different simulator) and compare Capek 0.5 against Qwen on that subset; if the gains vanish, the claim that the state-verification capability was acquired and retained is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 states that \"We construct all State Verification post-training data from BEHAVIOR-1K household-manipulation trajectories.\" Section 5.1.1 and Appendix A.2 describe Capek-StateBench as a benchmark built from the same BEHAVIOR-1K goal-condition and primitive-skill decompositions, with progress definitions identical to the training targets (Eq. 11 vs. Section 3.4). The paper never demonstrates that the benchmark episodes, checkpoints, or annotated frames are disjoint from the training trajectories, nor that the next-action targets and progress labels are not drawn from the same annotated runs. This is especially damaging because the benchmark's primary score (Eq. 9-10, s_T = (5s_p + 4s_a)/9) mirrors the training reward (Section 4.1.2: R_prog+act = 0.5 r_value + 0.4 r_action + 0.1 r_fmt), so the evaluation is essentially re-testing the training objective. If overlap exists, the claimed State Verification gains (StateBench-P +11.0, StateBench-T +2.45) are inflated. The VIGIL state-verification diagnostic (Table 6) shows no improvement over the Qwen backbone (96.0/84.8 vs 96.0/85.6 W/B), making Capek-StateBench the sole evidence for this capability family. The general deduplication statement in Section 3.5 applies to training-data construction and does not explicitly cover the new benchmark, so no disjointness guarantee is on record.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Capek 0.5, a vision-language model for embodied reasoning whose post-training is organized around four execution-centric capability families: Spatial Reasoning, Temporal Understanding, Action Guidance, and State Verification. For each capability, a specialist is trained from a shared Qwen backbone using GRPO with verifiable rewards; the specialists are then consolidated into a single inference-time model by TIES weight-space merging followed by routed multi-teacher on-policy distillation. The authors instantiate the recipe at 2B and 35B-A3B scales and evaluate it with three complementary protocols: a large public-benchmark suite augmented by the new Capek-StateBench, a controlled consolidation ablation comparing specialists, Mix-RL, TIES, MOPD, and TIES+MOPD, and closed-loop simulated evaluation in EmbodiedBench and VIGIL. The main empirical claims are that Capek 0.5 improves 28/34 and 30/34 matched benchmark rows over its Qwen initialization, retains all four specialized capabilities in one checkpoint, and transfers to closed-loop execution, with the largest gains on action-guidance and state-verification tasks.","tokens_in":35297,"tokens_out":5390,"duration_ms":52772,"significance":"If the benchmark evidence holds, the paper provides a practical and unusually auditable template for building unified embodied VLMs: same-origin specialists trained under capability-aligned verifiable rewards, followed by weight-space and policy-space consolidation. The evaluation design has genuine strengths: matched Capek–Qwen pairs share decoding settings, all external baseline entries are DeepInsight reruns rather than quoted numbers, the consolidation ablation compares all variants from the same backbone with identical prompts and scorers, and the closed-loop experiments are held out from specialist training. The strongest positive evidence is the consistent improvement across action-guidance and temporal benchmarks at both scales. However, the State Verification claim currently rests on a benchmark that may overlap with the training data, and the headline row-count claim is made without any uncertainty quantification. These issues are local in the sense that they can be addressed with additional analysis, but they are load-bearing for the paper's central contribution, so the appropriate outcome is a major revision.","major_comments":[{"comment":"The paper does not demonstrate that Capek-StateBench is a clean held-out evaluation. Section 3.4 states that all State Verification post-training data are constructed from BEHAVIOR-1K trajectories, and Section 5.1.1/Appendix A.2 build Capek-StateBench from the same BEHAVIOR-1K goal-condition and primitive-skill decompositions, with progress defined by Eq. (11) and scored by Eq. (9) using the same 25-point truncated proximity as the training reward r_value in Section 4.1.2. The manuscript never shows that the benchmark episodes, checkpoints, annotated frames, next-action targets, or progress labels are disjoint from the training trajectories; the general deduplication statement in Section 3.5 is about training-data construction and does not explicitly cover the new benchmark. This is consequential because the VIGIL state-verification diagnostic in Table 6 shows no improvement over the Qwen backbone (96.0/84.8 vs 96.0/85.6 W/B), leaving Capek-StateBench as the sole positive evidence for the State Verification capability family. The authors should provide a formal disjointness analysis (episode-, checkpoint-, and frame-level overlap counts, plus label provenance for next-action and progress targets) or replace/reinforce Capek-StateBench with an independently constructed external state-verification benchmark.","section":"§3.4, §5.1.1, Appendix A.2"},{"comment":"The headline claim that Capek 0.5 improves 28 of 34 and 30 of 34 matched rows is based on single evaluation runs without error bars, significance tests, or multiple decoding seeds, even though the inference configuration uses temperature 0.7 sampling. Several decisive row-wise differences are small (e.g., CV-Bench 88.56→89.99, MMBench-EN 92.35→92.40, RealWorldQA 82.09→83.40 at 35B-A3B), so the row-count claim is not robustly established for close rows. At minimum, the paper should report confidence intervals or bootstrap estimates for the matched Capek–Qwen pairs and should rerun the closest rows with multiple seeds; ideally this should also cover the EmbodiedBench and VIGIL success rates in Tables 5 and 6, where the reported gains are based on 300- and 1000-episode sets respectively but still lack any variance estimate.","section":"§5.1.3, §5.1.4, Tables 2 and 3"},{"comment":"The task-track benchmark score s_T = (5s_p + 4s_a)/9 in Eq. (10) is a near re-test of the training reward R_prog+act = 0.5 r_value + 0.4 r_action + 0.1 r_fmt in Section 4.1.2: both use the same 25-percentage-point truncated proximity for progress and the same semantic next-action matching. Metric alignment is acceptable when labels are truly held out, but the paper should state this explicitly and explain why the measured StateBench-T gain (+2.45 at 35B-A3B) reflects acquired capability rather than reward overfitting to the training objective. The absence of a corresponding gain on the VIGIL state-verification probe makes this concern concrete rather than hypothetical.","section":"§4.1.2 vs §5.1.1, Eq. (10)"}],"minor_comments":[{"comment":"Please clarify the size of Capek-StateBench: Appendix A.2 says the benchmark 'contains two 500-example tracks,' while Section 5.1.1 says 'The released 500-example split combines both schemes, with 213 task-condition records and 287 primitive-skill records.' The reader cannot tell whether each track has 500 examples or whether the 213/287 split is the entire task track.","section":"§5.1.1 and Appendix A.2"},{"comment":"The Mix-RL baseline is described only as 'data-mixing reinforcement learning.' Please specify what is mixed (all four specialist corpora in one GRPO run, the same total number of samples, and the same KL and reward settings) and give its training budget, so the comparison against TIES/MOPD is interpretable.","section":"Table 4"},{"comment":"The symbol M is used both for the total number of compared waypoints and as an index in the sum; please use a separate symbol for the index or restate the sum over waypoint pairs to avoid notational collision.","section":"Appendix A.3, Eq. (15)"},{"comment":"The bar chart uses many closely related colors that are hard to distinguish in print; adding hatched patterns or marker labels would make the five-domain comparison readable.","section":"Figure 1"},{"comment":"The text says next-step prediction is evaluated by 'semantic exact matching,' but Appendix A.2 says a 'fixed semantic judge' is used; please state whether this judge was frozen before the evaluation and whether the same judge was used during training rewards, since a learned judge could introduce a shared error surface.","section":"§5.1.1"}],"recommendation":"major_revision","confidential_remarks":"The central recipe is plausible and the controlled same-origin comparison is a genuine strength, but the State Verification evaluation currently has a potential train/evaluation overlap that directly affects a claimed capability family. I would ask for a leakage analysis or an external held-out benchmark before publication, and for uncertainty estimates on the close matched rows. The paper's scope fits the journal well; the main risk is that the State Verification gains are cited as evidence for the taxonomy even though the only positive evidence is a benchmark built from the same source as the training data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Capek 0.5 is a serious engineering-and-recipe paper. What's new is real: organizing post-training around capability families (Spatial, Temporal, Action Guidance, State Verification), training same-origin specialists with GRPO and typed verifiable rewards, then consolidating via TIES initialization plus routed MOPD. The controlled comparison of Mix-RL, TIES alone, MOPD alone, and TIES+MOPD is exactly the kind of ablation this literature needs. The benchmark table is also unusually clean: every entry is re-run under one harness, matched Capek-Qwen pairs share decoding settings, and the 2B track replicates the 35B-A3B pattern. The EmbodiedBench and VIGIL closed-loop gains are genuinely encouraging and independent of the in-house benchmark.\n\nThe soft spot is real and load-bearing for one of the four capability claims. Capek-StateBench is built from BEHAVIOR-1K, the same source as all State Verification training data, and the paper gives no disjointness guarantee. Worse, the task-track progress score (Eq. 9) is formally identical to the PVE training reward (r_value in Section 4.1.2), so StateBench-T is re-testing the training objective. The VIGIL state-verification probe shows no gain over the Qwen backbone (96.0/84.8 W/B vs 96.0/85.6), leaving Capek-StateBench as the sole evidence for that capability family. If benchmark episodes overlap training runs, the reported +11.0 and +2.45 gains are inflated.\n\nThat does not sink the paper. The spatial, temporal, and guidance results mostly come from external benchmarks, the consolidation study is well-designed, and the closed-loop gains stand independently. But the State Verification section needs to be rebuilt or re-scoped: the authors should release the benchmark with explicit train/test disjointness, add an externally sourced state-verification test, and confirm with the VIGIL probe. Also missing: any variance or significance analysis (single runs throughout), and no code, data, or weights despite the DeepInsight evaluation infrastructure.\n\nWho is this for? Embodied-VLM researchers working on multi-skill post-training; the consolidation recipe is directly usable and the capability-taxonomy framing will be cited. A serious referee should engage with it, but the revision needs to be substantive before I would trust the State Verification numbers.","headline":"State Verification results rest on a benchmark that mirrors the training reward, but the consolidation recipe and closed-loop results are solid enough to warrant a serious look.","tokens_in":663,"tokens_out":667,"would_cite":false,"duration_ms":26811,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Post-training a robot vision-language model by execution capability, not by dataset, improves most benchmarks and transfers to simulated robot execution.","keywords":["embodied vision-language model","capability taxonomy","reinforcement learning","GRPO","model merging","TIES","on-policy distillation","state verification"],"falsifier":"Check the 500 Capek-StateBench samples against the State Verification training corpus for shared episode identifiers, checkpoint timestamps, or evidence frames; if any overlap exists, measure StateBench-P and StateBench-T after retraining the State specialist on a provably disjoint split, and see whether the +11.0 and +2.45 point gains survive.","tokens_in":34752,"feed_emoji":"🤖","tokens_out":9468,"duration_ms":77504,"temperature":0.7,"pith_summary":"Vision-language models for robots typically pile post-training data from many tasks into one model, making it hard to tell which capabilities were acquired or kept. Capek 0.5 instead organizes post-training around four capabilities that every robot execution repeatedly needs—spatial reasoning, temporal understanding, action guidance, and state verification—trains a separate specialist for each from the same backbone with reinforcement learning and verifiable rewards, then merges and distills the specialists into one checkpoint. The paper reports that this recipe improves the large majority of matched benchmarks over its Qwen start at both 2B and 35B-A3B scales, keeps all four capabilities in a single model, and raises closed-loop simulated task success. If the recipe is right, it offers a practical template for building unified embodied vision-language models without task-specific heads or routing at inference.","feed_headline":"Capability-first training lifts a robot VLM on 28 of 34 benchmarks","feed_subtitle":"One checkpoint keeps four specialist skills and lifts closed-loop robot success in simulation.","key_machinery":"The load-bearing machinery is the execution-centric capability taxonomy plus the specialist-to-unified consolidation pipeline. Four specialists (Spatial Reasoning, Temporal Understanding, Action Guidance, State Verification) are post-trained from one shared backbone with token-level GRPO and capability-specific verifiable rewards; their parameter updates relative to that backbone are task vectors. A TIES merge resolves conflicting updates by trimming low-magnitude entries, electing a per-coordinate consensus sign, and averaging sign-consistent updates, producing the unified student's initialization. Routed MOPD then distills each specialist into that student on the student's own sampled prefixes, with the responsible teacher queried per sample and all teachers frozen; this is what makes the four behaviors compatible in a single autoregressive checkpoint.","core_discovery":"On its own terms, the paper's central discovery is that capability acquisition and capability consolidation can be separated cleanly: four same-origin specialists—trained with GRPO on capability-aligned data under verifiable rewards—define what the unified model should preserve, and a TIES weight-space merge followed by routed multi-teacher on-policy distillation defines how to consolidate them. The paper reports that this two-stage recipe improves 28 of 34 matched benchmark rows (35B-A3B track) and 30 of 34 (2B track) over the shared Qwen initialization, retains all four specialized capabilities in one checkpoint with quantified losses, and transfers to closed-loop simulation, with EB-HAB success rising from 46.0 to 63.0 and VIGIL benchmark success from 28.8 to 32.2. The claim is that execution-centric organization—grouping supervision by functional role rather than dataset—is what carries these gains, not any single benchmark or task.","pith_inferences":["An implicit consequence is that the same specialist-to-unified pipeline should extend to new capability families, such as tool use or skill orchestration, by adding new task vectors and new MOPD routes without changing the consolidation machinery.","A testable extension would be to swap the verifiable reward functions for open-vocabulary or 3D action outputs; the taxonomy and merge-then-distill structure would remain, but the gains would hinge on whether the new rewards capture the capability as cleanly.","The large closed-loop gains on long-horizon and complex-instruction subsets suggest the binding constraint in embodied VLMs may be coordinating multiple capabilities over many steps rather than any single perception or reasoning skill.","The paper's comparison treats the recipe as the cause of improvement, but a cleaner attribution would require ablating the taxonomy itself (e.g., grouping the same data by dataset rather than capability) while holding the consolidation method fixed."],"forward_implications":["At both 2B and 35B-A3B scales, the unified model improves the majority of matched benchmark rows over its Qwen initialization, including every Action Guidance row at both scales.","Consolidating via TIES plus MOPD retains all four specialist capabilities with smaller losses than either merging alone or mixing all data into a single RL run.","The gains carry into closed-loop simulated execution, with higher average success than the backbone on EmbodiedBench and higher benchmark success on VIGIL.","Inference uses a single autoregressive checkpoint with no task-specific heads and no routing, so the taxonomy and its specialist routes are training-time apparatus only.","The unified output interface—points, boxes, intervals, trajectories, verdicts, progress JSON—lets all four capability families share one backbone and one tokenizer."],"supporting_citations":[{"why":"supplies the 35B-A3B backbone that all four specialists start from and that Capek 0.5-35B-A3B is measured against.","marker":"[29]"},{"why":"supplies the 2B dense backbone for the compact Capek 0.5-2B track.","marker":"[30]"},{"why":"provides the TIES weight-merging algorithm that initializes the unified student from the four specialist task vectors.","marker":"[28]"},{"why":"provides the routed multi-teacher on-policy distillation method that consolidates specialist behaviors into the single student.","marker":"[26]"},{"why":"provides GRPO, the group-relative policy-optimization algorithm used to train every specialist with verifiable rewards.","marker":"[61, 62]"},{"why":"BEHAVIOR-1K trajectories are the source of all State Verification training data and of the Capek-StateBench benchmark.","marker":"[25]"},{"why":"EmbodiedBench is the closed-loop simulated evaluation showing the unified model's task-success gains over the backbone.","marker":"[23]"},{"why":"VIGIL is the frozen embodied-agent benchmark used to test how the four capabilities compose during interaction.","marker":"[31]"}],"fun_headline_variants":["Capability-first VLM lifts 28 of 34 robot benchmarks","Capability-first VLM: 28/34 robot benchmarks improved","One model, four skills: Capek 0.5 beats 28/34 benchmarks","Capek 0.5: capability-first VLM wins on 28/34 benchmarks","Four specialists distill into one VLM: 28/34 benchmarks up"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Capek-StateBench, the new state-verification benchmark, is genuinely held out from training: both it and the State Verification training data are built from BEHAVIOR-1K trajectories, and the paper does not demonstrate that benchmark episodes, checkpoints, or frames are disjoint from the training runs, so if they overlap the reported state-verification gains would be partly memorization.","fun_headline_variants_meta":{"raw":{"variants":["Capability-first VLM lifts 28 of 34 robot benchmarks","Capability-first VLM: 28/34 robot benchmarks improved","One model, four skills: Capek 0.5 beats 28/34 benchmarks","Capek 0.5: capability-first VLM wins on 28/34 benchmarks","Four specialists distill into one VLM: 28/34 benchmarks up"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000956,"raw_usage":{"total_tokens":4111,"prompt_tokens":1020,"completion_tokens":3091,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":2988}},"tokens_in":636,"tokens_out":3091,"duration_ms":19220,"temperature":1.0,"reasoning_tokens":2988,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:13:00.908785+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the 500 Capek-StateBench samples against the State Verification training corpus for shared episode identifiers, checkpoint timestamps, or evidence frames; if any overlap exists, measure StateBench-P and StateBench-T after retraining the State specialist on a provably disjoint split, and see whether the +11.0 and +2.45 point gains survive.","supporting_citations":[{"cited_title":"Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents","cited_arxiv_id":"2605.08747","evidence_quote":"VIGIL is the frozen embodied-agent benchmark used to test how the four capabilities compose during interaction."}],"review_version":1}