{"id":"456c0dec-6ce4-473e-b562-fc03c3d1e613","arxiv_id":"2505.03912","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"OpenHelix shows that a frozen vision-language model with a prompt-tuned token and an auxiliary action-prediction head beats full fine-tuning on CALVIN language generalization while training far fewer parameters.","lead":"This paper surveys dual-system vision-language-action (VLA) robot policies and runs controlled comparisons in the CALVIN simulator. It proposes OpenHelix, which keeps the large model frozen and uses a learnable token plus an auxiliary action head to guide a fast diffusion policy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The AUX visual-grounding mechanism is asserted but never tested: the paper's own Fig. 5 diagnostic is not re-run on the final model, so the central explanation for the Table 8 gains is unsupported.","rationale":"The reader's weakest assumption identified exactly this gap, and I agree. The paper's central contribution is not merely a set of CALVIN numbers; it is a specific mechanism (auxiliary task forces visual grounding) that motivates the architecture. The empirical comparison in Table 8 shows an improvement, but the paper's own earlier diagnostic (Figure 5) demonstrated that the untrained latent does not track visual changes, and no post-AUX diagnostic is provided. Without a visual-perturbation control, the most plausible alternative explanations (multi-task regularization, language prior, or the confounded Asy(10)) are not ruled out. The proposed test is cheap and uses the paper's own methodology. I do not see a reason to change the reader's conditional verdict: the concern is real but the underlying numbers may still hold; it needs the missing control before the explanatory claim can be accepted. Secondary issues (100-trial ablations without error bars, missing code artifacts, missing Helix/CALVIN citations) reinforce the conditional status but are not the primary load-bearing point.","tokens_in":14187,"tokens_out":7897,"duration_ms":82421,"concrete_test":"Re-run the Figure 5 latent-probing experiment on the final AUX model: fix the instruction 'Can you control the robot to take the blue block and rotate it to the right?', but at environment step 3 manually shift the blue block left; record z_<ACT> before and after the shift and compute the top-10 nearest words and P(left|z) vs P(right|z) using the same semantic-space mapping as Section 2.4. If the spatial probabilities track the moved object with AUX (whereas without AUX they stayed tied to 'right'), the visual-grounding mechanism is supported. If they remain instruction-bound, the auxiliary head is solved from language/priors and Section 3.2's explanation fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Section 3.2's claim that the auxiliary action head 'ensures the large model has to utilize visual input information.' This is the design rationale for the proposed model and the paper's explanation for the Table 8 improvements. It is not tested. The auxiliary head (Eq. 1) is trained to predict ground-truth actions from z_<ACT> in CALVIN, where actions are strongly predictable from instruction semantics and static-scene priors; nothing forces the frozen MLLM to use the image. Section 2.4 and Figure 5 show that, before adding AUX, z_<ACT> is largely insensitive to scene changes and mostly encodes instruction semantics. After adding AUX, the paper never re-runs that diagnostic; no visual-perturbation control and no final-model CALVIN-D dynamic results are reported. The gains in Table 8 (e.g., 5-of-5 46.0 vs 41.2 on CALVIN; 20.2 vs 6.1 on CALVIN-E) could therefore come from multi-task regularization, better use of language priors, or the concurrent Asy(10) change, rather than from genuine visual grounding. Because the architecture's motivation is specifically to make z_<ACT> multimodal, the central causal claim is unsupported as written. This is a correctness risk for the paper's interpretation, not an internal inconsistency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper combines a short survey of dual-system vision-language-action (VLA) architectures with an empirical study of design choices and a proposed method called OpenHelix. The empirical sections compare policy training paradigms, MLLM training strategies (frozen, fine-tuning, prompt tuning), projector pre-alignment, and asynchronous inference on the CALVIN benchmark, including a language-generalization variant (CALVIN-E) and a dynamic-scene variant (CALVIN-D). The proposed OpenHelix system keeps a LLaVA-7B MLLM frozen, optimizes only a learned <ACT> prompt embedding and a linear projector, and adds an auxiliary action-prediction head on top of the <ACT> latent to 'force' the MLLM to use visual information. The main result is reported in Table 8, where the auxiliary task plus 10-step asynchronous inference improves 5-of-5 success on CALVIN to 46.0% and on CALVIN-E to 20.2%.","tokens_in":14512,"tokens_out":5592,"duration_ms":51692,"significance":"If the central claims hold, the paper would provide a useful low-cost recipe for dual-system VLA design: a frozen 7B MLLM with a single trainable prompt token plus a lightweight auxiliary head, together with an empirical map of which design dimensions matter. The controlled comparison across MLLM training strategies and the two-stage pre-alignment finding (Table 6) are practically valuable, and the asynchronous-inference insensitivity result (Figure 4) is an interesting negative result for the community. The paper is also honest about its limitations and future-work status. However, the significance is materially reduced by three load-bearing gaps: the causal mechanism for the auxiliary task is asserted but not tested, the ablation evidence is drawn from a 100-trial subset without seeds or confidence intervals, and the headline Table 8 comparison confounds the auxiliary task with a change in asynchronous setting. These issues make the empirical contribution currently insufficient to support the paper's main design rationale.","major_comments":[{"comment":"The central causal claim that the auxiliary action-prediction head 'ensures that the large model has to utilize visual input information' is not tested. Section 2.4 and Figure 5 diagnose, on a model without AUX, that the <ACT> embedding is largely insensitive to scene changes and mostly encodes instruction semantics. The paper never re-runs this diagnostic on the final model with AUX, and no visual-perturbation or information-bottleneck control is reported. Because the auxiliary head is trained on the same ground-truth actions used by the policy and CALVIN scenes are near-static, the gains in Tables 7 and 8 could arise from multi-task regularization or better use of language priors rather than from genuine visual grounding. Please add a direct test: for example, re-run the Figure 5 embedding analysis on the AUX-equipped model, or evaluate with the input image perturbed (masked, shifted, or replaced) and measure whether the <ACT> embedding and downstream success change accordingly.","section":"Section 3.2, Eq. (1), Figure 5"},{"comment":"All ablation experiments are evaluated on the first 100 of the standard 1000 CALVIN evaluations, with no repeated seeds or reported confidence intervals. At 100 binary trials, the standard error of a 50% success rate is about 5 percentage points, so many reported differences (e.g., 94 vs 90 in Table 4, 94 vs 81 on CALVIN-E 1-of-5 in Table 5, 77 vs 72 in Table 7) are within one standard error. The representativeness of the first 100 trials for the full 1000-trial distribution is not checked. Please report ablations over at least three seeds (or binomial confidence intervals) and show that the 100-trial subset orders the design choices in the same way as the full evaluation.","section":"Section 2.1, Tables 3-7"},{"comment":"The conclusion that 'additional auxiliary tasks are very helpful' is based on a comparison that confounds the auxiliary task with the asynchronous setting: 'MLLM (PT) + AUX + Policy(P) + Asy(10)' is compared against 'MLLM (PT) + Policy(P)', which is not Asy(10). An isolated ablation with and without AUX under the same asynchronous setting is needed to attribute the difference to the auxiliary task. Table 7 provides an unconfounded comparison, but it uses the 100-trial subset and lacks error bars. Please add a full-1000-trial comparison with Asy(10) both with and without AUX.","section":"Table 8, Section 3.3"},{"comment":"The claim that single-system models 'directly fail' in dynamic scenarios is based on a single model (RF) and is explicitly acknowledged by the authors as not completely rigorous because π0 and GR00T-N1 were not tested. This is acceptable as a preliminary observation, but the paper subsequently uses this result to motivate the dual-system architecture. Please either soften the conclusion to a hypothesis or add at least one additional single-system baseline on CALVIN-D.","section":"Section 2.2, Table 2"}],"minor_comments":[{"comment":"Cross-reference error: 'The specific results are shown in Table 4' should refer to Table 3, which reports the policy training strategy comparison; the subsequent 'In Table 3' analysis is correct but the setup sentence is wrong.","section":"Section 2.3.1, Tables 3 and 4"},{"comment":"'As we discussed in section 3.3' should be 'Section 2.4-2.5', since the earlier discussion of the MLLM's visual-insensitivity is in the empirical analysis, not in Section 3.3.","section":"Section 3.2"},{"comment":"The loss weights ω1, ω2, ω3, and ω4 are introduced as hyperparameters but their values are never reported; please specify them for reproducibility, since the auxiliary loss is the proposed contribution.","section":"Eq. (1) and Eq. (2)"},{"comment":"The abstract and title promise an 'open-source model', but the manuscript does not include code, weights, or a URL in the paper body; if the project page or GitHub repository contains them, please state this explicitly in the paper.","section":"Abstract and Section 4"},{"comment":"The third-row text in Figure 5 is garbled and difficult to read; please replace it with a cleaner rendering or a table of the top similar words.","section":"Figure 5"},{"comment":"References [14] and [15] are duplicates of the same 'Visual Instruction Tuning' paper; consolidate them into one citation.","section":"References [14] and [15]"},{"comment":"The heading 'Intuitive hypothesis' appears twice in this section; use distinct headings such as 'Motivation' and 'Additional hypothesis' for clarity.","section":"Section 2.3.2"},{"comment":"The 'Helix' row contains N/A entries and is not from this work's experiments; clarify in the table caption that Helix is included for comparison based on public descriptions, not on results reproduced here.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads more like an evolving technical report than a finished journal submission; the authors themselves state it is 'only our initial version.' The core empirical recipe may be useful, but the unsupported causal claim about visual grounding and the lack of statistical rigor would need to be addressed before the paper can be recommended. The missing experiments (re-running the Fig. 5 diagnostic on the final model, visual-perturbation controls, multi-seed ablations, and an unconfounded AUX comparison) are all feasible within the scope of a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a genuinely useful empirical study of dual-system VLA design choices, and the thing to know is that the proposed cheap recipe (frozen MLLM + prompt-tuned <ACT> token + auxiliary action head) looks strong on CALVIN language generalization in Table 8. But the paper's explanation for why the auxiliary head helps is asserted, not tested, and the final configuration confounds AUX with asynchronous inference.\n\nWhat it does well: this is the first systematic ablation I know of comparing policy-from-scratch vs fine-tuned policy, frozen vs fine-tuned vs prompt-tuned MLLM, projector pre-alignment, CLIP loss, and async step count under otherwise fixed settings. Tables 3-7 are easy to read and the conclusions match the data. The Fig. 5 diagnostic is the most interesting part: it shows that without AUX the <ACT> latent is largely instruction semantics and barely tracks scene changes. That is a concrete, falsifiable observation the community can build on. The projector pre-alignment finding (Table 6) is also practically important.\n\nSoft spots: the main problem is Section 3.2. The paper says the auxiliary task 'ensures that the large model has to utilize visual input information,' but there is no direct evidence. They never re-run the Fig. 5 diagnostic on the final model, and they report no visual-perturbation or information-bottleneck control. Because CALVIN actions are strongly predictable from instruction language plus near-static scenes, the AUX gains could come from multi-task regularization or better language-prior use. The Table 8 comparison compounds this: the full system adds AUX and Asy(10) at the same time, so you cannot attribute the improvement to AUX alone. There are also no seeds or error bars, ablations use a 100-trial subset without a representativeness check, and no code or configs are included in the manuscript despite the 'open-source' framing. The survey part is thin but clearly labeled. Citation-wise, the CALVIN environment itself is not cited, which is a real omission for a benchmark-driven paper.\n\nWho it's for: people actively designing dual-system VLAs. Despite the mechanistic overclaim, the empirical recipe is a useful data point. The paper deserves a serious referee: I would send it to review with major-revision expectations rather than desk-reject. Ask for seeds, an isolated AUX ablation, a re-run of the visual-grounding diagnostic on the final model, and released code. If those land, this becomes a solid conference-level contribution.","headline":"Useful dual-system VLA ablation with a plausible but untested mechanism for its auxiliary head; deserves major-revision peer review, not desk rejection.","tokens_in":15066,"tokens_out":3219,"would_cite":true,"duration_ms":32707,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dual-system VLA with a frozen 7B MLLM, a prompt-tuned <ACT> token, and an auxiliary action head matches or beats fine-tuned baselines on CALVIN while preserving language generalization.","keywords":["dual-system VLA","robotic manipulation","prompt tuning","auxiliary action prediction","latent bridging","CALVIN benchmark","frozen MLLM","asynchronous inference"],"falsifier":"Take the authors' released model, run CALVIN tasks while occluding or perturbing the visual input at test time, and record whether the <ACT> embedding and predicted actions change. If the embedding stays nearly constant and success does not drop, the auxiliary task did not create visual grounding; alternatively, train the auxiliary head on randomized action targets and see whether the gains persist.","tokens_in":13991,"feed_emoji":"🤖","tokens_out":4280,"duration_ms":37552,"temperature":0.7,"pith_summary":"This paper argues that expensive fine-tuning of the large vision-language model in a dual-system robot policy is unnecessary: keeping a 7B MLLM completely frozen and learning only a single <ACT> prompt token plus an auxiliary action-prediction head matches or beats fine-tuned baselines on CALVIN. It also shows the latent goal token mostly transmits instruction semantics rather than visual state, and that asynchronous inference delays up to 60 policy steps cost little. The practical payoff is a low-cost, open-source recipe for connecting a strong frozen 'System 2' reasoner to a fast diffusion policy.","feed_headline":"Frozen 7B VLM plus one tuned token beats fine-tuned robot policies","feed_subtitle":"Auxiliary action head lifts CALVIN-E five-step chain success from 6.1% to 20.2%.","key_machinery":"The load-bearing object is the learnable <ACT> token appended to the instruction, a special vocabulary token whose embedding is the only part of the 7B MLLM that is updated. A linear auxiliary head reads the token's final-layer hidden state and predicts position, rotation, and gripper state, so the token must carry actionable information. That embedding is fed through a linear projector into a pre-trained 3D Diffuser Actor, which denoises action trajectories conditioned on 3D scene tokens. Two-stage training—first aligning the projector with both the MLLM and policy frozen, then jointly fine-tuning the prompt, projector, and policy—is what makes the connection between the two systems survive.","core_discovery":"The paper's central claim is that a dual-system VLA can be made simple and cheap without losing performance: freeze the MLLM, add one learnable <ACT> token to its vocabulary, train only that embedding and a linear projector, and add an auxiliary head that predicts the robot's actions from the <ACT> embedding. On CALVIN ABC-D this reaches 93.3% one-task and 46.0% five-task success, and on CALVIN-E with enriched language instructions it lifts five-task success to 20.2% versus 6.1% for the policy alone. The authors conclude that the auxiliary task is very helpful because it forces the frozen MLLM to encode visual information, and that prompt tuning preserves language generalization better than LoRA fine-tuning.","pith_inferences":["The auxiliary head is trained on the same ground-truth actions as the policy, so its benefit may come from shared multi-task regularization or language-action statistics rather than from new visual grounding; the paper's 'forces visual input' explanation is plausible but unverified.","A visual-perturbation test—for example, occluding the workspace or moving objects after the MLLM sees the frame—would settle whether the <ACT> embedding actually tracks visual change, especially since the paper's own semantic-similarity analysis suggests it mostly does not.","The method's modularity suggests it should transfer to other frozen MLLMs and diffusion policies; if it does, dual-system VLA training reduces to prompt tuning plus a small auxiliary head, which would cut compute sharply.","CALVIN's near-static scenes may understate the need for asynchronous visual updates; the dynamic-scenario results show that MLLM latency can be masked when the low-level policy is robust on its own."],"forward_implications":["An auxiliary action-prediction head on the latent token is worth adding: it raises CALVIN ABC-D five-task chain success from 40.9% to 46.0% and CALVIN-E from 10.3% to 20.2%.","Prompt tuning preserves language generalization better than fine-tuning or frozen MLLMs under CLIP loss, as shown by the CALVIN-E results.","Asynchronous inference is nearly free: stepping the MLLM every 10 or 60 policy steps changes final performance by less than a point.","Without a pre-alignment stage that trains the projector while both models are frozen, joint training collapses to 0% success.","Removing visual input from the MLLM (treating it as an LLM only) drops five-task success from 47% to 10%, so the vision path is doing real work."],"supporting_citations":[{"why":"Introduces the <ACT>-token latent-bridging dual-system design that this paper adopts as its baseline and comparison point.","marker":"[19]"},{"why":"Supplies the pre-trained 3D Diffuser Actor used as the low-level policy and the diffusion training loss.","marker":"[11]"},{"why":"Provides the LLaVA-7B MLLM used as the frozen System 2 backbone.","marker":"[15]"},{"why":"Recent dual-system VLA using OpenVLA with multiple <ACT> tokens and language-action latents, a key comparison for design choices.","marker":"[5]"},{"why":"Basis of the dual-process framing and a frozen-MLLM latent-representation design compared throughout.","marker":"[9]"},{"why":"HiRT's hierarchical transformer with MAP pooling serves as a contrasting dual-system integration method.","marker":"[24]"},{"why":"Diffusion Policy supplies the action-denoising paradigm underlying the low-level policy training.","marker":"[6]"},{"why":"Provides the RoboFlamingo single-system baseline that motivates the dynamic-scenario CALVIN-D evaluation.","marker":"[13]"}],"fun_headline_variants":["Freeze the VLM, add one token, boost robot task success","One learnable token beats full fine-tuning for robot policies","Auxiliary head lifts CALVIN-E five-step success from 6.1% to 20.2%","OpenHelix: frozen VLM plus single token achieves 93.3% on CALVIN","Minimal training: one token and linear head for robotic control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's explanation that the auxiliary head works by forcing the frozen MLLM to use visual information is never directly tested, so the improvement could come from extra regularization or language statistics rather than from true visual grounding.","fun_headline_variants_meta":{"raw":{"variants":["Freeze the VLM, add one token, boost robot task success","One learnable token beats full fine-tuning for robot policies","Auxiliary head lifts CALVIN-E five-step success from 6.1% to 20.2%","OpenHelix: frozen VLM plus single token achieves 93.3% on CALVIN","Minimal training: one token and linear head for robotic control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00051,"raw_usage":{"total_tokens":2420,"prompt_tokens":824,"completion_tokens":1596,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":1499}},"tokens_in":440,"tokens_out":1596,"duration_ms":9381,"temperature":1.0,"reasoning_tokens":1499,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:42:50.277731+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the authors' released model, run CALVIN tasks while occluding or perturbing the visual input at test time, and record whether the <ACT> embedding and predicted actions change. If the embedding stays nearly constant and success does not drop, the auxiliary task did not create visual grounding; alternatively, train the auxiliary head on randomized action targets and see whether the gains persist.","supporting_citations":[{"cited_title":"A dual process vla: Efficient robotic manipulation leveraging vlm","cited_arxiv_id":null,"evidence_quote":"Basis of the dual-process framing and a frozen-MLLM latent-representation design compared throughout."},{"cited_title":"Diffusion policy: Visuomotor policy learning via action diffu- sion","cited_arxiv_id":null,"evidence_quote":"Diffusion Policy supplies the action-denoising paradigm underlying the low-level policy training."}],"review_version":1}