{"id":"489c946b-a082-47ee-bebc-317418cdd419","arxiv_id":"2609.19138","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An off-the-shelf VLM wired into real robots through a context-to-action harness completed more of the tested tasks when given demonstrations, goal images, interaction history, or live human cues, with no parameter updates, though trials were few and execution errors persisted.","lead":"This paper tests whether off-the-shelf vision-language models can guide a real robot through new tasks by watching videos, goal images, or interaction examples at run time, without any retraining. It introduces GPT-Policy and reports small real-robot trials where such context helped complete more tasks, while planning well proved easier than executing precise, safe contact.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The claimed improvements rely on three trials per condition without statistical testing, making observed success rate differences (e.g., 0/3 to 2/3) indistinguishable from chance variation.","rationale":"The reader's verdict is CONDITIONAL, and the weakest assumption identified exactly matches the statistical reliability of small-N human-labeled trials. My concern is a sharper, more quantitative statement of that same issue. The framework and qualitative demonstrations remain valuable, so the verdict should not move to REJECT or ACCEPT. UNCHANGED correctly reflects that the existing CONDITIONAL verdict already accounts for this concern.","tokens_in":24555,"tokens_out":2169,"duration_ms":23190,"concrete_test":"Re-run each condition (e.g., Pick Red Towel with None and Human Video) with at least 20 trials, using blinded human labeling with multiple raters and inter-rater agreement reported. Compute a two-sided Fisher exact test or binomial confidence interval for the success rate difference. If the difference is not significant at α = 0.05 (or after Bonferroni correction for 5 task families), the quantitative improvement claim is unsupported, though the qualitative observation that context can help may still stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that human video demonstrations improve task completion (Pick Red Towel: 0/3 to 2/3; Pick Up Notebook: 0/3 to 2/3)—rests on three trials per condition with no reported variance or statistical test (Section 4.1, Table 1). A Fisher exact test on the 0/3 vs. 2/3 comparison yields p ≈ 0.2, which does not rule out chance. The paper also does not quantify inter-rater reliability for the human-assigned success labels (Appendix D, P4: 'A human assigns the final success/failure label') or describe blinding. For three other context families (target image, self-history, online interaction), no 'None' baseline is reported, so the paper cannot attribute the 3/3 success rates to context rather than the VLM's base capability. These gaps undermine the quantitative strength of the core claim that context improves task completion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper introduces GPT-Policy, a framework that connects an off-the-shelf vision-language model (GPT-6 Astra) to real robot arms through a closed-loop context-to-action interface. The core claim is that providing various forms of context—human demonstration videos, robot videos with or without recorded actions, goal images, self-interaction history, and online human interaction—can improve task completion in manipulation and human-robot interaction tasks without any gradient updates or persistent parameter changes. The evaluation is conducted on ten real-robot tasks with three trials per condition. The paper reports that human video improves success from 0/3 to 2/3 on two tasks, robot video+action raises success from 0/3 to 3/3 on one task and to 2/3 on another, and goal images, self-history, and online interaction all achieve 3/3 success. The framework design is described in detail, including prompt templates (Appendix D), IK execution, and context construction.","tokens_in":24807,"tokens_out":2123,"duration_ms":23399,"significance":"If the core claim holds, the paper demonstrates that a general-purpose VLM can use contextual references to guide real-robot behavior without any specialized robot training. This would be a significant step toward in-context robot learning using off-the-shelf models, potentially reducing the need for task-specific data collection and policy fine-tuning. The framework is clearly described and the real-robot experiments are a strength. However, the quantitative evidence for the central claim is weakened by the small number of trials (three per condition) and the absence of statistical tests or baselines for several context types. The significance is therefore conditional on whether the observed improvements are robust and statistically distinguishable from chance.","major_comments":[{"comment":"The main empirical evidence for context improving success rests on three trials per condition. For the core comparison (Human Video vs. None), the reported 0/3 to 2/3 improvement on both Pick Red Towel and Pick Up Notebook is not statistically significant (Fisher exact test yields p ≈ 0.2). The paper does not report any confidence intervals, effect sizes, or inferential tests. With only three trials, the observed success rates are highly sensitive to a single outcome flip. The claim that context improves task completion requires stronger statistical support or a demonstration that the effect is replicable across more trials.","section":"Section 4.1, Table 1"},{"comment":"For the Target Image, Self-History, and Online Human-Robot Interaction conditions, only the context condition is tested; no 'None' baseline is reported for these tasks. The paper therefore cannot attribute the 3/3 success rates to context rather than to the VLM's base capability on these tasks. Without a baseline, the evidence for context-driven improvement on these tasks is unsubstantiated. The authors should either add a None baseline or explicitly argue why success would not be 3/3 without context (e.g., by showing failures in preliminary attempts).","section":"Table 1 (Target Image, Self-History, Online Interaction rows)"},{"comment":"Success/failure labels are assigned by a human (P4: 'A human assigns the final success/failure label'). The paper does not describe whether this labeling was done blind to the condition (e.g., the human does not know whether context was provided) or whether any inter-rater reliability check was performed. If the human labeler is aware of the condition, conscious or unconscious bias could influence the results, especially on tasks where success criteria require subjective judgment (e.g., 'T-shape arrangement'). The authors should clarify the labeling procedure and ideally provide evidence of reliability (e.g., agreement between multiple labelers).","section":"Appendix D, P4; Section 4.1"}],"minor_comments":[{"comment":"The metric 'Decisions' is defined as the number of generated action targets or action blocks. In Table 1, decisions and time are averaged over all trials (including failures). For the Human Video tasks, decision count decreases when context is added, but this could be because successful trials terminate earlier; it would be more informative to report metrics separately for successful and failed trials, or at least clarify that the averages mix different trial outcomes.","section":"Section 4.1"},{"comment":"Keyframe selection for demonstrations uses an undisclosed vision model (prompts P0a/P0b). The paper does not specify whether this model is from the same family as the evaluated VLM (GPT-6 Astra). If it is, the evaluation could contain circularity: the demonstration selection biases the context toward what the VLM already 'understands'. Please state the model used for keyframe selection.","section":"Appendix C.3"},{"comment":"The comparison across models (GPT-6 Astra vs. Fable 5.1 vs. Kimi K3) is based on single runs with reported 'task progress' (not success rate). The paper correctly warns that these examples do not establish a reliable ranking, but the inclusion of token usage and run time on singletons is nevertheless potentially misleading. Consider removing or clearly labelling these as illustrative qualitative observations.","section":"Section 5, Table 2"},{"comment":"The 'Time (min)' column for Movable Exploration reports 25.53, while other entries use one decimal place (e.g., 15.8). Consistent formatting would improve readability.","section":"Table 1"},{"comment":"The claim that action references 'improve alignment' is supported by measured orientation differences at keyframes. However, no statistical comparison is provided for these measurements. Consider reporting the range or standard deviation across trials.","section":"Section 4.3, Figure 7 caption and text"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Here's my take on the GPT-Policy paper. The core contribution is a clean, well-documented framework for testing how off-the-shelf VLMs use context in real robot tasks, covering five context families and shipping full prompt templates and honest limitations. That is genuinely useful for the community. The systematic comparison of a fixed VLM across human video, robot video, goal image, self-history, and online interaction is new relative to prior work that trains dedicated policies. The qualitative observation that context can guide behavior is credible and worth exploring.\n\nBut the quantitative claims do not hold at the strength stated in the abstract and introduction. Every condition is run three times. The headline improvement from 0/3 to 2/3 is indistinguishable from chance (Fisher exact p≈0.2). There is no variance reporting, no blinding for human-assigned success labels, and missing None baselines for three of five context families. The paper itself flags the small series and incomplete ablations, which counts in its favor, but the narrative still oversells the evidence.\n\nTo its credit, the limitations section is not boilerplate, the appendices are detailed, and the authors provide code and videos. The framework and evaluation template are valuable for future studies, and the qualitative findings (e.g., that a fixed VLM can sometimes use a video to improve task understanding) are a useful starting point.\n\nThis paper is for researchers building VLM-based robot controllers. It will be cited for the framework and the qualitative observation, not for the specific success rates. I would send it to peer review with major revisions required: the authors should reframe as a preliminary study, add proper baselines, report confidence intervals or statistical tests, and temper the abstract's wording. The honesty and novelty warrant referee time.","headline":"Useful framework and honest reporting, but the quantitative claims are not backed by the paper's own three-trial experiments.","tokens_in":25477,"tokens_out":1259,"duration_ms":14025},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"A fixed general-purpose VLM can learn robot actions from demonstration videos on the fly.","keywords":["in-context learning","vision-language models","robot manipulation","GPT-Policy","VLM agents","context compiler","real-robot experiments","demonstration-guided control"],"falsifier":"Running the same tasks with at least 30 trials per condition and using a blinded evaluator (who does not know the context condition) to assign success labels would determine whether the context effect survives statistical scrutiny.","tokens_in":26080,"feed_emoji":"🤖","tokens_out":3288,"duration_ms":36045,"temperature":0.7,"pith_summary":"This paper tests whether a fixed, off-the-shelf vision-language model (GPT-6 Astra) can use contextual references—such as human videos, robot videos with recorded actions, goal images, self-interaction history, and live human interaction—to guide a real robot through new tasks without any gradient updates or persistent parameter changes. The GPT-Policy framework packages these references into a structured prompt and executes the VLM's proposed tool commands via a constrained controller. In real-robot trials with only three runs per condition, providing a human video raised success on two pick-up tasks from 0/3 to 2/3, and adding action references to a robot video raised success on bottle-cap unscrewing from 0/3 to 3/3. The results suggest that general-purpose VLMs already possess some capacity for in-context adaptation in robotics, though they also highlight that the same models still struggle with precise contact, outcome verification, and physical safety.","feed_headline":"Off-the-shelf VLM learns robot tasks from video on the fly","feed_subtitle":"Without any parameter updates, GPT-6 Astra uses human or robot demonstrations to improve real-robot task success.","key_machinery":"The central object is the GPT-Policy framework, which connects a fixed general-purpose VLM (GPT-6 Astra) to a robot via a closed-loop context-to-action interface. A context compiler selects and formats task-relevant keyframes, action records, and interaction history into the VLM prompt. The VLM proposes parameterized tool commands (e.g., move_to, set_gripper), which a constrained controller verifies for kinematic feasibility and executes via Cartesian motion with inverse kinematics. Execution feedback and new observations are returned for the next decision. The key machinery is this inference-only loop: the VLM's parameters remain frozen throughout, so any adaptation comes solely from the content of the contextual prompt.","core_discovery":"The paper claims that a fixed, general-purpose VLM (GPT-6 Astra), when given task-relevant context at deployment, can produce executable robot tool requests that improve task success compared to providing no context at all. The context is compiled into the VLM's prompt as interleaved images and text, and the model's proposed actions are verified and executed by a separate constrained controller. Empirically, human video demonstrations raised success on 'Pick Red Towel' and 'Pick Up Notebook' from 0/3 to 2/3; robot video with recorded actions raised 'Unscrew Bottle Cap' from 0/3 to 3/3 and 'Remove and Reinsert Plug' from 0/3 to 2/3; goal images, self-history, and online human interaction each achieved 3/3 on their respective tasks. The central discovery is that task-relevant context can improve VLM-driven robot performance without parameter updates, but the improvements are modest and reliability depends heavily on contact-execution quality, which the VLM alone does not guarantee.","pith_inferences":["The paper's triplicate runs per condition are too few to rule out chance as an explanation for the observed success differences; a more rigorous statistical evaluation would be needed to confirm the context effect.","Because the VLM makes decisions slowly and expensively, the framework may be best suited for high-level reasoning while a fast low-level policy handles fine-grained control—a System 1/System 2 split that the authors themselves suggest.","The reliance on a specific commercial VLM (GPT-6 Astra) makes the results hard to reproduce independently and raises questions about how well the findings generalize to other models or to open-source alternatives.","Context quality and prompt engineering are likely critical: small changes in keyframe selection or phrasing could substantially alter task success, implying that robustness to imperfect context is still unresolved."],"forward_implications":["If true, a single fixed VLM can serve as a robot controller for a variety of tasks without any robot-specific training, greatly simplifying deployment.","Human demonstration videos can transfer task strategies to robots even without robot action labels, enabling non-experts to teach robots by showing.","Providing aligned robot action references along with video can further improve contact-sensitive tasks by constraining motion timing and gripper coordination.","Goal images and self-interaction history can ground task understanding and enable exploratory behavior such as obstacle avoidance and object search.","Online human interaction (gestures, turn-taking) can be incorporated into the same framework, supporting collaborative tasks like tic-tac-toe."],"supporting_citations":[{"why":"Establishes the concept of in-context learning for language models, which this work extends to robot control.","marker":"(Brown et al., 2020)"},{"why":"Introduces GPT-6 Astra, the specific general-purpose VLM used as the fixed policy in all experiments.","marker":"(OpenAI, 2026)"},{"why":"Introduces one-shot imitation learning for robots, a direct precursor to the demonstration-based context studied here.","marker":"(Duan et al., 2017)"},{"why":"Demonstrates human-to-robot in-context learning from human play videos, closely related to the human video condition evaluated in this paper.","marker":"(Shah et al., 2025)"}],"fun_headline_variants":["VLM learns robot tasks from video without updates","GPT-6 Astra adapts robots via in-context video","No updates needed: VLM picks up robot tasks from demos","Fixed VLM improves robot success from human demos","In-context video lets GPT-6Astra control robot actions"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The observed improvements in success rate (e.g., 0/3 to 2/3) are assumed to reflect a genuine causal effect of context, but each condition is tested only three times and the success label is assigned by a human, so the results could be explained by run-to-run variation or evaluator bias.","fun_headline_variants_meta":{"raw":{"variants":["VLM learns robot tasks from video without updates","GPT-6 Astra adapts robots via in-context video","No updates needed: VLM picks up robot tasks from demos","Fixed VLM improves robot success from human demos","In-context video lets GPT-6Astra control robot actions"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"cost":0.000234388,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000234388,"upstream_inference_prompt_cost":0.000114828,"upstream_inference_completions_cost":0.00011956},"total_tokens":1452,"prompt_tokens":1025,"completion_tokens":427,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":343}},"tokens_in":1025,"tokens_out":427,"cost_source":"openrouter_usage","duration_ms":5852,"image_pages":[],"temperature":1.0,"reasoning_tokens":343,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-17T03:50:46.829803+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Running the same tasks with at least 30 trials per condition and using a blinded evaluator (who does not know the context condition) to assign success labels would determine whether the context effect survives statistical scrutiny.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}