Computer Vision and Pattern Recognition · arXiv:2609.19138 v1 · submitted 16 Sep 2026 · 3 references
In-Context Robot Learning with VLM Agents
The paper says
“Enabling robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of embodied AI.”
more of the abstract
No finite collection of demonstrations can cover every task and situation a robot will encounter, making the ability to learn from context at deployment essential for generalization. Such in-context learning (ICL), however, remains largely beyond the reach of existing robotic policies. The broad agentic capabilities of commercial vision-language models (VLMs), such as GPT-6 Astra, raise a compelling question: can these models learn from demonstrations, examples, and interaction feedback, then translate that information into executable and verifiable robot behavior from a new initial state without gradient updates or persistent changes to task-specific parameters? We introduce GPT-Policy, a general-agent framework for in-context robot learning. GPT-Policy integrates a context compiler that preserves task-relevant visual transitions, a VLM that proposes robot-tool actions, and a constrained controller that verifies and executes each action and reports its outcome. We evaluate its reliability and limitations through task success and efficiency metrics, matched comparisons across models, and controlled context ablations. In real-robot trials, human video demonstrations improve task completion even without robot action labels, while aligned action references yield further gains on contact-sensitive tasks. These findings position GPT-Policy as a step toward robot adaptation through in-context learning, providing an empirical foundation for translating the general-purpose capabilities of VLMs into physical behavior and clarifying the challenges that must be overcome for reliable deployment.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 17 Sep 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Useful framework and honest reporting, but the quantitative claims are not backed by the paper's own three-trial experiments. This line is wrong
3 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 3 references say.
The paper says, at Section 4.1, Table 1
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The main empirical evidence for context improving success rests on three trials per condition. For the core comparison (Human Video vs. None), the reported 0/3 to 2/3 improvement on both Pick Red Towel and Pick Up Notebook is not statistically significant (Fisher exact test yields p ≈ 0.2).
more of objection 1
The paper does not report any confidence intervals, effect sizes, or inferential tests. With only three trials, the observed success rates are highly sensitive to a single outcome flip. The claim that context improves task completion requires stronger statistical support or a demonstration that the effect is replicable across more trials.
The paper says, at Table 1 (Target Image, Self-History, Online Interaction rows)
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
For the Target Image, Self-History, and Online Human-Robot Interaction conditions, only the context condition is tested; no 'None' baseline is reported for these tasks. The paper therefore cannot attribute the 3/3 success rates to context rather than to the VLM's base capability on these tasks.
more of objection 2
Without a baseline, the evidence for context-driven improvement on these tasks is unsubstantiated. The authors should either add a None baseline or explicitly argue why success would not be 3/3 without context (e.g., by showing failures in preliminary attempts).
The paper says, at Appendix D, P4; Section 4.1
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
Success/failure labels are assigned by a human (P4: 'A human assigns the final success/failure label'). The paper does not describe whether this labeling was done blind to the condition (e.g., the human does not know whether context was provided) or whether any inter-rater reliability check was performed.
more of objection 3
If the human labeler is aware of the condition, conscious or unconscious bias could influence the results, especially on tasks where success criteria require subjective judgment (e.g., 'T-shape arrangement'). The authors should clarify the labeling procedure and ideally provide evidence of reliability (e.g., agreement between multiple labelers).
5 minor objections
The paper says, at Section 4.1
Pith says: minor objection 1 of 5
The metric 'Decisions' is defined as the number of generated action targets or action blocks. In Table 1, decisions and time are averaged over all trials (including failures). For the Human Video tasks, decision count decreases when context is added, but this could be because successful trials terminate earlier; it would be more informative to report metrics separately for successful and failed trials, or at least clarify that the averages mix different trial outcomes.
The paper says, at Appendix C.3
Pith says: minor objection 2 of 5
Keyframe selection for demonstrations uses an undisclosed vision model (prompts P0a/P0b). The paper does not specify whether this model is from the same family as the evaluated VLM (GPT-6 Astra). If it is, the evaluation could contain circularity: the demonstration selection biases the context toward what the VLM already 'understands'. Please state the model used for keyframe selection.
show 3 more
The paper says, at Section 5, Table 2
Pith says: minor objection 3 of 5
The comparison across models (GPT-6 Astra vs. Fable 5.1 vs. Kimi K3) is based on single runs with reported 'task progress' (not success rate). The paper correctly warns that these examples do not establish a reliable ranking, but the inclusion of token usage and run time on singletons is nevertheless potentially misleading. Consider removing or clearly labelling these as illustrative qualitative observations.
The paper says, at Table 1
Pith says: minor objection 4 of 5
The 'Time (min)' column for Movable Exploration reports 25.53, while other entries use one decimal place (e.g., 15.8). Consistent formatting would improve readability.
The paper says, at Section 4.3, Figure 7 caption and text
Pith says: minor objection 5 of 5
The claim that action references 'improve alignment' is supported by measured orientation differences at keyframes. However, no statistical comparison is provided for these measurements. Consider reporting the range or standard deviation across trials.
What would settle it
Re-run each condition (e.g., Pick Red Towel with None and Human Video) with at least 20 trials, using blinded human labeling with multiple raters and inter-rater agreement reported. Compute a two-sided Fisher exact test or binomial confidence interval for the success rate difference. If the difference is not significant at α = 0.05 (or after Bonferroni correction for 5 task families), the quantitative improvement claim is unsupported, though the qualitative observation that context can help may still stand.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 17 Sep 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Here's my take on the GPT-Policy paper. The core contribution is a clean, well-documented framework for testing how off-the-shelf VLMs use context in real robot tasks, covering five context families and shipping full prompt templates and honest limitations. That is genuinely useful for the community. The systematic comparison of a fixed VLM across human video, robot video, goal image, self-history, and online interaction is new relative to prior work that trains dedicated policies. The qualitative observation that context can guide behavior is credible and worth exploring.
But the quantitative claims do not hold at the strength stated in the abstract and introduction. Every condition is run three times. The headline improvement from 0/3 to 2/3 is indistinguishable from chance (Fisher exact p≈0.2). There is no variance reporting, no blinding for human-assigned success labels, and missing None baselines for three of five context families. The paper itself flags the small series and incomplete ablations, which counts in its favor, but the narrative still oversells the evidence.
To its credit, the limitations section is not boilerplate, the appendices are detailed, and the authors provide code and videos. The framework and evaluation template are valuable for future studies, and the qualitative findings (e.g., that a fixed VLM can sometimes use a video to improve task understanding) are a useful starting point.
This paper is for researchers building VLM-based robot controllers. It will be cited for the framework and the qualitative observation, not for the specific success rates. I would send it to peer review with major revisions required: the authors should reframe as a preliminary study, add proper baselines, report confidence intervals or statistical tests, and temper the abstract's wording. The honesty and novelty warrant referee time.
The strongest objection
The central empirical claim—that human video demonstrations improve task completion (Pick Red Towel: 0/3 to 2/3; Pick Up Notebook: 0/3 to 2/3)—rests on three trials per condition with no reported variance or statistical test (Section 4.1, Table 1). A Fisher exact test on the 0/3 vs. 2/3 comparison yields p ≈ 0.2, which does not rule out chance. The paper also does not quantify inter-rater reliability for the human-assigned success labels (Appendix D, P4: 'A human assigns the final success/failure label') or describe blinding. For three other context families (target image, self-history, online interaction), no 'None' baseline is reported, so the paper cannot attribute the 3/3 success rates to context rather than the VLM's base capability. These gaps undermine the quantitative strength of the core claim that context improves task completion.
The reader's verdict is CONDITIONAL, and the weakest assumption identified exactly matches the statistical reliability of small-N human-labeled trials. My concern is a sharper, more quantitative statement of that same issue. The framework and qualitative demonstrations remain valuable, so the verdict should not move to REJECT or ACCEPT. UNCHANGED correctly reflects that the existing CONDITIONAL verdict already accounts for this concern.
The referee's report
If the core claim holds, the paper demonstrates that a general-purpose VLM can use contextual references to guide real-robot behavior without any specialized robot training. This would be a significant step toward in-context robot learning using off-the-shelf models, potentially reducing the need for task-specific data collection and policy fine-tuning. The framework is clearly described and the real-robot experiments are a strength. However, the quantitative evidence for the central claim is weakened by the small number of trials (three per condition) and the absence of statistical tests or baselines for several context types. The significance is therefore conditional on whether the observed improvements are robust and statistically distinguishable from chance.
Pith's reading of the paper (a machine's summary, not the authors')
This paper tests whether a fixed, off-the-shelf vision-language model (GPT-6 Astra) can use contextual references—such as human videos, robot videos with recorded actions, goal images, self-interaction history, and live human interaction—to guide a real robot through new tasks without any gradient updates or persistent parameter changes. The GPT-Policy framework packages these references into a structured prompt and executes the VLM's proposed tool commands via a constrained controller. In real-robot trials with only three runs per condition, providing a human video raised success on two pick-up tasks from 0/3 to 2/3, and adding action references to a robot video raised success on bottle-cap unscrewing from 0/3 to 3/3. The results suggest that general-purpose VLMs already possess some capacity for in-context adaptation in robotics, though they also highlight that the same models still struggle with precise contact, outcome verification, and physical safety.
What carries the argument. The central object is the GPT-Policy framework, which connects a fixed general-purpose VLM (GPT-6 Astra) to a robot via a closed-loop context-to-action interface. A context compiler selects and formats task-relevant keyframes, action records, and interaction history into the VLM prompt. The VLM proposes parameterized tool commands (e.g., move_to, set_gripper), which a constrained controller verifies for kinematic feasibility and executes via Cartesian motion with inverse kinematics. Execution feedback and new observations are returned for the next decision. The key machinery is this inference-only loop: the VLM's parameters remain frozen throughout, so any adaptation comes solely from the content of the contextual prompt.
Core claim. The paper claims that a fixed, general-purpose VLM (GPT-6 Astra), when given task-relevant context at deployment, can produce executable robot tool requests that improve task success compared to providing no context at all. The context is compiled into the VLM's prompt as interleaved images and text, and the model's proposed actions are verified and executed by a separate constrained controller. Empirically, human video demonstrations raised success on 'Pick Red Towel' and 'Pick Up Notebook' from 0/3 to 2/3; robot video with recorded actions raised 'Unscrew Bottle Cap' from 0/3 to 3/3 and 'Remove and Reinsert Plug' from 0/3 to 2/3; goal images, self-history, and online human interaction each achieved 3/3 on their respective tasks. The central discovery is that task-relevant context can improve VLM-driven robot performance without parameter updates, but the improvements are modest and reliability depends heavily on contact-execution quality, which the VLM alone does not guarantee.
The premise it rests on. The observed improvements in success rate (e.g., 0/3 to 2/3) are assumed to reflect a genuine causal effect of context, but each condition is tested only three times and the success label is assigned by a human, so the results could be explained by run-to-run variation or evaluator bias.
If the paper is right (Pith's inferences, not the authors')
If true, a single fixed VLM can serve as a robot controller for a variety of tasks without any robot-specific training, greatly simplifying deployment.
Human demonstration videos can transfer task strategies to robots even without robot action labels, enabling non-experts to teach robots by showing.
Providing aligned robot action references along with video can further improve contact-sensitive tasks by constraining motion timing and gripper coordination.
Goal images and self-interaction history can ground task understanding and enable exploratory behavior such as obstacle avoidance and object search.
Online human interaction (gestures, turn-taking) can be incorporated into the same framework, supporting collaborative tasks like tic-tac-toe.
The paper's triplicate runs per condition are too few to rule out chance as an explanation for the observed success differences; a more rigorous statistical evaluation would be needed to confirm the context effect.
Because the VLM makes decisions slowly and expensively, the framework may be best suited for high-level reasoning while a fast low-level policy handles fine-grained control—a System 1/System 2 split that the authors themselves suggest.
The reliance on a specific commercial VLM (GPT-6 Astra) makes the results hard to reproduce independently and raises questions about how well the findings generalize to other models or to open-source alternatives.
Context quality and prompt engineering are likely critical: small changes in keyframe selection or phrasing could substantially alter task success, implying that robustness to imperfect context is still unresolved.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Enabling robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of embodied AI. No finite collection of demonstrations can cover every task and situation a robot will encounter, making the ability to learn from context at deployment essential for generalization. Such in-context learning (ICL), however, remains largely beyond the reach of existing robotic policies. The broad agentic capabilities of commercial vision-language models (VLMs), such as GPT-6 Astra, raise a compelling question: can these models learn from demonstrations, examples, and interaction feedback, then translate that information into executable and verifiable robot behavior from a new initial state without gradient updates or persistent changes to task-specific parameters? We introduce GPT-Policy, a general-agent framework for in-context robot learning. GPT-Policy integrates a context compiler that preserves task-relevant visual transitions, a VLM that proposes robot-tool actions, and a constrained controller that verifies and executes each action and reports its outcome. We evaluate its reliability and limitations through task success and efficiency metrics, matched comparisons across models, and controlled context ablations. In real-robot trials, human video demonstrations improve task completion even without robot action labels, while aligned action references yield further gains on contact-sensitive tasks. These findings position GPT-Policy as a step toward robot adaptation through in-context learning, providing an empirical foundation for translating the general-purpose capabilities of VLMs into physical behavior and clarifying the challenges that must be overcome for reliable deployment.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 0.99
“These findings position GPT-Policy as a step toward robot adaptation through in-context learning, providing an empirical foundation for translating the general-purpose capabilities of VLMs into physical behavior and clarifying the challenges that must be overcome for reliable deployment.”
How the main claim is stated: asserted 0.50
What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.86 · human participants 0.40 · controlled comparison 0.74 · randomized assignment 0.02 · states quantitative result 0.06 · compares with prior results 0.80 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Enabling robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of embodied AI. | background 1.00 | 0.04 | 0.02 | asserted 0.99 |
| No finite collection of demonstrations can cover every task and situation a robot will encounter, making the ability to learn from context at deployment essential for generalization. | background 0.96 | 0.05 | 0.03 | asserted 1.00 |
| Such in-context learning (ICL), however, remains largely beyond the reach of existing robotic policies. | gap 0.99 | 0.04 | 0.03 | asserted 0.99 |
| The broad agentic capabilities of commercial vision-language models (VLMs), such as GPT-6 Astra, raise a compelling question: can these models learn from demonstrations, examples, and interaction feedback, then translate that information into executable and verifiable robot behavior from a new initial state without gradient updates or persistent changes to task-specific parameters? | background 0.56 | 0.05 | 0.04 | hedged possibility 0.49 |
| We introduce GPT-Policy, a general-agent framework for in-context robot learning. | contribution 0.98 | 0.89 | 0.05 | asserted 1.00 |
| GPT-Policy integrates a context compiler that preserves task-relevant visual transitions, a VLM that proposes robot-tool actions, and a constrained controller that verifies and executes each action and reports its outcome. | method 0.96 | 0.77 | 0.04 | asserted 1.00 |
| We evaluate its reliability and limitations through task success and efficiency metrics, matched comparisons across models, and controlled context ablations. | method 0.94 | 0.95 | 0.04 | asserted 0.95 |
| In real-robot trials, human video demonstrations improve task completion even without robot action labels, while aligned action references yield further gains on contact-sensitive tasks. | result 1.00 | 0.93 | 0.17 | asserted 1.00 |
| These findings position GPT-Policy as a step toward robot adaptation through in-context learning, providing an empirical foundation for translating the general-purpose capabilities of VLMs into physical behavior and clarifying the challenges that must be overcome for reliable deployment. | interpretation 1.00 | 0.78 | 0.87 | asserted 0.50 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,981 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
References (3)
- [3] URLhttps://arxiv.org/abs/2410.12782. Z.ai. GLM-5.3. Z.AI Developer Documentation, 2026. URLhttps://docs.z.ai/guides/llm/glm-5.3. Jiaming Zhou, Qihang Zhang, Gangwei Xu, Cunxin Fan, Yujie Zhao, Ruilin Wang, Yiming Luo, Shuai Yang, Xing Zhu, Yujun Shen, Junwei Liang, and Yinghao... on PithPith's reviewarXiv
- [2023] URLhttps://arxiv.org/abs/2307.15818. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Ad... on PithPith's reviewarXiv
- [2025] URLhttps://arxiv.org/abs/2509.09769. Lucy Xiaoyang Shi, Brian Ichter, Michael Robert Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi robot: Open... on PithPith's reviewarXiv
Cited by 18 Pith papers
Reviewed papers on Pith that cite this one. OpenAlex reports about 23 citations worldwide. The full citation record
- Embodied First Exam cs.RO · Nov 2026
- Review the Episodes, Refine the Lessons: Simple Memory Evolution for Embodied Agents cs.RO · Nov 2026
- Ego-CEVR: Learning Cross-Embodiment Visual Representations from Human Egocentric Videos cs.RO · Oct 2026
- Pretraining Knows What, Simulation Teaches How: Compositional Generalization in a Sim-Trained Robot Agent cs.RO · Oct 2026
- MOTIO: Agentic Skill Engineering for Reality-Grounded Humanoid Loco-Manipulation cs.RO · Oct 2026
- Embodied Task Agents Can Perform Fine Manipulation through Tactile In-Context Learning cs.RO · Oct 2026
- LADDER: A Long-Horizon Robot Manipulation Benchmark cs.RO · Oct 2026
- Can a Frontier VLM Serve as a Robot Policy for Memory-Dependent Manipulation? cs.RO · Oct 2026
- UniMetric: Universal Metric Scale Recovery for Visual Geometry Foundation Models cs.CV · Oct 2026
- ACE-CaP-0: Real-World Code-as-Policies in the Era of Experience cs.RO · Nov 2026
- Learning from Physical Experience:Context-Space Adaptation for Robotic Manipulation cs.RO · Oct 2026
- RoboDoctor: Training-Free Recursive Self-Improvement for Robot Policies cs.RO · Oct 2026
- Large Multimodal Models for Direct Robot Control: GPT-6 as a Beginning cs.RO · Oct 2026
- GPT-6 Astra Is a Native Visuo-Tactile Policy for Contact-Rich Manipulation cs.RO · Oct 2026
- Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Tools cs.RO · Sep 2026
- RoboICL: Embodied In-Context Learning with GPT-6 Astra cs.RO · Sep 2026
- From Egocentric Human Video to Robot Manipulation in the Era of Embodied Intelligence cs.RO · Nov 2026
- Towards Agentic Robots: Model, Data, Environment, and Harness cs.RO · Oct 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment