{"id":"16a5f269-b750-43c7-bf4c-739384aedd4b","arxiv_id":"2411.12633","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A graph-based diffusion model trained only on procedurally generated pseudo-demonstrations lets a robot perform new manipulation tasks from one or two test-time demonstrations.","lead":"Instant Policy is a robot control system that can pick up a new manipulation task from one or two demonstrations, with no extra training. It works by representing demonstrations, camera views, and robot actions as a graph and using a diffusion model trained on computer-generated 'pseudo-demonstrations'.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The pseudo-only claim is tested on a modified RLBench restricted to Cartesian-planned demos and object orientations within [-pi/3, pi/3], so the 'unseen tasks' result may reflect distribution matching rather than task-agnostic in-context learning.","rationale":"The method is well-engineered and the pseudo-demonstration pretraining idea is genuinely interesting; the conditional recommendation is justified. The main weakness is empirical rather than logical. Table 1's headline success is obtained on a benchmark that was modified in two ways, both disclosed in Appendix F, and those modifications align the evaluation distribution with the pseudo-demonstration distribution. If the model can achieve similar success on standard RLBench without these modifications, the concern is resolved and the claim is stronger. If it cannot, the paper should state clearly that 'unseen tasks' means tasks within the matched orientation and planning distribution, not tasks in general. The real-world result is additionally confounded by co-fine-tuning on real demonstrations, so it cannot independently establish the pseudo-only claim. I do not regard this as a soundness failure: the method may still work when context supplies enough information; the issue is that the current experiments do not isolate that condition. A single reproduction on unrestricted RLBench would settle whether the restriction is essential, so the conditional verdict should stand.","tokens_in":19435,"tokens_out":8735,"duration_ms":101727,"concrete_test":"Run the pre-fine-tuned, pseudo-demonstration-only model on the same 24 RLBench tasks under the standard RLBench protocol: full object orientation range and demonstrations that include RRT-planned trajectories, with no additional training. Compare the average success rate to the 0.71 PD-only average in Table 1. If it drops substantially, the central 'pseudo-only / unseen-task' claim must be scoped to the orientation-limited, Cartesian-planned distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that pseudo-demonstrations, which are procedurally generated and not kinematically feasible, suffice as a training pool because test-time context supplies the missing task constraints. The strongest controlled evidence is Table 1, but Appendix F states that the evaluation deliberately made two changes to standard RLBench: only Cartesian-space-planned demonstrations were used, and object orientations were restricted to [-pi/3, pi/3] 'to match the distribution of object poses to the one present in our generated pseudo-demonstrations.' This is a direct admission that the pseudo-demonstration distribution does not cover the standard benchmark; the benchmark was narrowed to fit the training data. A second gap appears in the real-world experiments: before evaluation, the model was co-fine-tuned on five real demonstrations from five non-evaluation tasks, so Table 3 is not a pseudo-only result. Thus the concern is not merely that the training trajectories are infeasible in the abstract, but that the evaluation was tailored to the pseudo-demo distribution whenever the central claim was tested. If pseudo-demos really provide a task-agnostic prior, this tailoring should be unnecessary; the paper does not show what happens when it is removed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Instant Policy, an in-context imitation learning method that, at test time, takes one or two demonstrations of a novel task and immediately produces closed-loop SE(3) end-effector actions without further training. The method represents demonstrations, current point-cloud observations, and future actions in a heterogeneous graph and trains a diffusion model over that graph using procedurally generated 'pseudo-demonstrations' that are not required to be dynamically or kinematically feasible. The authors evaluate the approach on 24 RLBench tasks with 100 rollouts each, on 16 real-world everyday tasks with 10 rollouts each, and in additional studies of novel-object generalization, cross-embodiment transfer, and language-conditioned deployment. They report higher success rates than three adapted in-context learning baselines, positive scaling trends with data and model size, and an emergent ability to generalize to novel object geometries.","tokens_in":19733,"tokens_out":4115,"duration_ms":45147,"significance":"If the central claim holds, the paper makes a notable contribution: it offers a scalable training-data recipe for task-agnostic imitation learning by replacing laborious real demonstrations with procedurally generated pseudo-demos, and it introduces a novel graph-diffusion formulation that is well matched to structured SE(3) manipulation. The paper is experimentally broad, including 24 simulated and 16 real-world tasks, a scaling study with several model sizes, and ablations of action parameterization and inference-time hyperparameters. The authors are also transparent in disclosing code and videos, and Appendix F explicitly reports the changes made to the RLBench setup. These strengths make the work potentially valuable to the robot learning community. However, as detailed below, the evaluation protocol contains two load-bearing gaps: the simulated evaluation is performed on a modified RLBench whose object-pose distribution is deliberately matched to the pseudo-demonstration distribution, and the real-world evaluation uses a model that was co-fine-tuned on real demonstrations from non-evaluation tasks.","major_comments":[{"comment":"The simulated evaluation is not on the standard RLBench benchmark. Appendix F states two modifications: only Cartesian-space-planned demonstrations were kept (RRT-planned demonstrations were discarded), and object orientations were restricted to [-pi/3, pi/3] 'to match the distribution of object poses to the one present in our generated pseudo-demonstrations'. The second modification is particularly relevant because the paper's central claim is that pseudo-demonstrations provide a task-agnostic prior that generalizes to unseen tasks; restricting the evaluation distribution to match the training distribution weakens that claim. I request either results on the unmodified RLBench protocol or an ablation that quantifies how success rates degrade as the orientation range is widened and when RRT-planned demonstrations are included. Without such evidence, Table 1 should be described as an evaluation on a pose-restricted subset of RLBench rather than on 'unseen tasks' generally.","section":"Appendix F, Section 4.1, Table 1"},{"comment":"The real-world experiments do not evaluate a pseudo-only model. The text says the model used in previous experiments was 'further co-fine-tuned ... using 5 demos from 5 tasks not included in the evaluation.' Thus the high success rates in Table 3 are obtained with a model that has seen additional real demonstrations, even though those demonstrations are from different tasks. This is not by itself improper, but it conflicts with the abstract's framing of 'without further training' and with the claim that the method works from pseudo-demonstrations alone. Please report a pseudo-only real-world baseline, or an ablation showing the effect of the co-fine-tuning step, and adjust the wording so that the real-world results are not presented as pseudo-only evidence.","section":"Section 4.3, Table 3"},{"comment":"Success rates are reported as point estimates without confidence intervals or significance tests. This is especially relevant in Table 3, where only 10 rollouts per task are used and several comparative differences are small (e.g., Instant Policy 8/10 vs. BC-Z* 8/10 for 'Open Box', or 5/10 vs. 5/10 for 'Push Cans Together'). The text in Section 4.3 states that Instant Policy outperforms baselines 'by a large margin' on the average, but the average difference is not accompanied by any statistical measure. I ask for confidence intervals or a significance test, at least on the aggregate success rates, and for the per-task numbers where the paper makes comparative claims.","section":"Tables 1 and 3"},{"comment":"The paper's feasibility argument rests on the assertion that because task specifications are provided as context at inference, pseudo-demonstrations need not be dynamically or kinematically feasible, and that 'with enough randomisation, the convex hull of the generated trajectories would encapsulate all the possible test-time tasks.' This is a strong assumption: it presumes that the context demonstrations carry enough information to compensate for the absence of physical constraints in the training data. The evaluation provides some support, but Appendix F's orientation restriction suggests that the pseudo-demonstration distribution did not in fact cover the natural test distribution, and the paper does not directly test how much of the performance comes from the context rather than from matching the training distribution. A concrete test would be to train on pseudo-demonstrations with a deliberately mismatched object-pose distribution and show that the context still drives successful task execution, or to compare against a variant where the pseudo-demo distribution is matched to the full RLBench pose range.","section":"Section 3.4, 'An Infinite Pool of Data'"}],"minor_comments":[{"comment":"The notation in Equation (1) is hard to parse: the argument of G mixes a normal distribution over actions with the graph construction, and the parentheses are unbalanced. Please rewrite the forward process more explicitly. Also, the symbol α in Equation (2) is used without definition; it presumably refers to the standard DDPM noise schedule, but it should be defined.","section":"Section 3.3, Equations (1)-(2)"},{"comment":"The ablation table reports only the average percentage change in success rate relative to the base model, without the base success rate or absolute values. This makes it difficult to judge, for example, whether a -71% change for L=1 corresponds to a drop from 0.71 to 0.20 or from 0.10 to 0.03. Please report absolute success rates alongside the relative changes.","section":"Section 4.2, Table 2"},{"comment":"There is a typo in the task name 'Put rubish in bin'; it should be 'Put rubbish in bin'. Also, the table legend uses 'PD only / PD++', but the meaning of the two numbers in each cell is not fully explained in the main text; please make the slash notation explicit in the caption.","section":"Table 1"},{"comment":"The scaling section reports validation loss curves and states qualitatively that success rates on unseen RLBench tasks follow a similar trend, but no quantitative success rates are given. Since the paper's main claims are about task success rather than likelihood, please include a corresponding success-rate plot or table for the scaling study.","section":"Section 4.2, 'Scaling Trends'"},{"comment":"The phrase 'arbitrary motions that would not be captured by our observations of segmented point clouds and end-effector poses' is unclear: the end-effector pose trajectory is exactly what is recorded, so it is not obvious why RRT-planned motions are 'not captured'. Please clarify what information is lost and why Cartesian planning is necessary for the method.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its experimental choices in the appendix, which is to its credit, but those choices directly affect the interpretation of the main results. In my view, the authors should be asked to either run the RLBench evaluation on the unmodified benchmark or clearly reposition the claims as applying to a pose-restricted subset, and to provide a pseudo-only real-world result or an explicit ablation of the co-fine-tuning step. These are substantial but well-scoped additions rather than unfixable flaws. I would not recommend rejection, because the core method is novel and the reported results are promising, but the current framing overstates the strength of the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. It is a well-built paper. The core idea — treat ICIL as graph generation, train a diffusion model on procedurally generated pseudo-demonstrations, and let test-time demos supply the task constraints — is genuinely new in this combination, and the RLBench numbers (Table 1) show the graph+diffusion design beats the adapted BC-Z, Vid2Robot, and GPT2 baselines by a lot on both PD-only and PD++ settings. The ablations in Table 2 are informative: the translation/rotation decoupling and the flow prediction both earn their keep. The scaling curves suggest compute-limited gains. The novel-object generalization in Table 4 is the most interesting result, even if it is on four tasks.\n\nNow the soft spots, in rough order of importance. First, Appendix F admits the RLBench evaluation limited demonstrations to Cartesian-space planning and object orientations to [-pi/3, pi/3] specifically \"to match the distribution of object poses to the one present in our generated pseudo-demonstrations.\" That is a direct admission that the synthetic training distribution does not cover the standard benchmark; the benchmark was narrowed instead of the data coverage widened. It does not sink the paper — task-agnostic ICIL can still be studied on a restricted pose distribution — but it materially weakens the \"unseen tasks\" claim. Second, the real-world evaluation is not pseudo-only: the model was co-fine-tuned on five real demonstrations from five non-evaluation tasks before testing. The paper says this clearly, but the abstract and intro phrasing can leave the wrong impression that no real training data was used at all. Third, success rates come without confidence intervals or significance tests, and some per-task differences against baselines are small. Fourth, code is promised but not actually released; the method has enough moving parts that the absence of code matters.\n\nThe pseudo-demonstration idea itself is not as brittle as it might look. The trajectories are kinematically infeasible by construction, but that is fine in principle: the context demos should supply feasibility and task semantics. The paper is upfront about this in Section 3.4. The real issue is not infeasibility, it is distribution mismatch: the pseudo-demo generator was tuned to match the evaluation distribution, so the claim of task-agnostic generalization is weaker than advertised.\n\nBottom line: worth a serious referee. It is a strong empirical paper with an honest limitations section; it needs revisions on statistical rigor, benchmark restrictions, and code release before I would treat the central claim as established. Send it to review.","headline":"Solid engineering and a genuinely useful way to think about ICIL, but the headline claim is bigger than the evidence once you read Appendix F and the real-world fine-tuning step.","tokens_in":20191,"tokens_out":1956,"would_cite":true,"duration_ms":20431,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that one or two demonstrations of a novel task suffice for immediate closed-loop SE(3) control, with a model trained only on procedurally generated pseudo-demonstrations and a graph-diffusion action generator.","keywords":["in-context imitation learning","graph diffusion","pseudo-demonstrations","robot manipulation","SE(3) actions","point-cloud representation","zero-shot task transfer","behavioural cloning"],"falsifier":"Give the trained model hand-guided demonstrations of a task whose only feasible solution requires dynamics or contact the pseudo-demonstrations cannot exhibit, such as an inertial flip or a constrained path around an obstacle, and measure success over many random object poses. Near-zero success on that task while simpler geometric tasks succeed would show that the context demonstrations do not compensate for the physically invalid training data.","tokens_in":19240,"feed_emoji":"🤖","tokens_out":12325,"duration_ms":112876,"temperature":0.7,"pith_summary":"The paper tries to establish that a robot can learn a new manipulation task instantly from one or two demonstrations, with no weight updates, if the imitation problem is posed as conditional graph generation. The proposed model, Instant Policy, binds demonstrations, the current point-cloud observation, and future actions into a single heterogeneous graph and denoises only the action nodes with a diffusion process. Training uses pseudo-demonstrations, procedurally generated trajectories that are not required to be dynamically or kinematically feasible, because the test-time demonstrations are meant to supply the task-specific constraints. Experiments on 24 simulated and 16 real-world everyday tasks report high success rates, with performance improving as more pseudo-demonstration data and model capacity are added. If the claim is right, it replaces hundreds or thousands of task-specific demonstrations with one or two test-time examples and makes robot skill acquisition limited mainly by simulation diversity and compute.","feed_headline":"One or two demos teach a robot a new task instantly","feed_subtitle":"A graph-diffusion policy turns one or two demonstrations into closed-loop robot control, with no training at test time.","key_machinery":"The load-bearing object is a heterogeneous graph that unifies context, observation, and prediction in one space: scene nodes encode local geometry from a frozen pretrained point-cloud encoder, gripper nodes appear for the current state, for each downsampled demonstration waypoint, and for each of the T future actions, and edge types carry relative position embeddings, including time-links between consecutive gripper nodes and links from all demonstration gripper nodes to the current one. Information flows through a heterogeneous graph transformer with a deliberately structured sequence of three sub-networks: local geometry, context aggregation, then action refinement. The second load-bearing mechanism is pseudo-demonstration generation, which samples object-centric waypoints, interpolates between them, and attaches or detaches objects from the gripper to produce semantically consistent but physically infeasible trajectories. The model is trained by denoising action nodes on these trajectories and deployed by a standard deterministic denoising update of fresh action nodes, using least-squares alignment to recover each SE(3) action and a decoupled translation and rotation flow parameterization to prevent one component from dominating the other.","core_discovery":"On the paper's own terms, the discovery is that in-context imitation learning can be reduced to conditional graph generation. At test time the model builds one heterogeneous graph whose nodes are local geometry features of the segmented point cloud, keypoints on the current gripper, keypoints along one or two demonstration trajectories, and keypoints of imagined future gripper poses; edges encode relative positions with high-frequency sine and cosine embeddings. A diffusion process trained with standard forward noising and a denoising objective then iteratively refines only the action nodes, and the refined keypoint sets are converted into SE(3) end-effector displacements by least-squares alignment. This single policy, with no weight updates, reaches average success of 0.71 on 24 unseen simulated manipulation tasks when trained only on pseudo-demonstrations, and 0.88 on 16 real-world everyday tasks after co-fine-tuning on a small set of unrelated demos, while also showing cross-embodiment and language-conditioned transfer in qualitative experiments.","pith_inferences":["Inference: the paper's own one-waypoint result suggests that the context demonstration must be dense enough to disambiguate the task; if so, the practical frontier is deciding which waypoints matter, not increasing model size.","Inference: the simulation setup restricted object orientations to a fixed range, a sign that the pseudo-demonstration distribution still constrained what the model could do; generating pseudo-demos with unconstrained orientations and object arrangements would test whether the method can shed that restriction.","Inference: the graph bottleneck used for language transfer could serve as a general context interface, so in-context imitation need not be limited to demonstrations: goal images, natural language, or human video could condition the same frozen action denoiser.","Inference: if scaling continues, the main obstacle to a universal instant-policy foundation is not policy capacity but the semantic consistency and diversity of auto-generated pseudo-tasks; investing in task generators with realistic contact and collision models is a direct next step."],"forward_implications":["A robot can be taught a new everyday task interactively: the user provides one or two demonstrations, and the robot then performs the task immediately in closed loop, with no gradient updates and no per-task data collection.","Because the same weights serve every task whose semantics are conveyed by context, the approach decouples task acquisition from model training, so adding a new task is a demonstration-recording step rather than an optimization run.","Success on the simulated benchmark rises when additional per-task demonstrations are mixed into training, so real data can be layered on top of the pseudo-demonstration pool to improve performance on demanding tasks.","The model's performance scales with training time and parameter count on a hold-out pseudo-demonstration validation set, indicating that more compute and larger models should yield better instant policies.","The same trained action denoiser can be conditioned through a bottleneck representation by a different embodiment or by a language description, enabling cross-embodiment transfer and zero-shot language-defined tasks without retraining the action network."],"supporting_citations":[{"why":"Frames in-context learning as test-time adaptation to novel tasks from examples, the paradigm Instant Policy transfers to robot manipulation.","marker":"Brown, 2020"},{"why":"Supplies the forward noising and reverse denoising formulation and the loss used to train the graph diffusion model.","marker":"Ho et al., 2020"},{"why":"Provides the deterministic denoising update used to denoise action nodes at deployment.","marker":"Song et al., 2020"},{"why":"Supplies set-abstraction layers that convert point clouds into local geometry node features.","marker":"Qi et al., 2017"},{"why":"Supplies the occupancy-network pretraining objective that gives the geometry features their shape meaning.","marker":"Mescheder et al., 2019"},{"why":"Supplies the diverse object geometries sampled to generate pseudo-demonstrations.","marker":"Chang et al., 2015"},{"why":"Supplies the 24-task simulation benchmark and evaluation protocol used for the main quantitative results.","marker":"James et al., 2020"},{"why":"Supplies the heterogeneous graph attention update used as the denoising network.","marker":"Shi et al., 2020"},{"why":"Supplies the least-squares alignment used to extract SE(3) actions from denoised keypoint sets.","marker":"Arun et al., 1987"}],"fun_headline_variants":["Graph diffusion gives robots instant skills from one or two demos","Instant Policy: learn new robot tasks from two demos, no retraining","In-context imitation via graph diffusion: robot learns on the fly","One demo, no weight updates: graph diffusion for robot imitation","Robot learns new tasks instantly from single demo via graph diffusion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method stands or falls on whether procedurally generated, physically invalid trajectories are sufficient training data because test-time demonstrations are supposed to supply the missing task constraints; if one or two demonstrations cannot carry enough information about dynamics, kinematics, and grasps, the model has no correct policy to fall back on.","fun_headline_variants_meta":{"raw":{"variants":["Graph diffusion gives robots instant skills from one or two demos","Instant Policy: learn new robot tasks from two demos, no retraining","In-context imitation via graph diffusion: robot learns on the fly","One demo, no weight updates: graph diffusion for robot imitation","Robot learns new tasks instantly from single demo via graph diffusion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000739,"raw_usage":{"total_tokens":3281,"prompt_tokens":909,"completion_tokens":2372,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":2293}},"tokens_in":525,"tokens_out":2372,"duration_ms":15705,"temperature":1.0,"reasoning_tokens":2293,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:19:55.941493+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the trained model hand-guided demonstrations of a task whose only feasible solution requires dynamics or contact the pseudo-demonstrations cannot exhibit, such as an inertial flip or a constrained path around an obstacle, and measure success over many random object poses. Near-zero success on that task while simpler geometric tasks succeed would show that the context demonstrations do not compensate for the physically invalid training data.","supporting_citations":[],"review_version":1}