Pith. sign in

REVIEW 4 major objections 5 minor 6 cited by

Instant Policy: In-Context Imitation Learning via Graph Diffusion

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that one or two demonstrations of a novel task suffice for immediate closed-loop SE(3) control, with a model trained only on procedurally generated pseudo-demonstrations and a graph-diffusion action generator.

desk verdict Solid engineering and a genuinely useful way to think about ICIL, but the headline claim is bigger than the evidence once you read Appendix F and the real-world fine-tuning step. read the letter →

arxiv 2411.12633 v2 pith:QKBD6KHN submitted 2024-11-19 cs.RO cs.AIcs.CVcs.LG

Vitalis Vosylius , Edward Johns This is my paper · ORCID
classification cs.ROcs.AIcs.CVcs.LG
keywords in-contextimitationlearninggraphdiffusionpseudo-demonstrationsrobotmanipulationSE(3)actionspoint-cloudrepresentationzero-shottasktransferbehaviouralcloning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a robot can learn a new manipulation task instantly from one or two demonstrations, with no weight updates, if the imitation problem is posed as conditional graph generation. The proposed model, Instant Policy, binds demonstrations, the current point-cloud observation, and future actions into a single heterogeneous graph and denoises only the action nodes with a diffusion process. Training uses pseudo-demonstrations, procedurally generated trajectories that are not required to be dynamically or kinematically feasible, because the test-time demonstrations are meant to supply the task-specific constraints. Experiments on 24 simulated and 16 real-world everyday tasks report high success rates, with performance improving as more pseudo-demonstration data and model capacity are added. If the claim is right, it replaces hundreds or thousands of task-specific demonstrations with one or two test-time examples and makes robot skill acquisition limited mainly by simulation diversity and compute.

What carries the argument

The load-bearing object is a heterogeneous graph that unifies context, observation, and prediction in one space: scene nodes encode local geometry from a frozen pretrained point-cloud encoder, gripper nodes appear for the current state, for each downsampled demonstration waypoint, and for each of the T future actions, and edge types carry relative position embeddings, including time-links between consecutive gripper nodes and links from all demonstration gripper nodes to the current one. Information flows through a heterogeneous graph transformer with a deliberately structured sequence of three sub-networks: local geometry, context aggregation, then action refinement. The second load-bearing mechanism is pseudo-demonstration generation, which samples object-centric waypoints, interpolates between them, and attaches or detaches objects from the gripper to produce semantically consistent but physically infeasible trajectories. The model is trained by denoising action nodes on these trajectories and deployed by a standard deterministic denoising update of fresh action nodes, using least-squares alignment to recover each SE(3) action and a decoupled translation and rotation flow parameterization to prevent one component from dominating the other.

What would settle it

Give the trained model hand-guided demonstrations of a task whose only feasible solution requires dynamics or contact the pseudo-demonstrations cannot exhibit, such as an inertial flip or a constrained path around an obstacle, and measure success over many random object poses. Near-zero success on that task while simpler geometric tasks succeed would show that the context demonstrations do not compensate for the physically invalid training data.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that in-context imitation learning can be reduced to conditional graph generation. At test time the model builds one heterogeneous graph whose nodes are local geometry features of the segmented point cloud, keypoints on the current gripper, keypoints along one or two demonstration trajectories, and keypoints of imagined future gripper poses; edges encode relative positions with high-frequency sine and cosine embeddings. A diffusion process trained with standard forward noising and a denoising objective then iteratively refines only the action nodes, and the refined keypoint sets are converted into SE(3) end-effector displacements by least-squares alignment. This single policy, with no weight updates, reaches average success of 0.71 on 24 unseen simulated manipulation tasks when trained only on pseudo-demonstrations, and 0.88 on 16 real-world everyday tasks after co-fine-tuning on a small set of unrelated demos, while also showing cross-embodiment and language-conditioned transfer in qualitative experiments.

Load-bearing premise

The method stands or falls on whether procedurally generated, physically invalid trajectories are sufficient training data because test-time demonstrations are supposed to supply the missing task constraints; if one or two demonstrations cannot carry enough information about dynamics, kinematics, and grasps, the model has no correct policy to fall back on.

Editorial extensions

If this is right

  • A robot can be taught a new everyday task interactively: the user provides one or two demonstrations, and the robot then performs the task immediately in closed loop, with no gradient updates and no per-task data collection.
  • Because the same weights serve every task whose semantics are conveyed by context, the approach decouples task acquisition from model training, so adding a new task is a demonstration-recording step rather than an optimization run.
  • Success on the simulated benchmark rises when additional per-task demonstrations are mixed into training, so real data can be layered on top of the pseudo-demonstration pool to improve performance on demanding tasks.
  • The model's performance scales with training time and parameter count on a hold-out pseudo-demonstration validation set, indicating that more compute and larger models should yield better instant policies.
  • The same trained action denoiser can be conditioned through a bottleneck representation by a different embodiment or by a language description, enabling cross-embodiment transfer and zero-shot language-defined tasks without retraining the action network.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the paper's own one-waypoint result suggests that the context demonstration must be dense enough to disambiguate the task; if so, the practical frontier is deciding which waypoints matter, not increasing model size.
  • Inference: the simulation setup restricted object orientations to a fixed range, a sign that the pseudo-demonstration distribution still constrained what the model could do; generating pseudo-demos with unconstrained orientations and object arrangements would test whether the method can shed that restriction.
  • Inference: the graph bottleneck used for language transfer could serve as a general context interface, so in-context imitation need not be limited to demonstrations: goal images, natural language, or human video could condition the same frozen action denoiser.
  • Inference: if scaling continues, the main obstacle to a universal instant-policy foundation is not policy capacity but the semantic consistency and diversity of auto-generated pseudo-tasks; investing in task generators with realistic contact and collision models is a direct next step.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Instant Policy, an in-context imitation learning method that, at test time, takes one or two demonstrations of a novel task and immediately produces closed-loop SE(3) end-effector actions without further training. The method represents demonstrations, current point-cloud observations, and future actions in a heterogeneous graph and trains a diffusion model over that graph using procedurally generated 'pseudo-demonstrations' that are not required to be dynamically or kinematically feasible. The authors evaluate the approach on 24 RLBench tasks with 100 rollouts each, on 16 real-world everyday tasks with 10 rollouts each, and in additional studies of novel-object generalization, cross-embodiment transfer, and language-conditioned deployment. They report higher success rates than three adapted in-context learning baselines, positive scaling trends with data and model size, and an emergent ability to generalize to novel object geometries.

Significance. If the central claim holds, the paper makes a notable contribution: it offers a scalable training-data recipe for task-agnostic imitation learning by replacing laborious real demonstrations with procedurally generated pseudo-demos, and it introduces a novel graph-diffusion formulation that is well matched to structured SE(3) manipulation. The paper is experimentally broad, including 24 simulated and 16 real-world tasks, a scaling study with several model sizes, and ablations of action parameterization and inference-time hyperparameters. The authors are also transparent in disclosing code and videos, and Appendix F explicitly reports the changes made to the RLBench setup. These strengths make the work potentially valuable to the robot learning community. However, as detailed below, the evaluation protocol contains two load-bearing gaps: the simulated evaluation is performed on a modified RLBench whose object-pose distribution is deliberately matched to the pseudo-demonstration distribution, and the real-world evaluation uses a model that was co-fine-tuned on real demonstrations from non-evaluation tasks.

major comments (4)
  1. [Appendix F, Section 4.1, Table 1] The simulated evaluation is not on the standard RLBench benchmark. Appendix F states two modifications: only Cartesian-space-planned demonstrations were kept (RRT-planned demonstrations were discarded), and object orientations were restricted to [-pi/3, pi/3] 'to match the distribution of object poses to the one present in our generated pseudo-demonstrations'. The second modification is particularly relevant because the paper's central claim is that pseudo-demonstrations provide a task-agnostic prior that generalizes to unseen tasks; restricting the evaluation distribution to match the training distribution weakens that claim. I request either results on the unmodified RLBench protocol or an ablation that quantifies how success rates degrade as the orientation range is widened and when RRT-planned demonstrations are included. Without such evidence, Table 1 should be described as an evaluation on a pose-restricted subset of RLBench rather than on 'unseen tasks' generally.
  2. [Section 4.3, Table 3] The real-world experiments do not evaluate a pseudo-only model. The text says the model used in previous experiments was 'further co-fine-tuned ... using 5 demos from 5 tasks not included in the evaluation.' Thus the high success rates in Table 3 are obtained with a model that has seen additional real demonstrations, even though those demonstrations are from different tasks. This is not by itself improper, but it conflicts with the abstract's framing of 'without further training' and with the claim that the method works from pseudo-demonstrations alone. Please report a pseudo-only real-world baseline, or an ablation showing the effect of the co-fine-tuning step, and adjust the wording so that the real-world results are not presented as pseudo-only evidence.
  3. [Tables 1 and 3] Success rates are reported as point estimates without confidence intervals or significance tests. This is especially relevant in Table 3, where only 10 rollouts per task are used and several comparative differences are small (e.g., Instant Policy 8/10 vs. BC-Z* 8/10 for 'Open Box', or 5/10 vs. 5/10 for 'Push Cans Together'). The text in Section 4.3 states that Instant Policy outperforms baselines 'by a large margin' on the average, but the average difference is not accompanied by any statistical measure. I ask for confidence intervals or a significance test, at least on the aggregate success rates, and for the per-task numbers where the paper makes comparative claims.
  4. [Section 3.4, 'An Infinite Pool of Data'] The paper's feasibility argument rests on the assertion that because task specifications are provided as context at inference, pseudo-demonstrations need not be dynamically or kinematically feasible, and that 'with enough randomisation, the convex hull of the generated trajectories would encapsulate all the possible test-time tasks.' This is a strong assumption: it presumes that the context demonstrations carry enough information to compensate for the absence of physical constraints in the training data. The evaluation provides some support, but Appendix F's orientation restriction suggests that the pseudo-demonstration distribution did not in fact cover the natural test distribution, and the paper does not directly test how much of the performance comes from the context rather than from matching the training distribution. A concrete test would be to train on pseudo-demonstrations with a deliberately mismatched object-pose distribution and show that the context still drives successful task execution, or to compare against a variant where the pseudo-demo distribution is matched to the full RLBench pose range.
minor comments (5)
  1. [Section 3.3, Equations (1)-(2)] The notation in Equation (1) is hard to parse: the argument of G mixes a normal distribution over actions with the graph construction, and the parentheses are unbalanced. Please rewrite the forward process more explicitly. Also, the symbol α in Equation (2) is used without definition; it presumably refers to the standard DDPM noise schedule, but it should be defined.
  2. [Section 4.2, Table 2] The ablation table reports only the average percentage change in success rate relative to the base model, without the base success rate or absolute values. This makes it difficult to judge, for example, whether a -71% change for L=1 corresponds to a drop from 0.71 to 0.20 or from 0.10 to 0.03. Please report absolute success rates alongside the relative changes.
  3. [Table 1] There is a typo in the task name 'Put rubish in bin'; it should be 'Put rubbish in bin'. Also, the table legend uses 'PD only / PD++', but the meaning of the two numbers in each cell is not fully explained in the main text; please make the slash notation explicit in the caption.
  4. [Section 4.2, 'Scaling Trends'] The scaling section reports validation loss curves and states qualitatively that success rates on unseen RLBench tasks follow a similar trend, but no quantitative success rates are given. Since the paper's main claims are about task success rather than likelihood, please include a corresponding success-rate plot or table for the scaling study.
  5. [Appendix F] The phrase 'arbitrary motions that would not be captured by our observations of segmented point clouds and end-effector poses' is unclear: the end-effector pose trajectory is exactly what is recorded, so it is not obvious why RRT-planned motions are 'not captured'. Please clarify what information is lost and why Cartesian planning is necessary for the method.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found: the central claim is tested against external RLBench and real-world tasks, and no prediction reduces to a fitted parameter or self-citation.

full rationale

Instant Policy is an empirical systems paper rather than a derivation chain, and no load-bearing step equates an output to its input by construction. The model is trained only on procedurally generated pseudo-demonstrations, while the headline evaluations use held-out RLBench tasks and real-world tasks whose demonstrations and success criteria are not used to fit the weights. Appendix F's restriction of RLBench demonstrations to Cartesian-space planning and object orientations to [-pi/3, pi/3] 'to match the distribution of object poses to the one present in our generated pseudo-demonstrations' is a genuine external-validity limitation, because it narrows the evaluation distribution to the training distribution, but it is not circularity: the model still must interpret the provided context at inference and is not fitted to the RLBench success labels. Likewise, the real-world experiments' co-fine-tuning on five non-evaluation tasks weakens the pure pseudo-demonstration framing of Table 3 but does not make the evaluation tasks inputs to the prediction. Self-citations (Vosylius and Johns 2023a, 2023b; Vosylius et al. 2024) are used only for related work and peripheral technical details, alongside external citations such as Urain et al. 2023, and none is load-bearing for the central claim. No uniqueness theorem is imported from the authors, no known result is merely renamed, and no fitted parameter is relabeled as a prediction. The paper therefore exhibits no significant circularity.

Assumptions & free parameters 7 free parameters · 7 assumptions · 0 invented entities

The main load-bearing premises are the representativeness of pseudo-demonstrations, the availability of clean segmented point clouds, the Markovian assumption, and the transferability of the pretrained geometry encoder. The free parameters are standard but numerous; the most consequential are the action normalization scale and the context and horizon choices, which were validated on the benchmark tasks. No new physical entities are introduced.

free parameters (7)
  • Action normalization scale = 1 cm translation, 3 degrees rotation
    Hand-chosen in Appendix E; all action and flow predictions are normalized to [-1,1] relative to this scale, so it defines the action space and directly affects what the model can express.
  • Context demonstration length L = 10 waypoints
    Demos are downsampled to a fixed length L=10 in experiments (Section 4 and Appendix E); ablations in Table 2 show performance is sensitive to L.
  • Number of context demonstrations N = 2 (base model)
    Base experiments use N=2 demos; ablations in Table 2 show N affects performance, with one demo being much worse for some tasks.
  • Action prediction horizon T = 8 future actions
    The base model predicts T=8 actions; Table 2 shows horizon affects success and compute, with T=1 being much worse.
  • Diffusion steps K = 8 (base, inferred from Table 2)
    Number of denoising steps at inference; Table 2 shows K strongly affects performance, with K=1 reducing success by 16 percent.
  • Pseudo-task bias sampling mix = 50% biased tasks, 50% random waypoints
    Appendix D biases half of generated pseudo-demos toward grasping, pick-and-place, opening, and closing; this hand-chosen mixture shapes the training distribution.
  • Data augmentation probabilities = 30% disturbance trajectories, 10% gripper state changes, 10% input gripper flips
    Appendix D and E; these hand-set probabilities are important for recovery behavior and for avoiding gripper-state overfitting.
assumptions (7)
  • standard math Denoising diffusion probabilistic models and DDIM sampling are valid for the graph-based action representation.
    Section 3.3 relies on Ho et al. (2020) and Song et al. (2020) without derivation; the flow prediction formulation in Equation 2 is adapted from Diffusion Policy.
  • domain assumption The pre-trained occupancy network features generalize to unseen object geometries and real-world point clouds.
    Appendix A; the geometry encoder is frozen during policy training, so all downstream performance depends on these features transferring.
  • domain assumption Segmented point clouds of task-relevant objects are available at test time.
    Section 5 Limitations; simulation uses ground truth segmentation, the real world uses SAM plus XMem++, and Appendix G reports that segmentation failures are the most common real-world failure mode.
  • domain assumption Pseudo-demonstrations need not be dynamically or kinematically feasible because task constraints are conveyed by the test-time context.
    Section 3.4 and Appendix D; the model is trained on physically invalid trajectories, yet must infer feasible manipulation from the demonstrations.
  • domain assumption The Markovian assumption: the current observation and context are sufficient to predict robot actions.
    Section 5 Limitations; the model has no memory of past observations, and the authors identify Markovianity as a limitation and a cause of stalling or oscillation in Appendix G.
  • domain assumption Actions are small enough that adding noise in the tangent space of SE(3) is a valid approximation of adding noise on the manifold.
    Appendix B states this can be done because actions are sufficiently small around 1 cm and 3 degrees; this justifies the Logmap-Expmap noise schedule.
  • ad hoc to paper Restricting RLBench object orientations to [-pi/3, pi/3] and discarding RRT-planned demonstrations does not invalidate the comparison.
    Appendix F introduces these restrictions to match the pseudo-demonstration distribution, narrowing the generality of the reported success rates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Instant Policy: In-Context Imitation Learning via Graph Diffusion." pith.science (2026). https://pith.science/paper/QKBD6KHN

@misc{pith2026241112633,
  author       = {Pith},
  title        = {Pith review of: Instant Policy: In-Context Imitation Learning via Graph Diffusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QKBD6KHN}},
  note         = {Machine review of arXiv:2411.12633}
}
read the original abstract

Following the impressive capabilities of in-context learning with large transformers, In-Context Imitation Learning (ICIL) is a promising opportunity for robotics. We introduce Instant Policy, which learns new tasks instantly (without further training) from just one or two demonstrations, achieving ICIL through two key components. First, we introduce inductive biases through a graph representation and model ICIL as a graph generation problem with a learned diffusion process, enabling structured reasoning over demonstrations, observations, and actions. Second, we show that such a model can be trained using pseudo-demonstrations - arbitrary trajectories generated in simulation - as a virtually infinite pool of training data. Simulated and real experiments show that Instant Policy enables rapid learning of various everyday robot tasks. We also show how it can serve as a foundation for cross-embodiment and zero-shot transfer to language-defined tasks. Code and videos are available at https://www.robot-learning.uk/instant-policy.

Figures

Figures reproduced from arXiv: 2411.12633 by the authors.

Figure 1
Figure 1. Instant Policy acquires skills instantly after providing demos at test time. We model in￾context imitation learning as a graph-based diffusion process, trained using pseudo-demonstrations. 1 INTRODUCTION Robot policies acquired through Imitation Learning (IL) have recently shown impressive capabili￾ties, but today’s Behavioural Cloning (BC) methods still require hundreds or thousands of demon￾strations per task (Zha… view at source ↗
Figure 2
Figure 2. (Left) A local graph, Gl , representing the robot’s state (blue nodes) and local geometries of the objects (green nodes). (Right) A graph representing 2 demos (3 waypoints each), the current state, and 2 future actions. Edge colours represent different edge types in a heterogeneous graph. Local Representation. The core building block of our representation is the observation at time step t, which we express as a loca… view at source ↗
Figure 3
Figure 3. (Left) High-level structure of the network used to train graph-based diffusion model. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Examples of the simulated trajectories - 3 pseudo-demonstrations for 2 pseudo-tasks. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Attention weights visualised on sub-graph edges at two different timesteps in the phone [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Validation loss curves for three different [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The 16 tasks used in our real-world evaluation. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Objects used in the generalisation ex￾periment (numbers indicate their usage stage). Generalisation to Novel Objects. While all of our previous experiments focused on evaluat￾ing our method’s performance on the same ob￾jects used in the demonstrations, here we aim to t…
Figure 9
Figure 9. Figure 9: High-level structure of the occupancy network. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: High-level overview of the training process. (Left) A data point is sampled from the [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos

    cs.RO 2025-09 conditional novelty 7.0 of 10

    Trained only on unlabeled human play videos, MimicDroid lets a GR1 humanoid perform new manipulation tasks from one to three demonstration videos, with roughly twice the real-world success of prior video-conditioned methods.

  2. Adaptation of Generalist Robot Policies with Minimal Data

    cs.RO 2026-08 conditional novelty 6.0 of 10

    MiDAS, a two-stage recipe of one-demo behavior cloning plus residual online RL on a frozen VLA backbone, reaches high success from a single demonstration in simulation and improves real-world bimanual manipulation.

  3. Try Once, Then Optimal: De-Redundified Procedure Memory for Cross-Episode Exploration Amortization

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Object-centric procedure memory amortizes hidden-state exploration across encounters, cutting robot manipulation operations 16–30% at non-regressing success.

  4. Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)

    cs.RO 2025-09 conditional novelty 6.0 of 10

    A robot policy trained on one real demonstration plus AI-generated 3D views succeeds from novel initial poses, including opposite-side starts, across six real manipulation tasks.

  5. Knowledge-Driven Imitation Learning: Enabling Generalization Across Diverse Conditions

    cs.RO 2025-06 conditional novelty 6.0 of 10

    A semantic keypoint graph matched to novel objects lets imitation-learned manipulation policies generalize with a quarter of the demonstrations.

  6. Novel Demonstration Generation with Gaussian Splatting Enables Robust One-Shot Manipulation

    cs.RO 2025-04 conditional novelty 6.0 of 10

    RoboSplat edits 3D Gaussian scene reconstructions to synthesize diverse robot demonstrations from one expert trajectory, and behavior-cloned policies trained on this data generalize robustly across six disturbance typ...

Reference graph

Works this paper leans on

33 extracted references · 11 canonical work pages · cited by 6 Pith papers

  1. [2]

    We did so to match the distribution of object poses to the one present in our generated pseudo-demonstrations

    We restricted the orientations of the objects in the environment to be within [−π/3, π/3]. We did so to match the distribution of object poses to the one present in our generated pseudo-demonstrations. It also ensured that most tasks could be solved without complex motions requiring motion planners. G F AILURE CASES Here we discuss the observed failure mo...

    work page 2020

  2. [3]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165,

    arXiv 2005

  3. [4]

    To add noise to the action expressed as (TEA ∈ SE(3), ag ∈ R, we first project TEA to se(3) using a Logmap, normalise the resulting vectors, add the noise as described by Ho et al

    the prediction and ground truth labels are used to calculate the loss function, and weights of the network are updated accordingly. To add noise to the action expressed as (TEA ∈ SE(3), ag ∈ R, we first project TEA to se(3) using a Logmap, normalise the resulting vectors, add the noise as described by Ho et al. (2020), unnormalise the result and extract t...

    work page 2020

  4. [6]

    Keypoint action tokens enable in-context imitation learning in robotics

    Norman Di Palo and Edward Johns. Keypoint action tokens enable in-context imitation learning in robotics. arXiv preprint arXiv:2403.19578,

    arXiv

  5. [9]

    Ogb-lsc: A large-scale challenge for machine learning on graphs

    Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-lsc: A large-scale challenge for machine learning on graphs. arXiv preprint arXiv:2103.09430,

    arXiv

  6. [11]

    Vid2robot: End-to-end video- conditioned policy learning with cross-attention transformers

    Vidhi Jain, Maria Attarian, Nikhil J Joshi, Ayzaan Wahid, Danny Driess, Quan Vuong, Pannag R Sanketi, Pierre Sermanet, Stefan Welker, Christine Chan, et al. Vid2robot: End-to-end video- conditioned policy learning with cross-attention transformers. arXiv preprint arXiv:2403.12943,

    arXiv

  7. [12]

    Rrt-connect: An efficient approach to single-query path planning

    James J Kuffner and Steven M LaValle. Rrt-connect: An efficient approach to single-query path planning. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 2, pp. 995–

    work page 2000

  8. [14]

    Mediapipe: A framework for building perception pipelines

    Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172,

    arXiv 1906

Show all 33 references
  1. [16]

    R+ x: Retrieval and execution from everyday human videos

    Georgios Papagiannis, Norman Di Palo, Pietro Vitiello, and Edward Johns. R+ x: Retrieval and execution from everyday human videos. arXiv preprint arXiv:2407.12957,

    arXiv

  2. [17]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    N Reimers. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084,

    1908 arXiv

  3. [18]

    Body trans- former: Leveraging robot embodiment for policy learning

    Carmelo Sferrazza, Dun-Ming Huang, Fangchen Liu, Jongmin Lee, and Pieter Abbeel. Body trans- former: Leveraging robot embodiment for policy learning. arXiv preprint arXiv:2408.06316 ,

    arXiv

  4. [20]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502,

    2010 arXiv

  5. [21]

    Se (3)-diffusionfields: Learning smooth cost functions for joint grasp and motion optimization through diffusion

    Julen Urain, Niklas Funk, Jan Peters, and Georgia Chalvatzaki. Se (3)-diffusionfields: Learning smooth cost functions for joint grasp and motion optimization through diffusion. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5923–5930. IEEE,

    2023

  6. [22]

    A system for learning continuous human-robot interactions from human-human demonstrations

    David V ogt, Simon Stepputtis, Steve Grehl, Bernhard Jung, and Heni Ben Amor. A system for learning continuous human-robot interactions from human-human demonstrations. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2882–2889. IEEE,

    2017

  7. [23]

    Few-shot in-context imitation learning via implicit graph align- ment

    Vitalis V osylius and Edward Johns. Few-shot in-context imitation learning via implicit graph align- ment. arXiv preprint arXiv:2310.12238, 2023a. Vitalis V osylius and Edward Johns. Where to start? transferring simple skills to complex environ- ments. In Conference on Robot L...

    arXiv

  8. [24]

    Scaling robot learning with semantically imag- ined experience

    Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Jodilyn Peralta, Brian Ichter, et al. Scaling robot learning with semantically imag- ined experience. arXiv preprint arXiv:2302.11550,

    arXiv

  9. [25]

    One-shot imitation learning with invariance matching for robotic manipulation

    Xinyu Zhang and Abdeslam Boularias. One-shot imitation learning with invariance matching for robotic manipulation. arXiv preprint arXiv:2405.13178,

    arXiv

  10. [26]

    Formally, the local encoder encodes the dense point cloud into a set of feature vectors together with their associated positions as: {F i, pi}M i=1 = ϕ(P )

    13 APPENDIX A G EOMETRY ENCODER Here, we describe the local geometry encoder used to represent observations of the environment as a set of nodes. Formally, the local encoder encodes the dense point cloud into a set of feature vectors together with their associated positions as...

    2019

  11. [27]

    It samples M centroids from the dense point cloud and embeds the local geometries around them into feature vectors of size

    enhanced with Nerf-like sine/cosine embeddings (Mildenhall et al., 2021). It samples M centroids from the dense point cloud and embeds the local geometries around them into feature vectors of size

    2021

  12. [29]

    We ensure that the spacing between the subsequent spaces is constant and uniform (1cm and 3 degrees, same as used for the normalisation of actions)

    and three simulated depth cameras. We ensure that the spacing between the subsequent spaces is constant and uniform (1cm and 3 degrees, same as used for the normalisation of actions). Moving objects to different poses, choosing a different starting gripper pose and repeating t...

    2023

  13. [30]

    5 days on a single NVIDIA GeForce RTX 3080-ti) followed by a 50K steps learning rate cool-down period

    optimiser with a 1e−5 learn- ing rate for 2.5M optimisation steps (approx. 5 days on a single NVIDIA GeForce RTX 3080-ti) followed by a 50K steps learning rate cool-down period. For efficient training, we used float16 precision and compiled our models using torch compile capab...

    2019

  14. [31]

    We did so to ensure that the demonstrations did not have arbitrary motions that would not be captured by our observations of segmented point clouds and end-effector poses

    using only Cartesian Space planning - we disregarded all demonstrations that were generated using an RRT-based motion planner (Kuffner & LaValle, 2000). We did so to ensure that the demonstrations did not have arbitrary motions that would not be captured by our observations of...

    2000

  15. [33]

    and manually designing a mapping between these key points and the corresponding robot’s end-effector pose. We model the position of the end-effector to be represented by the mid- way position between the index finger and the thumb and estimate the orientation using an addition...

    2024

  16. [1987]

    Layer normalization

    Jimmy Lei Ba. Layer normalization. arXiv preprint arXiv:1607.06450,

    arXiv

  17. [2000]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,

    arXiv

  18. [2015]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137,

    arXiv

  19. [2017]

    In-context imitation learning via next-token prediction

    Letian Fu, Huang Huang, Gaurav Datta, Lawrence Yunliang Chen, William Chung-Ho Panitch, Fangchen Liu, Hui Li, and Ken Goldberg. In-context imitation learning via next-token prediction. arXiv preprint arXiv:2408.15980,

    arXiv

  20. [2018]

    Masked label prediction: Unified message passing model for semi-supervised classification

    Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv preprint arXiv:2009.03509,

    2009 arXiv

  21. [2019]

    Mimicgen: A data generation system for scalable robot learning using human demonstrations

    Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310.17596,

    arXiv

  22. [2020]

    Shapenet: An information-rich 3d model repository

    Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012,

    arXiv

  23. [2021]

    Perceiver io: A general architecture for structured inputs & outputs

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795,

    arXiv

  24. [2023]

    Zero-shot robotic manipulation with pretrained image-editing diffusion models

    Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310.10639,

    arXiv

  25. [2024]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415,

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.