Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

P3-PO: Prescriptive Point Priors for Visuo-Spatial Generalization of Robot Policies

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 5-second human point annotation beats image-based robot policies by 43%.

desk verdict Good idea, honest artifacts, but the headline 43% is confounded by observation history, so the causal claim needs an ablation before it stands. read the letter →

arxiv 2412.06784 v1 pith:G4UWKAQJ submitted 2024-12-09 cs.RO cs.AIcs.CVcs.LG

Mara Levy , Siddhant Haldar , Lerrel Pinto , Abhinav Shirivastava This is my paper · ORCID
classification cs.ROcs.AIcs.CVcs.LG
keywords pointpriorsrobotmanipulationimitationlearningvisualgeneralizationsemanticcorrespondencetrackingbehaviorcloningobject-centricrepresentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that raw-pixel policies entangle object identity, lighting, and background with the spatial structure the policy actually needs, which is why they fail out of distribution. P3-PO replaces the image with a small set of semantically meaningful 3D points: a human annotates one demonstration frame for a few seconds, and pre-trained semantic-correspondence and point-tracking models propagate those points to every other frame of every demonstration. A transformer policy trained on this point stream instead of images is reported to achieve a 43% absolute improvement over strong baselines in training settings, 58% on novel object instances, and 80% with background distractors, across four real-world tasks. If true, the practical upshot is that a few seconds of human prescriptive guidance plus off-the-shelf vision models can substitute for much larger and more diverse robot datasets.

What carries the argument

The load-bearing object is the prescriptive point prior: a set of semantically meaningful points that a human annotates once on a single demonstration frame, typically in under five seconds. DIFT establishes semantic correspondences to the first frames of other demonstrations, Co-Tracker propagates the points through each trajectory, and the resulting 2D points are back-projected to 3D using depth information. An MLP encoder feeds this point stream as tokens into a BAKU causal transformer policy with action chunking, so the policy reasons about spatial relations between annotated points without ever seeing raw pixel values.

What would settle it

Run P3-PO on a task where the tracked object is briefly occluded by a distractor, and compare success rates against the same task without occlusion; a strong correlation between tracker failures and policy failures would confirm the method's dependence on point-propagation reliability.

Watch

Extended reading notes

Core claim

The paper claims that a sparse set of 3D points, prescribed once by a human and propagated through demonstrations by the semantic-correspondence model DIFT and the point tracker Co-Tracker, is a sufficient and highly generalizable state representation for robot manipulation policies. Trained with a BAKU transformer on this point stream, policies are reported to outperform RGB, RGB-D, and object-centric point-cloud baselines on four real-world kitchen tasks, and to generalize to novel object instances, background distractors, and predicted metric depth. The claim is that decoupling perception from planning in this way transfers better than learning visual features from images.

Load-bearing premise

The method assumes the off-the-shelf correspondence and tracking models will reliably carry the human-prescribed points onto every frame of every demonstration, including unseen object instances and under occlusion; when the points drift or disappear, the policy receives incorrect spatial inputs.

Editorial extensions

If this is right

  • Each new task requires only a few seconds of human keypoint annotation, so the method could make robot learning substantially cheaper than collecting additional demonstrations.
  • Because the representation is tied to semantic points rather than pixel appearance, policies trained with P3-PO generalize to novel object instances of the same category, with a claimed 58% improvement over baselines.
  • P3-PO works with predicted depth from Depth Anything 2, which the paper suggests could make the approach applicable to large robot datasets that do not include real depth sensors.
  • The policy does not use proprioception, so the learned representation is robot-agnostic and the same point priors could transfer across different robot embodiments.
  • Point-based input keeps the policy robust to background changes because the raw scene image is never shown to the policy, leading to the claimed 80% improvement in distractor experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The results suggest that pixel appearance is a primary cause of out-of-distribution failure in behavior-cloned manipulation policies; if so, other appearance-invariant representations, such as learned dense descriptors, might yield similar gains without human annotation.
  • The graph-prior experiment, which found no benefit from explicitly encoding pairwise 3D distances, hints that the transformer already extracts the needed spatial relations from raw point coordinates; a testable extension is that richer relational encodings will only help if they capture dynamics rather than static geometry.
  • Because the method's ceiling is tied to the reliability of the point tracker, a practical extension would be to fuse multiple trackers or add a learned fallback when tracking confidence is low.
  • The predicted-depth result, if it holds across more tasks, would let practitioners train manipulation policies directly on internet-scale video where depth is never measured.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. Levy et al. propose P3-PO, a behavior-cloning framework that replaces raw image inputs with a small set of human-prescribed semantic keypoints. The points are annotated once on a single demonstration frame, transferred to all demonstrations with DIFT, propagated through time with Co-Tracker, back-projected to 3D using depth, flattened in annotation order, and encoded by an MLP into a BAKU transformer policy with action chunking. On four xArm kitchen tasks, the paper reports an average 43% absolute improvement over RGB/RGB-D baselines in-domain, 58% on novel object instances, and 80% under background distractors, plus qualitative results on sweeping and microwave-opening tasks and equivalence with predicted depth from Depth Anything 2.

Significance. The proposed pipeline is simple, has low annotation cost, and leverages off-the-shelf vision models, so if the reported gains survive controlled comparisons it would be a practically useful representation for real-world manipulation. The authors commit to releasing datasets and code, explicitly discuss limitations such as tracking failures under occlusion and loss of scene context, and provide supplementary videos. The main weakness is that the headline comparisons are confounded by observation history, and the paper provides no ablation isolating the human point prior; therefore the causal attribution of the gains to point priors is not yet established.

major comments (3)
  1. [Section V-D, Tables I-III] The central comparison is confounded by observation history. Section V-D states that P3-PO and GROOT use observation history while the RGB and RGB-D baselines do not, citing BAKU. Because P3-PO builds on BAKU, which uses action chunking with exponential temporal averaging (Section IV-B), the history mechanism can smooth single-frame perception errors; giving history only to P3-PO means the 43%, 58%, and 80% gaps cannot be attributed to the point representation. GROOT is not a control because it simultaneously changes the representation and performs poorly. The proprioception difference favors the baselines but does not remove this confound. Please add controlled ablations: BAKU with RGB and RGB-D inputs using the same observation history as P3-PO, and a P3-PO variant without history, ideally also varying proprioception.
  2. [Section V-I, Table V] The paper does not ablate the human-prescribed semantic prior. The only input variant tested is a graph-structured encoding of the same keypoints; there is no comparison against random point sets, automatically detected keypoints, or a dense point grid. Since the central claim is that prescriptive point priors, rather than any point input or the transformer architecture, drive generalization, the absence of this ablation leaves the causal role of the one-time human annotation unmeasured.
  3. [Section V-G, Table III] The distractor result is based on five trials per cell, or twenty total trials per method. With binary outcomes, the reported 80% average improvement is extremely sensitive: one or two successes among the baselines move the gap by 20 to 40 percentage points. Please provide more trials, confidence intervals, or per-trial logs before treating the 80% robustness claim as strongly supported.
minor comments (5)
  1. [Section V-A vs Section V-D] Section V-A says the policies are trained with RGB-D images and robot proprioception as input, while Section V-D says P3-PO does not use robot proprioception; please clarify that the first statement refers to the baselines or to the overall data-collection setup.
  2. [Abstract and Section V-E] The arithmetic behind the headline 43%, 58%, and 80% numbers should be stated explicitly, for example as the average per-task success-rate difference against the strongest baseline; otherwise readers cannot reproduce these numbers directly from Tables I-III.
  3. [Section IV-B] The point representation flattens keypoints in the order in which they were annotated, but the paper does not analyze sensitivity to point count, point choice, or point ordering; a short study or at least a statement of the default choices would improve reproducibility.
  4. [Section I and Section VII] The paper states that all datasets and code are public, but the text only gives a project website; please include a repository link or checkpoints in the final version.
  5. [Section V-J] The sweeping and open-microwave results are useful feasibility demonstrations, but they do not include baseline comparisons and should not be read as evidence of relative improvement over prior methods.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: P3-PO is an empirical representation study measured by robot rollouts; the observation-history mismatch is a confound, not a circular reduction.

full rationale

P3-PO's derivation chain is: a human annotates one demonstration frame; DIFT initializes semantic correspondences on other first frames; Co-Tracker propagates the points through each trajectory; points are back-projected to 3D and encoded into the BAKU transformer policy, which is trained by behavior cloning. No target quantity is defined as the input, and no fitted parameter is renamed as a prediction. The headline 43%, 58%, and 80% claims are empirical success rates from physical robot trials against RGB, RGB-D, and GROOT baselines, so they are measurements, not consequences of a construction. The same-group citations (BAKU [18], Open Teach [50]) supply an off-the-shelf architecture and teleoperation tool; the central contribution is the point-prior representation, and the BAKU-as-RGB-baseline comparison tests that contribution rather than assuming it. Section V-D's disclosure that P3-PO uses observation history while RGB/RGB-D baselines do not is a genuine experimental confound for attributing the gains to point priors, but it is not circularity: no equation reduces to its own output and no input is fitted to the evaluation target. The Section VI limitation about tracking failures under occlusion concerns input-pipeline robustness, not circularity. Thus, no circular step is identified.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical entities. It relies on human-prescribed keypoints as a hand-chosen representation, plus several off-the-shelf vision-model assumptions. The free parameters are largely human choices and fixed hyperparameters rather than fitted physical constants, but they still shape the central result and are not systematically ablated.

free parameters (3)
  • semantic keypoint selection = human-chosen points on one frame, not quantitative
    The central representation depends on which points the annotator picks; the paper provides no rule for choosing them, only that they are semantically meaningful and take under 5 seconds. This is a hand-chosen, task-specific prior.
  • point ordering in flattened vector = annotation order
    Section IV-B states the points are flattened into a single vector in the order they are annotated. This arbitrary ordering may affect the learned policy and is not ablated.
  • DIFT correspondence configuration = first layer of the hundredth time step, ensemble size 4
    Section V-D specifies these hyperparameters for semantic correspondence without reporting a sweep or sensitivity analysis, so their effect on point quality is unknown.
assumptions (5)
  • domain assumption DIFT produces correct semantic correspondences across object instances and scenes.
    Section IV-A uses DIFT to transfer points from one annotated frame to first frames of all other demonstrations and to the first frame at inference; if this fails, the policy receives wrong points.
  • domain assumption Co-Tracker tracks points accurately through trajectories, including occlusions and camera motion.
    Section IV-A uses Co-Tracker to propagate points through every frame of each trajectory. Section VI explicitly concedes that tracking failures under occlusion hurt performance.
  • domain assumption RGB-D depth from the camera is accurate enough for 3D back-projection of points.
    Section IV-B back-projects each 2D point to 3D using depth information; Section V-K attributes some failures to depth noise.
  • domain assumption Expert demonstrations collected via VR teleoperation are sufficient for behavior cloning.
    Section V-A collects 160 demonstrations with Open Teach and trains policies via imitation; this is the standard behavior-cloning setup the paper adopts.
  • domain assumption A transformer policy can learn successful actions from point tokens without raw pixels or proprioception.
    Section IV-B and Section V-D describe feeding only point tokens into BAKU and omitting proprioception; the paper provides no ablation showing this information is sufficient in general.

how reviews work

0 comments
Cite this review

Pith. "Pith review of P3-PO: Prescriptive Point Priors for Visuo-Spatial Generalization of Robot Policies." pith.science (2026). https://pith.science/paper/G4UWKAQJ

@misc{pith2026241206784,
  author       = {Pith},
  title        = {Pith review of: P3-PO: Prescriptive Point Priors for Visuo-Spatial Generalization of Robot Policies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G4UWKAQJ}},
  note         = {Machine review of arXiv:2412.06784}
}
read the original abstract

Developing generalizable robot policies that can robustly handle varied environmental conditions and object instances remains a fundamental challenge in robot learning. While considerable efforts have focused on collecting large robot datasets and developing policy architectures to learn from such data, naively learning from visual inputs often results in brittle policies that fail to transfer beyond the training data. This work presents Prescriptive Point Priors for Policies or P3-PO, a novel framework that constructs a unique state representation of the environment leveraging recent advances in computer vision and robot learning to achieve improved out-of-distribution generalization for robot manipulation. This representation is obtained through two steps. First, a human annotator prescribes a set of semantically meaningful points on a single demonstration frame. These points are then propagated through the dataset using off-the-shelf vision models. The derived points serve as an input to state-of-the-art policy architectures for policy learning. Our experiments across four real-world tasks demonstrate an overall 43% absolute improvement over prior methods when evaluated in identical settings as training. Further, P3-PO exhibits 58% and 80% gains across tasks for new object instances and more cluttered environments respectively. Videos illustrating the robot's performance are best viewed at point-priors.github.io.

Figures

Figures reproduced from arXiv: 2412.06784 by the authors.

Figure 1
Figure 1. A human prescribes key points one time for one [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the Prescriptive Point Priors for Policies ( [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Results of the correspondence model when used on [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Illustration of objects used in our experiments. In each [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Real-world rollouts showing P3-PO’s capability on [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Rollouts of the two complex tasks. On the top we show that for both of these tasks P3-PO can generalize to the object [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Demonstrations of success and failure for each task. On the left we show successful demonstrations and on the right [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Feel the Force: Contact-Driven Learning from Humans

    cs.RO 2025-06 conditional novelty 7.0 of 10

    FeelTheForce trains a robot policy on human tactile demonstrations, predicting desired contact forces and using a PD controller to track them on the robot gripper, achieving 77% success across five force-sensitive tasks.

  2. Touch begins where vision ends: Generalizable policies for contact-rich manipulation

    cs.RO 2025-06 conditional novelty 6.0 of 10

    A localize-then-execute policy that combines vision-language reaching, semantic background augmentation, and residual reinforcement learning with tactile sensing reaches about 90% success on millimeter-precision manip...

  3. EgoZero: Robot Learning from Smart Glasses

    cs.RO 2025-05 conditional novelty 6.0 of 10

    Robot policies trained only on egocentric human videos from smart glasses transfer zero-shot to a Franka gripper, with 70% success across 7 manipulation tasks.

Reference graph

Works this paper leans on

71 extracted references · 37 canonical work pages · cited by 3 Pith papers

  1. [1]

    Improving image generation with better captions,

    J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y . Guo, et al., “Improving image generation with better captions,” Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, vol. 2, no. 3, p. 8, 2023

    work page 2023

  2. [2]

    Photorealistic text-to-image diffusion models with deep language understanding,

    C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. , “Photorealistic text-to-image diffusion models with deep language understanding,” Advances in neural information processing systems , vol. 35, pp. 36 479–36 494, 2022

    work page 2022

  3. [3]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  4. [4]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lil- licrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al. , “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024

    arXiv 2024

  5. [5]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  6. [6]

    Open x- embodiment: Robotic learning datasets and rt-x models,

    A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al., “Open x- embodiment: Robotic learning datasets and rt-x models,” arXiv preprint arXiv:2310.08864, 2023

    arXiv 2023

  7. [7]

    Learning fine-grained bimanual manipulation with low-cost hardware,

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” arXiv preprint arXiv:2304.13705 , 2023

    arXiv 2023

  8. [8]

    On bringing robots home,

    N. M. M. Shafiullah, A. Rai, H. Etukuru, Y . Liu, I. Misra, S. Chintala, and L. Pinto, “On bringing robots home,” arXiv preprint arXiv:2311.16098, 2023

    arXiv 2023

Show all 71 references
  1. [9]

    Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots,

    C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song, “Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots,” arXiv preprint arXiv:2402.10329, 2024

    2024 arXiv

  2. [10]

    Robot utility models: General policies for zero-shot deploy- ment in new environments,

    H. Etukuru, N. Naka, Z. Hu, S. Lee, J. Mehu, A. Edsinger, C. Paxton, S. Chintala, L. Pinto, and N. M. M. Shafiullah, “Robot utility models: General policies for zero-shot deploy- ment in new environments,” arXiv preprint arXiv:2409.05865 , 2024

    2024 arXiv

  3. [11]

    Graph-structured visual imitation,

    M. Sieb, Z. Xian, A. Huang, O. Kroemer, and K. Fragkiadaki, “Graph-structured visual imitation,” in Conference on Robot Learning. PMLR, 2020, pp. 979–989

    2020

  4. [12]

    Viola: Imitation learning for vision-based manipulation with object proposal priors,

    Y . Zhu, A. Joshi, P. Stone, and Y . Zhu, “Viola: Imitation learning for vision-based manipulation with object proposal priors,” in Conference on Robot Learning . PMLR, 2023, pp. 1199–1210

    2023

  5. [13]

    Learning generalizable manipulation policies with object-centric 3d representations,

    Y . Zhu, Z. Jiang, P. Stone, and Y . Zhu, “Learning generalizable manipulation policies with object-centric 3d representations,” in 7th Annual Conference on Robot Learning , 2023

    2023

  6. [14]

    Tax-pose: Task-specific cross-pose estimation for robot manipulation,

    C. Pan, B. Okorn, H. Zhang, B. Eisner, and D. Held, “Tax-pose: Task-specific cross-pose estimation for robot manipulation,” in Conference on Robot Learning . PMLR, 2023, pp. 1783–1792

    2023

  7. [15]

    One-shot imitation learning: A pose estimation perspective,

    P. Vitiello, K. Dreczkowski, and E. Johns, “One-shot imitation learning: A pose estimation perspective,” arXiv preprint arXiv:2310.12077, 2023

    2023 arXiv

  8. [16]

    Emergent correspondence from image diffusion,

    L. Tang, M. Jia, Q. Wang, C. P. Phoo, and B. Hariharan, “Emergent correspondence from image diffusion,” in Thirty- seventh Conference on Neural Information Processing Systems , 2023

    2023

  9. [17]

    Cotracker: It is better to track together,

    N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rupprecht, “Cotracker: It is better to track together,” 2023

    2023

  10. [18]

    Baku: An efficient transformer for multi-task policy learning,

    S. Haldar, Z. Peng, and L. Pinto, “Baku: An efficient transformer for multi-task policy learning,” arXiv preprint arXiv:2406.07539, 2024

    2024 arXiv

  11. [19]

    Depth anything: Unleashing the power of large-scale unlabeled data,

    L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, “Depth anything: Unleashing the power of large-scale unlabeled data,” in CVPR, 2024

    2024

  12. [20]

    Depth anything v2,

    L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao, “Depth anything v2,” arXiv:2406.09414, 2024

    2024 arXiv

  13. [21]

    Imitation learning: A survey of learning methods,

    A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne, “Imitation learning: A survey of learning methods,” ACM Comput. Surv., vol. 50, no. 2, apr 2017

    2017

  14. [22]

    Algorithms for inverse reinforce- ment learning,

    A. Y . Ng and S. J. Russell, “Algorithms for inverse reinforce- ment learning,” in Proceedings of the Seventeenth International Conference on Machine Learning , ser. ICML ’00. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2000, p. 663–670

    2000

  15. [23]

    Apprenticeship learning via inverse reinforcement learning,

    P. Abbeel and A. Y . Ng, “Apprenticeship learning via inverse reinforcement learning,” in Proceedings of the Twenty-First International Conference on Machine Learning , ser. ICML ’04. New York, NY , USA: Association for Computing Machinery, 2004, p. 1

    2004

  16. [24]

    Wayex: Waypoint exploration using a single demonstration,

    M. Levy, N. Saini, and A. Shrivastava, “Wayex: Waypoint exploration using a single demonstration,” 2024

    2024

  17. [25]

    Watch and match: Supercharging imitation with regularized optimal transport,

    S. Haldar, V . Mathur, D. Yarats, and L. Pinto, “Watch and match: Supercharging imitation with regularized optimal transport,” in Conference on Robot Learning . PMLR, 2023, pp. 32–43

    2023

  18. [26]

    Teach a robot to fish: Versatile imitation from one minute of demonstrations,

    S. Haldar, J. Pari, A. Rai, and L. Pinto, “Teach a robot to fish: Versatile imitation from one minute of demonstrations,” 2023

    2023

  19. [27]

    Generative adversarial imitation learning,

    J. Ho and S. Ermon, “Generative adversarial imitation learning,” CoRR, vol. abs/1606.03476, 2016

    2016 arXiv

  20. [28]

    Awac: Accelerat- ing online reinforcement learning with offline datasets,

    A. Nair, A. Gupta, M. Dalal, and S. Levine, “Awac: Accelerat- ing online reinforcement learning with offline datasets,” arXiv preprint arXiv:2006.09359, 2020

    2006 arXiv

  21. [29]

    Alvinn: An autonomous land vehicle in a neural network,

    D. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” in Proceedings of (NeurIPS) Neural Information Processing Systems, D. Touretzky, Ed. Morgan Kaufmann, December 1989, pp. 305 – 313

    1989

  22. [30]

    Recent advances in imitation learning from observation,

    F. Torabi, G. Warnell, and P. Stone, “Recent advances in imitation learning from observation,” arXiv preprint arXiv:1905.13566, 2019

    1905 arXiv

  23. [31]

    Learning from demonstration,

    S. Schaal, “Learning from demonstration,” Advances in neural information processing systems , vol. 9, 1996

    1996

  24. [32]

    A reduction of imitation learning and structured prediction to no-regret online learning,

    S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in Proceedings of the fourteenth international conference on artificial intelligence and statistics . JMLR Workshop and Conference Proceedings, 2011,...

    2011

  25. [33]

    Sequential dexterity: Chaining dexterous policies for long-horizon manip- ulation,

    Y . Chen, C. Wang, L. Fei-Fei, and C. K. Liu, “Sequential dexterity: Chaining dexterous policies for long-horizon manip- ulation,” 2023

    2023

  26. [34]

    Learning to generalize across long-horizon tasks from human demonstrations,

    A. Mandlekar, D. Xu, R. Martín-Martín, S. Savarese, and L. Fei- Fei, “Learning to generalize across long-horizon tasks from human demonstrations,” CoRR, vol. abs/2003.06085, 2020

    2003 arXiv

  27. [35]

    Cliport: What and where pathways for robotic manipulation,

    M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where pathways for robotic manipulation,” CoRR, vol. abs/2109.12098, 2021

    2021 arXiv

  28. [36]

    Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking,

    H. Bharadhwaj, J. Vakil, M. Sharma, A. Gupta, S. Tulsiani, and V . Kumar, “Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking,” arXiv preprint arXiv:2309.01918 , 2023

    2023 arXiv

  29. [37]

    Track2act: Predicting point tracks from internet videos en- ables diverse zero-shot robot manipulation,

    H. Bharadhwaj, R. Mottaghi, A. Gupta, and S. Tulsiani, “Track2act: Predicting point tracks from internet videos en- ables diverse zero-shot robot manipulation,” arXiv preprint arXiv:2405.01527, 2024

    2024 arXiv

  30. [38]

    Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,

    T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Gold- berg, and P. Abbeel, “Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,” 2018

    2018

  31. [39]

    Diffusion policy: Visuomotor policy learning via action diffusion,

    C. Chi, S. Feng, Y . Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” in Proceedings of Robotics: Science and Systems (RSS), 2023

    2023

  32. [40]

    Bc-z: Zero-shot task generalization with robotic imitation learning,

    E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, “Bc-z: Zero-shot task generalization with robotic imitation learning,” in Conference on Robot Learning. PMLR, 2022, pp. 991–1002

    2022

  33. [41]

    Deep object- centric representations for generalizable robot learning,

    C. Devin, P. Abbeel, T. Darrell, and S. Levine, “Deep object- centric representations for generalizable robot learning,” CoRR, vol. abs/1708.04225, 2017

    2017 arXiv

  34. [42]

    One-shot imitation learning,

    Y . Duan, M. Andrychowicz, B. C. Stadie, J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba, “One-shot imitation learning,” CoRR, vol. abs/1703.07326, 2017

    2017 arXiv

  35. [43]

    Anygrasp: Robust and efficient grasp perception in spatial and temporal domains,

    H.-S. Fang, C. Wang, H. Fang, M. Gou, J. Liu, H. Yan, W. Liu, Y . Xie, and C. Lu, “Anygrasp: Robust and efficient grasp perception in spatial and temporal domains,”IEEE Transactions on Robotics (T-RO), 2023

    2023

  36. [44]

    Deep object pose estimation for semantic robotic grasping of household objects,

    J. Tremblay, T. To, B. Sundaralingam, Y . Xiang, D. Fox, and S. Birchfield, “Deep object pose estimation for semantic robotic grasping of household objects,” CoRR, vol. abs/1809.10790, 2018

    2018 arXiv

  37. [45]

    6-dof pose estimation of household objects for robotic manipulation: An accessible dataset and benchmark,

    S. Tyree, J. Tremblay, T. To, J. Cheng, T. Mosier, J. Smith, and S. Birchfield, “6-dof pose estimation of household objects for robotic manipulation: An accessible dataset and benchmark,” in International Conference on Intelligent Robots and Systems (IROS), 2022

    2022

  38. [46]

    Reagent: Point cloud registration using imitation and reinforcement learning,

    D. Bauer, T. Patten, and M. Vincze, “Reagent: Point cloud registration using imitation and reinforcement learning,” 2021

    2021

  39. [47]

    Robo-abc: Affordance generalization beyond categories via semantic correspondence for robot manipulation,

    Y . Ju, K. Hu, G. Zhang, G. Zhang, M. Jiang, and H. Xu, “Robo-abc: Affordance generalization beyond categories via semantic correspondence for robot manipulation,” in European Conference on Computer Vision. Springer, 2025, pp. 222–239

    2025

  40. [48]

    An autonomous land vehicle in a neural network,

    D. Pomerleau, “An autonomous land vehicle in a neural network,” Advances in Neural Information Processing Systems , vol. 1, 1998

    1998

  41. [49]

    Supervised policy learning for real robots,

    N. M. M. Shafiullah, S. Feng, L. Pinto, and R. Tedrake, “Supervised policy learning for real robots,” July 2024, tutorial presented at the Robotics: Science and Systems (RSS), Delft. [Online]. Available: https://supervised-robot-learning.github.io

    2024

  42. [50]

    Open teach: A versatile teleoperation system for robotic manipulation,

    A. Iyer, Z. Peng, Y . Dai, I. Guzey, S. Haldar, S. Chintala, and L. Pinto, “Open teach: A versatile teleoperation system for robotic manipulation,” arXiv preprint arXiv:2403.07870 , 2024

    2024 arXiv

  43. [51]

    From play to policy: Conditional behavior generation from uncurated robot data,

    Z. J. Cui, Y . Wang, N. M. M. Shafiullah, and L. Pinto, “From play to policy: Conditional behavior generation from uncurated robot data,” arXiv preprint arXiv:2210.10047 , 2022

    2022 arXiv

  44. [52]

    Behavior generation with latent actions,

    S. Lee, Y . Wang, H. Etukuru, H. J. Kim, N. M. M. Shafiullah, and L. Pinto, “Behavior generation with latent actions,” arXiv preprint arXiv:2403.03181, 2024

    2024 arXiv

  45. [53]

    Lindeberg, Scale Invariant Feature Transform , 05 2012, vol

    T. Lindeberg, Scale Invariant Feature Transform , 05 2012, vol. 7

    2012

  46. [54]

    Image registration methods: a survey,

    B. Zitova and J. Flusser, “Image registration methods: a survey,” Image and vision computing , vol. 21, no. 11, pp. 977–1000, 2003

    2003

  47. [55]

    Nerf: Representing scenes as neural radiance fields for view synthesis,

    B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2021

    2021

  48. [56]

    3d gaussian splatting for real-time radiance field rendering

    B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering.” ACM Trans. Graph., vol. 42, no. 4, pp. 139–1, 2023

    2023

  49. [57]

    Particle video revisited: Tracking through occlusions using point trajectories,

    A. W. Harley, Z. Fang, and K. Fragkiadaki, “Particle video revisited: Tracking through occlusions using point trajectories,” in ECCV, 2022

    2022

  50. [58]

    Pointodyssey: A large-scale synthetic dataset for long-term point tracking,

    Y . Zheng, A. W. Harley, B. Shen, G. Wetzstein, and L. J. Guibas, “Pointodyssey: A large-scale synthetic dataset for long-term point tracking,” in ICCV, 2023

    2023

  51. [59]

    Tapir: Tracking any point with per-frame initialization and temporal refinement,

    C. Doersch, Y . Yang, M. Vecerik, D. Gokay, A. Gupta, Y . Aytar, J. Carreira, and A. Zisserman, “Tapir: Tracking any point with per-frame initialization and temporal refinement,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 10 061–10 072

    2023

  52. [60]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Dollár, and R. Girshick, “Segment anything,” 2023

    2023

  53. [61]

    Deep learning in medical image registration: a review,

    Y . Fu, Y . Lei, T. Wang, W. J. Curran, T. Liu, and X. Yang, “Deep learning in medical image registration: a review,”Physics in Medicine & Biology , vol. 65, no. 20, p. 20TR01, 2020

    2020

  54. [62]

    Learning semantic correspondence with sparse annotations,

    S. Huang, L. Yang, B. He, S. Zhang, X. He, and A. Shrivastava, “Learning semantic correspondence with sparse annotations,” in Proceedings of the European Conference on Computer Vision(ECCV), 2022

    2022

  55. [63]

    Asic: Aligning sparse in-the-wild image collections,

    K. Gupta, V . Jampani, C. Esteves, A. Shrivastava, A. Makadia, N. Snavely, and A. Kar, “Asic: Aligning sparse in-the-wild image collections,” in Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV) , October 2023, pp. 4134–4145

    2023

  56. [64]

    Human motion analysis: A review,

    J. K. Aggarwal and Q. Cai, “Human motion analysis: A review,” Computer vision and image understanding , vol. 73, no. 3, pp. 428–440, 1999

    1999

  57. [65]

    Object tracking: A survey,

    A. Yilmaz, O. Javed, and M. Shah, “Object tracking: A survey,” Acm computing surveys (CSUR), vol. 38, no. 4, pp. 13–es, 2006

    2006

  58. [66]

    Visual odometry,

    D. Nistér, O. Naroditsky, and J. Bergen, “Visual odometry,” in Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., vol. 1. Ieee, 2004, pp. I–I

    2004

  59. [67]

    Droid: A large-scale in-the-wild robot manipulation dataset,

    A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis, et al. , “Droid: A large-scale in-the-wild robot manipulation dataset,” arXiv preprint arXiv:2403.12945 , 2024

    2024 arXiv

  60. [68]

    Graph inverse reinforcement learning from diverse videos,

    S. Kumar, J. Zamora, N. Hansen, R. Jangir, and X. Wang, “Graph inverse reinforcement learning from diverse videos,” in Conference on Robot Learning . PMLR, 2023, pp. 55–66

    2023

  61. [69]

    Graph-structured visual imitation,

    M. Sieb, X. Zhou, A. Huang, O. Kroemer, and K. Fragkiadaki, “Graph-structured visual imitation,” in Conference on Robot Learning, 2019. [Online]. Available: https://api.semanticscholar. org/CorpusID:196470945

    2019

  62. [70]

    Graph learning in robotics: A survey,

    F. Pistilli and G. Averta, “Graph learning in robotics: A survey,” IEEE Access, vol. PP, pp. 1–1, 01 2023

    2023

  63. [71]

    Offline imitation learning through graph search and retrieval,

    Z.-H. Yin and P. Abbeel, “Offline imitation learning through graph search and retrieval,” Robotics: Science and Systems , 2024

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.