Pith. sign in

REVIEW 3 major objections 6 minor 44 references

3D robot actions can be learned as pixel classification on camera images, not as continuous regression or huge discrete tokens.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 09:51 UTC pith:FGZL7UEN

load-bearing objection Clean closed-loop idea: multi-view keypoint heatmaps + CE + triangulation beats DiffPo/ACT/OAT on MimicGen and real long-horizon tasks, with real speed and laser-cue wins; main soft spots are eval hygiene and calibrated multi-view dependence, not a broken claim. the 3 major comments →

arxiv 2607.10706 v1 pith:FGZL7UEN submitted 2026-07-12 cs.RO cs.AIcs.CVcs.LG

Action Map Policy: Learning 3D Closed-loop Manipulation via Pixel Classification

Haojie Huang , Zhang Ye , Linfeng Zhao , Boce Hu , Mingxi Jia , Yu Qi , Ahmed Agha , Dian Wang
show 2 more authors
Robert Platt Robin Walters
This is my paper
classification cs.RO cs.AIcs.CVcs.LG
keywords robot manipulationimitation learningaction representationpixel classificationkeypoint heatmapsclosed-loop visuomotor policymulti-view triangulationdiffusion policy baseline
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Robot policies struggle with high-dimensional, multi-modal, long-horizon actions. This paper claims that the right fix is to stop treating actions as free continuous vectors or as an exploding codebook of tokens. Instead it projects gripper keypoints into the camera image planes and asks the network to classify which pixel each keypoint should hit at each future step. The resulting heatmaps keep multi-modality, preserve millimeter-scale precision at ordinary image resolutions, and let the whole action chunk be predicted in one forward pass. On simulated and real manipulation tasks the method beats strong regression, token, and diffusion baselines on success rate, inference speed, and response to tiny spatial cues such as a laser pointer. A sympathetic reader cares because the formulation aligns observation and action in the same spatial grid, turning policy learning into ordinary dense classification while still recovering executable 3D poses by triangulation.

Core claim

Action Map Policy shows that 3D closed-loop manipulation can be cast as multi-view pixel classification of projected action keypoints. Predicting dense heatmaps over image planes, then recovering poses by argmax and triangulation, yields higher success rates than regression, tokenization, and diffusion baselines, single-pass inference much faster than diffusion, and stronger use of fine-grained visual cues, all without a combinatorial action vocabulary.

What carries the argument

Action Map Policy (AMP) with the X-Net backbone: 3D end-effector poses are converted to a fixed set of gripper keypoints, projected into multi-view soft heatmaps, and the network is trained with cross-entropy to predict those heatmaps over a temporal horizon; executable actions are recovered only at inference by argmax plus triangulation.

Load-bearing premise

That the single brightest pixel in each independently predicted multi-view heatmap, once triangulated with the known cameras, recovers accurate and temporally consistent 3D keypoint trajectories under real calibration error, occlusion, and motion that leaves the calibrated workspace.

What would settle it

On the real coffee, toast, or egg tasks, replace the argmax-plus-triangulation step with ground-truth projected pixels or with deliberately noisy heatmaps; if success rates collapse or fail to match the reported gains over diffusion and ACT, the claim that pixel classification plus triangulation is sufficient fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Millimeter translation and degree-level rotation precision become available at ordinary 224 imes224 resolution without building a 10^10-token action codebook.
  • Closed-loop action chunks can be generated in a single forward pass, removing the need for multi-step denoising schedules at inference.
  • Because observation and action live on the same pixel grid, equivariant image augmentations transfer directly to the action labels.
  • The explicit per-pixel action distribution supplies a natural interface for later reweighting or reinforcement-learning fine-tuning.
  • The same keypoint-heatmap design extends to other end-effectors by changing only the geometric keypoint layout.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The classification objective may transfer more cleanly into vision-language-action models than continuous regression or diffusion, because both language and pixel heatmaps are already discrete token spaces.
  • Independent per-view argmax can still produce inconsistent 3D geometry under heavy occlusion; a joint multi-view consistency regularizer would be a natural next test.
  • If the heatmaps are already multi-modal, sampling from them rather than always taking argmax could give a cheap way to explore alternative grasps without a second generative model.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Action Map Policy (AMP), which reformulates 3D closed-loop visuomotor policy learning as multi-view pixel classification: end-effector poses are encoded as a fixed set of 3D keypoints (m=5), projected into calibrated camera planes, and supervised as soft Gaussian heatmaps over a temporal action chunk. An X-Net backbone (U-Net encoders, multi-view transformer with in- and cross-image attention, U-Net decoders) predicts dense heatmaps; inference recovers 3D keypoints by per-view argmax plus triangulation, then reconstructs pose via centroid and Gram–Schmidt axes. The method is trained with cross-entropy only, yields single-pass action chunks, and is evaluated on six MimicGen tasks plus five real-world tasks (three long-horizon, two laser-pointer spatial-reasoning), reporting higher success rates than DiffPo, ACT, OAT, and Motion Track, faster inference than diffusion, and millimeter-scale reconstruction precision at 224×224 (Table 1 / Fig. 4).

Significance. If the empirical gains hold under fair comparison, AMP is a useful contribution to action representation for imitation learning: it sidesteps combinatorial token vocabularies while retaining multi-modality via classification, aligns action and observation in image space (enabling equivariant augmentation), and offers single-pass inference that is practically faster than iterative diffusion. The precision analysis (Table 1) is concrete and falsifiable; the laser-pointer experiments give a clear demonstration of fine-grained spatial responsiveness that regression and diffusion baselines struggle with. Strengths include a clean geometric pipeline (projection/triangulation, Algorithms 1–2), ablations on soft labels and in-hand view (Table 3), and real-robot validation on long-horizon tasks with reported wall-clock latency. These make the work of interest to the robot learning community even if some engineering dependencies remain.

major comments (3)
  1. Table 2 (and Tables 4–5): success rates are reported as single percentages over 50 (sim) or 20 (real) trials with no standard errors, confidence intervals, or multi-seed statistics. The headline average gain of ~20.7% over DiffPo and the large real-world margins (e.g., 80–90% vs 15–40%) are load-bearing for the central claim; without variance or seed-level reporting it is hard to judge whether differences are robust, especially on threading-d2 (30% vs 26%) and under the small real-world demo budgets (70/40 demos).
  2. Method §3 and Algorithms 1–2: inference treats each view’s heatmap independently (argmax then triangulation T). Table 1 / Fig. 4 quantify only ideal discretized (P,T) reconstruction error on ground-truth projections, not closed-loop error under network prediction, occlusion, or partial FOV loss. The paper notes workspace limits and calibration offsets (§5, footnote 2) and truncates out-of-scope keypoints (Appendix 6.2), but does not measure cross-view argmax consistency or pose recovery failure rates when one view is occluded. A quantitative consistency/ablation (e.g., drop one camera at test time; report triangulation residual and success) is needed to support the claim that millimeter-level policy precision is achieved in the full pipeline, not only in the geometric map.
  3. §4.2 baselines and Appendix 6.4: equivariant joint image–heatmap augmentation is a structural advantage of AMP (Table 7 shows large drops without it: −32/−26/−12 points). It is unclear whether DiffPo, ACT, OAT, and Motion Track received comparably strong multi-view geometric augmentation or only standard image augmentations. If baselines were not given an analogous geometric prior, part of the reported gap may be attributable to data augmentation rather than the classification action map itself. Please state the exact augmentation protocol for every baseline and, if possible, re-run the strongest baseline with the strongest applicable multi-view augmentation.
minor comments (6)
  1. Figure 1 caption and §1: the comparison of multi-modality handling is clear conceptually, but a short quantitative multi-modality diagnostic (e.g., entropy of heatmaps vs mode collapse rate of ACT on the laser-pointer tasks) would strengthen the narrative beyond success rates alone.
  2. §3, gripper width formula: w = d+/(d++d−) with p5 is fine for a parallel jaw, but the text should state explicitly that p1–p4 distances are fixed and cannot encode aperture (currently only in footnote 1).
  3. Implementation: decoder outputs 60 channels (5 keypoints × 12 steps) but only the first 8 steps are executed; justify the train/execute horizon mismatch and whether longer executed horizons hurt consistency.
  4. Related work: Motion Track [39] and dense open-loop methods [17–22] are discussed; a clearer sentence on how AMP differs from 2D keypoint diffusion (independent coordinate denoising vs joint dense classification with cross-view attention) would help readers place the contribution.
  5. Typos / polish: abstract and intro repeat similar claims; “Muti-view” in Fig. 2; arXiv IDs in references for concurrent work (OAT, etc.) should be double-checked for final citation form.
  6. Appendix 6.5 visualization is helpful; consider moving one qualitative multi-view heatmap figure into the main paper near the spatial-reasoning results.

Circularity Check

0 steps flagged

No significant circularity: empirical policy method whose success rates and precision claims are measured against external tasks and reconstruction geometry, not forced by definition or self-citation.

full rationale

Action Map Policy is a methods paper that reformulates closed-loop 3D manipulation as multi-view pixel classification of projected keypoints, trains with cross-entropy on soft heatmap labels, and recovers poses by argmax + triangulation + Gram–Schmidt. The central claims (higher success rates than DiffPo/ACT/OAT/Motion Track, single-pass speed, laser-pointer spatial reasoning, ~1 mm reconstruction at 224×224) are evaluated on held-out MimicGen and real-world trials; they are not algebraic identities of the training objective or of fitted parameters. Soft-label σ, chunk length, and architecture are ordinary hyperparameters (ablated in Tables 3, 6, 7), not quantities later re-presented as predictions. The (P,T) bijection and Table 1 reconstruction errors quantify discretization of known camera geometry on demonstration data; they do not circularly manufacture the policy success rates. Self-citations (e.g., prior transporter/equivariant work) appear only as related work and do not load-bear uniqueness or force the AMP formulation. No equation reduces a claimed prediction to a fitted input by construction. Score 0 is therefore appropriate.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central claim rests on standard multi-view geometry, imitation-learning assumptions, and several design choices (keypoint set, soft-label width, which views predict heatmaps). No new physical entities are postulated; X-Net and AMP are architectural constructs. Free parameters are ordinary ML hyperparameters that affect reported success rates.

free parameters (4)
  • soft-label Gaussian width σ
    Chosen and ablated (σ=0/2/4); σ=2 selected for best success. Directly shapes the classification targets and reported performance.
  • keypoint count and gripper geometry (m=5)
    Hand-designed mapping from pose (T,R,w) to five 3D points; recovery formulas for T,R,w depend on this fixed layout.
  • action chunk length and execution horizon (l=12, execute first 8)
    Design choice controlling temporal output channels and closed-loop replan frequency; not derived from first principles.
  • X-Net capacity and training schedule (ResBlocks, 6-layer MVT, lr 1e-4, 200 epochs, etc.)
    Standard fitted architecture/optimization choices that the empirical claim depends on.
axioms (4)
  • domain assumption Camera intrinsics/extrinsics are known and multi-view projection plus triangulation recover 3D points sufficiently for control.
    Stated in Setup and Heatmap-to-Action Conversion; precision table assumes this geometry.
  • domain assumption Expert demonstrations define the target multi-modal action distribution to imitate.
    Standard imitation-learning premise underlying all reported success rates.
  • standard math Under generic camera configurations, projection and triangulation form a bijection between R³ and the multi-view image of points.
    Invoked in §3 to justify representing 3D actions as multi-view 2D heatmaps.
  • ad hoc to paper Cross-entropy on soft pixel labels is an adequate surrogate for multi-modal trajectory learning without explicit 3D geometric consistency loss.
    Training is supervised only in heatmap space; jitter and consistency issues are deferred to future work in §5.
invented entities (2)
  • Action Map Policy (AMP) action representation independent evidence
    purpose: Encode closed-loop 3D action chunks as spatiotemporal multi-view keypoint heatmaps for classification.
    Core proposed representation; evaluated empirically, not an unobserved physical object.
  • X-Net multi-view backbone independent evidence
    purpose: Map multi-camera images to dense action heatmaps while preserving spatial structure via U-Net + multi-view transformer.
    Architectural invention supporting AMP; success is measured by task metrics, not external detection of a new entity.

pith-pipeline@v1.1.0-grok45 · 19531 in / 3101 out tokens · 35762 ms · 2026-07-14T09:51:33.920162+00:00 · methodology

0 comments
read the original abstract

The action space poses a major challenge in robot learning, since it is often high-dimensional, can span long time horizons, and frequently admits multi-modal optimal solutions. A good choice of action representation and loss function can help to address these concerns, but there are often trade offs. We propose Action Map Policy (AMP), which casts 3D closed-loop manipulation policy learning as a classification problem in image space. While classification has been an effective formulation in generative language models, applying it to robot action learning is difficult because naively discretizing high-dimensional continuous actions explodes the token vocabulary. Our key idea is to project 3D actions onto the camera image planes and treat each pixel location as a discrete class, thus controlling dimensionality while retaining multi-modality. This method supports millimeter-level precision for high-dimensional actions without requiring a prohibitively large vocabulary, while preserving fine-grained pixel-wise visual signals. Furthermore, it can predict the entire action chunk in a single forward pass, avoiding complex noise scheduling and iterative denoising while achieving substantially faster inference than diffusion policies. Experiments on various manipulation tasks show that AMP outperforms strong baselines, achieving higher success rates, faster inference, and enhanced spatial reasoning.

Figures

Figures reproduced from arXiv: 2607.10706 by Ahmed Agha, Boce Hu, Dian Wang, Haojie Huang, Linfeng Zhao, Mingxi Jia, Robert Platt, Robin Walters, Yu Qi, Zhang Ye.

Figure 1
Figure 1. Figure 1: Comparison of different policy-learning frameworks. Regression methods can easily collapse to [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Architecture of Action Map Policy. The left branches take one in-hand image and two side-view images as input. The center features a multi-view transformer that enables communication between the in-hand features and the side-view context features. The right branch consists of two decoders that generate heatmaps of keypoints across the temporal horizon [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Keypoint-based Action Representation. relate 3D points to 2D coordinates via the camera matrices. Under generic camera configurations, (P, T ) establish a bijection between R 3 and Im(P) ⊂ (R 2 ) n. We then discretize the projected coordinates into pixel locations and use them to create soft heatmap labels. For keypoint j at timestep i in camera view k, the label hijk ∈ R H×W is a normalized distribution o… view at source ↗
Figure 4
Figure 4. Figure 4: Reconstruction error (left) and conven￾tional codebook size (right) vs. image resolution. The codebook grows by six orders of magnitude as resolution increases. Resolution Trans. (mm) Rot. (◦ ) # Tokens 96 × 96 2.33 ± 0.21 3.03 ± 0.14 ( 10 2.33 ) 3 ( 360 3.03 ) 3 128 × 128 1.75 ± 0.16 2.28 ± 0.11 ( 10 1.75 ) 3 ( 360 2.28 ) 3 224 × 224 1.00 ± 0.09 1.30 ± 0.06 ( 10 1.00 ) 3 ( 360 1.30 ) 3 512 × 512 0.44 ± 0.… view at source ↗
Figure 5
Figure 5. Figure 5: Representative tasks from MimicGen. From left to right: stack-three-block, hammer-cleanup-d1, [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Real-world system setup, ground-truth keypoint projection visualization, and action map prediction. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Real-world tasks: make-coffee, toast-bread, and steam-egg. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Real-world spatial reasoning tasks: stack [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Simulation camera setup for the six MimicGen tasks. The first row shows side-view camera 1, the [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Heatmap prediction visualization on the MAKE-COFFEE task, evaluated at the midpoint of a downward grasp toward the coffee pod. Each row corresponds to one calibrated side view. The columns show the dense prediction process, including the raw decoder output, cross-entropy supervision, spatial softmax, argmax pixel selection, and the final horizon-wide trajectory. Columns (1)–(2) summarize the full 5×12 key… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 21 linked inside Pith

  1. [1]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    G. Team, P. Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi`ere, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    Goyal, H

    A. Goyal, H. Hadfield, X. Yang, V . Blukis, and F. Ramos. Vla-0: Building state-of-the-art vlas with zero modification.arXiv preprint arXiv:2510.13054, 2025

    arXiv 2025

  5. [5]

    C. Liu, X. Han, J. Gao, Y . Zhao, H. Chen, and Y . Du. Oat: Ordered action tokenization.arXiv preprint arXiv:2602.04215, 2026

    arXiv 2026

  6. [6]

    Zitkovich, T

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. InConference on Robot Learning, pages 2165–2183. PMLR, 2023

    2023

  7. [7]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware.arXiv preprint arXiv:2304.13705, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Haldar, Z

    S. Haldar, Z. Peng, and L. Pinto. Baku: An efficient transformer for multi-task policy learning. Advances in Neural Information Processing Systems, 37:141208–141239, 2024

    2024

  10. [10]

    Mandlekar, D

    A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y . Zhu, and R. Mart´ın-Mart´ın. What matters in learning from offline human demonstrations for robot manipulation.arXiv preprint arXiv:2108.03298, 2021

    Pith/arXiv arXiv 2021

  11. [11]

    X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177, 2019

    Pith/arXiv arXiv 1910

  12. [12]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  13. [13]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025

    2025

  14. [14]

    D. Wang, S. Hart, D. Surovik, T. Kelestemur, H. Huang, H. Zhao, M. Yeatman, J. Wang, R. Walters, and R. Platt. Equivariant diffusion policy.arXiv preprint arXiv:2407.01812, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    B. Hu, D. Wang, D. Klee, H. Tian, X. Zhu, H. Huang, R. Platt, and R. Walters. 3d equivariant visuomotor policy learning via spherical projection.Advances in Neural Information Processing Systems, 38:5685–5715, 2026. 9

    2026

  16. [16]

    K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. InAdvances in Neural Information Processing Systems 28, pages 3483–3491

  17. [17]

    A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V . Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. InConference on Robot Learning, pages 726–747. PMLR, 2021

    2021

  18. [18]

    Huang, D

    H. Huang, D. Wang, R. Walters, and R. Platt. Equivariant transporter network.arXiv preprint arXiv:2202.09400, 2022

    Pith/arXiv arXiv 2022

  19. [19]

    M. Jia, H. Huang, Z. Zhang, C. Wang, L. Zhao, D. Wang, J. X. Liu, R. Walters, R. Platt, and S. Tellex. Learning efficient and robust language-conditioned manipulation using textual-visual relevancy and equivariant language mapping.IEEE Robotics and Automation Letters, 2025

    2025

  20. [20]

    Shridhar, L

    M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. InConference on Robot Learning, pages 785–799. PMLR, 2023

    2023

  21. [21]

    Huang, O

    H. Huang, O. Howell, D. Wang, X. Zhu, R. Platt, and R. Walters. Fourier transporter: Bi- equivariant robotic manipulation in 3d. InInternational Conference on Learning Representa- tions, volume 2024, pages 40733–40754, 2024

    2024

  22. [22]

    Goyal, J

    A. Goyal, J. Xu, Y . Guo, V . Blukis, Y .-W. Chao, and D. Fox. Rvt: Robotic view transformer for 3d object manipulation. InConference on Robot Learning, pages 694–710. PMLR, 2023

    2023

  23. [23]

    C. Pan, B. Okorn, H. Zhang, B. Eisner, and D. Held. Tax-pose: Task-specific cross-pose estimation for robot manipulation. InConference on Robot Learning, pages 1783–1792. PMLR, 2023

    2023

  24. [24]

    Eisner, Y

    B. Eisner, Y . Yang, T. Davchev, M. Vecerik, J. Scholz, and D. Held. Deep SE(3)-equivariant geometric reasoning for precise placement tasks. InInternational Conference on Learning Representations (ICLR), 2024. URLhttps://openreview.net/forum?id=2inBuwTyL2

    2024

  25. [25]

    Huang, K

    H. Huang, K. Schmeckpeper, D. Wang, O. Biza, Y . Qian, H. Liu, M. Jia, R. Platt, and R. Walters. Imagination policy: Using generative point cloud models for learning manipulation policies. arXiv preprint arXiv:2406.11740, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Huang, H

    H. Huang, H. Liu, D. Wang, R. Walters, and R. Platt. Match policy: A simple pipeline from point cloud registration to manipulation policies. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 16907–16914. IEEE, 2025

    2025

  27. [27]

    Pearce, T

    T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V . Macua, S. Z. Tan, I. Momennejad, K. Hofmann, et al. Imitating human behaviour with diffusion models.arXiv preprint arXiv:2301.10677, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    C. Pan, G. Anantharaman, N.-C. Huang, C. Jin, D. Pfrommer, C. Yuan, F. Permenter, G. Qu, N. Boffi, G. Shi, et al. Much ado about noising: Dispelling the myths of generative robotic control.arXiv preprint arXiv:2512.01809, 2025

    arXiv 2025

  29. [29]

    X. Hu, B. Liu, X. Liu, and Q. Liu. Adaflow: Imitation learning with variance-adaptive flow- based policies.Advances in Neural Information Processing Systems, 37:138836–138858, 2024

    2024

  30. [30]

    Lipman, R

    Y . Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022

    Pith/arXiv arXiv 2022

  31. [31]

    G. Yan, J. Zhu, Y . Deng, S. Yang, R.-Z. Qiu, X. Cheng, M. Memmel, R. Krishna, A. Goyal, X. Wang, et al. Maniflow: A general robot manipulation policy via consistency flow training. arXiv preprint arXiv:2509.01819, 2025. 10

    Pith/arXiv arXiv 2025

  32. [32]

    Haldar, L

    S. Haldar, L. Johannsmeier, L. Pinto, A. Gupta, D. Fox, Y . Narang, and A. Mandlekar. Point bridge: 3d representations for cross domain policy learning.arXiv preprint arXiv:2601.16212, 2026

    arXiv 2026

  33. [33]

    Huang, D

    H. Huang, D. Wang, X. Zhu, R. Walters, and R. Platt. Edge grasp network: A graph-based se (3)-invariant approach to grasp detection.arXiv preprint arXiv:2211.00191, 2022

    Pith/arXiv arXiv 2022

  34. [34]

    B. Hu, X. Zhu, D. Wang, Z. Dong, H. Huang, C. Wang, R. Walters, and R. Platt. Orbitgrasp: se(3)-equivariant grasp learning.arXiv preprint arXiv:2407.03531, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Huang, C

    W. Huang, C. Wang, Y . Li, R. Zhang, and L. Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation.arXiv preprint arXiv:2409.01652, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Manuelli, W

    L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category- level robotic manipulation. InThe International Symposium of Robotics Research, pages 132–148. Springer, 2019

    2019

  37. [37]

    W. Yuan, J. Duan, V . Blukis, W. Pumacay, R. Krishna, A. Murali, A. Mousavian, and D. Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics.arXiv preprint arXiv:2406.10721, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Cheng, J

    L. Cheng, J. Duan, Y .-R. Wang, H.-S. Fang, B. Li, Y . Huang, and L. Wang. Pointarena: Probing multimodal grounding through language-guided pointing.arXiv preprint arXiv:2501.12345, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    J. Ren, P. Sundaresan, D. Sadigh, S. Choudhury, and J. Bohg. Motion tracks: A unified repre- sentation for human-robot transfer in few-shot imitation learning. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 8802–8810. IEEE, 2025

    2025

  40. [40]

    M. Xu, Z. Xu, Y . Xu, C. Chi, G. Wetzstein, M. Veloso, and S. Song. Flow as the cross-domain manipulation interface.arXiv preprint arXiv:2407.15208, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Ronneberger, P

    O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. InMedical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015

    2015

  42. [42]

    Hamdi, S

    A. Hamdi, S. Giancola, and B. Ghanem. Mvtn: Multi-view transformation network for 3d shape recognition. InProceedings of the IEEE/CVF international conference on computer vision, pages 1–11, 2021

    2021

  43. [43]

    Mandlekar, S

    A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y . Narang, L. Fan, Y . Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310.17596, 2023

    Pith/arXiv arXiv 2023

  44. [44]

    P. Wu, Y . Shentu, Z. Yi, X. Lin, and P. Abbeel. Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators. In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 12156–12163. IEEE, 2024. 11 6 Appendix 6.1 Simulation Details Figure 10: Simulation camera setup for the six MimicGen tasks. The ...

    2024