Pith. sign in

REVIEW 4 major objections 4 minor 50 references

Sparse Meets Dense: Correspondence Guided Robotic Manipulation with Rigid-Deformable Interactions

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a hybrid sparse-plus-dense correspondence representation enables one-shot transfer of rigid-deformable manipulation skills to new shapes, outperforming purely sparse or purely dense baselines.

desk verdict The hybrid sparse-dense representation is sensible and the ablations are consistent, but the paper's generalization to new shapes relies on a rigid dense correspondence module that is never defined, trained, or ablated, which makes the central claim under-supported. read the letter →

arxiv 2608.01083 v1 pith:JXBJRJI4 submitted 2026-08-02 cs.RO

Ziyu Zhu , Yue Chen , Xirui Liang , Hojin Bae , Yuran Wang , Zhen Yuan , Ruihai Wu , Hao Dong This is my paper · ORCID
classification cs.RO
keywords rigid-deformablemanipulationsparsekeypointcorrespondencedenseone-shottransferconstraintoptimizationdeformableobjecttrackingrobot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Household chores such as hanging a shirt on a hanger or putting a book into a backpack require a robot to coordinate a rigid object with a soft, deformable one. This paper argues that neither sparse keypoints nor dense correspondences alone capture these interactions, and proposes a hybrid representation: sparse, task-aware keypoint pairs define the geometric constraints that guide motion, while dense correspondences track those keypoints on the deforming body and transfer them to novel shapes. From a single demonstration, the method extracts the keypoint pairs once and then re-anchors them on new objects through dense feature matching. In simulation the hybrid method reports 73.5% average success across four tasks, compared with 58.5% for sparse-only and 54.9% for dense-only baselines; in real-world trials it reports 65.0% versus 47.5% and 50.0%. The claim is that this combination is what makes one-shot transfer to new shapes and deformations possible.

What carries the argument

The load-bearing object is the sparse keypoint correspondence set $C_{\mathrm{sparse}}$, formed by mutually nearest structural keypoints on rigid and deformable point clouds filtered by a distance threshold, combined with a dense correspondence module over the deformable surface. Dense point features, initialized by distilling a planar garment-correspondence model into a 3D PointNet++ and refined with a contrastive loss on annotated 3D pairs, anchor each sparse keypoint and track it across deformation; a closed-loop optimizer minimizes the positional loss $c_{\mathrm{pos}}$ (mean Euclidean distance between paired keypoints) and the directional loss $c_{\mathrm{ori}}$ (angle between normals of planes fitted to the paired keypoints). This pairing is what carries the argument: sparse keypoints make the task constraint explicit and interpretable, while dense correspondences provide the tracking and cross-shape generalization that sparse-only and dense-only methods each lack.

What would settle it

Run one of the four tasks with the final-state relative pose annotation rotated or translated by a small controlled amount (e.g., 5 cm or 10 degrees). If success rates collapse, the reported generalization is inherited largely from the manual annotation rather than from the learned dense correspondences; if success stays flat, the hybrid mapping is doing the work as claimed.

Watch

Extended reading notes

Core claim

The central claim is that rigid-deformable manipulation can be planned and executed through a hybrid correspondence representation. Sparse keypoint correspondences, built by matching structural keypoints on the rigid and deformable objects and filtering them by physical proximity at the demonstrated final state, supply the task-relevant alignment information. Dense correspondences on the deformable body, distilled from a garment-correspondence model and fine-tuned with a small set of 3D annotations, keep those keypoints trackable under occlusion and deformation, and map them onto novel shapes. Constraint optimization then moves the end effector to minimize positional and directional discrepancies of the keypoint pairs at each time step. The paper concludes that taking the best of both representations yields robust, generalizable manipulation across new shapes and deformations, and supports this with simulation and real-world comparisons against sparse-only and dense-only alternatives.

Load-bearing premise

The assumption that carries the method is that a single manually annotated relative pose between the rigid and deformable objects at the demonstration's final state correctly captures the task's required alignment, and that dense feature matching can transfer that alignment to novel shapes without fresh task-specific annotation.

Editorial extensions

If this is right

  • One demonstration is enough to set up a new rigid-deformable task: the same sparse keypoint pairs, once extracted, can be re-anchored on novel object shapes through dense feature matching.
  • Keypoints on deformable bodies remain trackable during long-horizon interactions, because dense correspondences update the constraints even under occlusion and changing deformation of the soft body.
  • The positional constraint carries most of the task success; the paper's ablations show removing it drops success to 0% on the tested tasks, while removing the orientational constraint is less damaging.
  • The hybrid representation transfers from simulation to a real dual-arm robot with a dexterous hand, reaching 65% average success across four tasks.
  • VLMs handle only high-level stage decomposition and keypoint selection; low-level constraints come from physical correspondences, which the paper finds more reliable than constraints generated directly by the VLM.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the hybrid representation is as general as claimed, the same sparse-plus-dense pattern could be pressed into deformable-deformable tasks, where the paper's own limitation note says representation and modeling remain open.
  • The manual final-state relative pose annotation is the single human input; a learned estimator for that pose would make the pipeline fully autonomous and is a natural next step implied by the method.
  • The framework's success depends on the dense features remaining discriminative across object topology; a testable extension is to measure how success degrades as novel shapes move farther from the training garment distribution in skeleton structure.
  • Because the error analysis attributes most failures to rigid-object tracking under occlusion, pairing the dense correspondence module with an occlusion-robust rigid tracker should produce the largest near-term gain.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a hybrid correspondence-based representation for robotic manipulation with rigid-deformable interactions. Sparse keypoints are extracted from structural candidates and filtered using a manually annotated final-state relative pose, producing task-relevant keypoint pairs that are used as positional and directional constraints in a closed-loop optimization. Dense correspondences, built by distilling UniGarmentManip features into a 3D PointNet++ model, are used to track sparse keypoints on deformable objects and to transfer them to novel shapes. Experiments are reported for four tasks (hanger-into-shirt, racket-into-bag, book-into-backpack, scoop-peanuts-into-bag) in the GarmentLab simulation environment and on real robot platforms, with comparisons against ReKep and UniGarment baselines. The paper claims that the hybrid representation yields higher success than sparse-only and dense-only baselines in both simulation (73.5% average vs. 58.5% and 54.9%) and the real world (65.0% vs. 47.5% and 50.0%), and that it generalizes to novel object shapes through dense-correspondence keypoint mapping.

Significance. If the claims hold, the hybrid sparse-plus-dense representation is a practically useful idea for contact-rich rigid-deformable manipulation, a setting where pure 6D poses or pure dense features are known to be insufficient. The paper includes real-robot experiments, ablations of the main constraints and tracking modules, and an interpretable error breakdown, which are strengths. The four tasks are relevant and the qualitative generalization example in Fig. 5 is suggestive. However, the central generalization claim currently rests on a component that is not described, trained, or evaluated, and the quantitative evaluation lacks trial counts and error bars. For these reasons the paper is not yet acceptable in its present form.

major comments (4)
  1. [IV-D, Eq. (11)] Section IV-D, Eq. (11) requires a rigid dense correspondence C^r_dense to map demonstration sparse keypoints onto novel rigid objects, but no such module is ever defined, trained, or ablated. Section IV-C defines dense correspondence only for deformable garments ('point-wise matching between two garments'), and the only rigid-object module described is the within-instance 3D tracker CoTracker, which cannot perform cross-instance correspondence between a demonstration hanger and a novel hanger of different shape. No architecture, training set, loss function, or retrieval evaluation is given for C^r_dense. Since Table III is the paper's evidence for generalization to new shapes, this missing component is load-bearing: the reported generalization numbers cannot be attributed to the proposed hybrid representation, and the rigid mapping may be done by an unstated feature space. The authors should specify and evaluate this component, or remove the generalization claim.
  2. [IV-B.2, Eq. (1)] The sparse correspondences are built from a manually annotated relative pose between the rigid and deformable objects in the final demonstration state, yet the paper does not state how this annotation is obtained, whether it is re-annotated per object pair, or how sensitive downstream success is to inaccuracies in this pose. Because the keypoint pairs in Eq. (1) are the sole carrier of task-specific interaction information, the 'one-shot transfer' claim requires either an explicit justification that the same annotation is task-representative across all evaluated objects or a sensitivity/noise ablation. As written, the method is not fully autonomous and the contribution of the manual annotation relative to the learned components is unclear.
  3. [V-C, Tables I-III] No trial counts, standard deviations, or confidence intervals are reported for any simulation result. Percentages such as 71.4% versus 65.2% in Table I or the differences in Table III cannot be assessed for statistical significance, and the real-world evaluation has only 10 trials per task. The paper should report the number of runs for every cell and ideally repeated-seed or repeated-configuration variances. This is particularly important because some claimed margins over the dense baseline are modest and could be within run-to-run noise.
  4. [V-E, Table III] The generalization evaluation is limited to three simulation tasks, with no 'Scoop peanuts into bag' generalization, no real-world generalization, and no quantitative evaluation of the cross-instance keypoint mapping itself. Without measuring the retrieval accuracy of the dense mapping or the validity of the mapped constraints on novel objects, the generalization claim is supported only by the qualitative illustration in Fig. 5. The authors should report retrieval success (e.g., fraction of mapped keypoints that land on the intended structure) or a related quantitative metric.
minor comments (4)
  1. [Throughout] There are several typos and inconsistencies: 'Unigarmentmanip' is used in IV-B.1 while 'UniGarmentManip' is used elsewhere; 'network' appears as 'newwork' in IV-C.2; 'colum' should be 'column' in V-F; the baseline name is written both as 'Annot. Rekep' and 'ReKep'.
  2. [Fig. 4 and Table II] The text refers to 'the second row of Fig. 4', but the caption describes top and bottom rows, and it is unclear which row corresponds to which ablation. Also, the rows 'Ours w/o C_dense' and 'Ours w/3D-Tracker' in Table II report identical values (56.4% and 63.2%), which is suspicious and should be explained or corrected.
  3. [Appendix C.2] The UniGarment baseline is described as being extended to compute correspondences between rigid and deformable objects 'based on their relative positions in the final configuration'. If this baseline also uses a manually annotated relative pose, the comparison should state that both methods receive the same annotation; otherwise the baseline may be unfairly disadvantaged.
  4. [V-F / Appendix A] The paper claims real-time inference and execution but does not provide runtime measurements for the closed-loop optimization, dense feature extraction, or tracker modules. A short table or paragraph with per-stage timing would make the real-time claim verifiable.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reduction found: task-level success criteria are independent of the optimized keypoint loss, and the self-citations are components/baselines rather than load-bearing conclusions.

full rationale

The derivation chain is not circular. Sparse correspondences are constructed from a manually annotated final-state relative pose and mutual-nearest-neighbor selection (Eq. 1); the optimization loss (Eqs. 3 and 5) minimizes those keypoint displacements, but success is evaluated by independent task-level criteria in Appendix C.1 (hanger fully inserted, racket fully inside bag, book placed with correct spatial alignment, all peanuts transferred), not by the loss itself, so the reported outcome is not defined as the input. The dense correspondence module is a learned component: it distills pretrained UniGarmentManip features and is fine-tuned with ground-truth 3D correspondences (Eq. 10 and Appendix B), so generalization through Eq. (11) is a learned mapping rather than a fitted rename of the training targets. The self-citations (UniGarmentManip [42], GarmentLab [21]) are used as a component and simulation environment respectively, and the dense-only baseline UniGarment is directly compared rather than assumed, so no load-bearing conclusion reduces to a self-citation. One non-circular transparency gap should be flagged: C^r_dense in Eq. (11) is never defined, trained, or ablated, and no rigid dense correspondence module is specified in Section IV or Appendix B; this is a reproducibility and correctness risk for the generalization claims, but it is not a circular reduction because no equation ties C^r_dense back to the measured success counts or to a fitted parameter. Therefore the circularity score is 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central method depends on several human inputs and prior models: manually annotated relative pose, an unreported threshold tau, task-specific keypoint counts, and a pretrained dense correspondence model from the same group. These are not derived in the paper, so they are the real cost of the contribution.

free parameters (6)
  • manually annotated relative pose = annotated per demonstration, not published
    Used in Section IV-B.2 to transform point clouds into the same space and define sparse correspondences; accuracy and human effort are not analyzed.
  • distance threshold tau = not reported
    Eq. 1 keeps keypoint pairs closer than tau; no value or sensitivity analysis is given.
  • sparse keypoint count N_s = 3 to 6 per stage
    Eq. 2 samples N_s keypoint pairs via FPS; the choice per task is not justified.
  • candidate keypoint counts N_r and N_d = 20-100 rigid, 40-200 deformable
    Section IV-B.1 gives ranges but no exact choices or sensitivity analysis.
  • temperature tau in contrastive losses = not reported
    Appendix B losses use temperature tau; it is a standard hyperparameter but unspecified.
  • success spatial threshold = not quantified
    Problem formulation references a desired spatial threshold but no numeric value is given.
assumptions (4)
  • domain assumption The interaction between rigid and deformable objects can be captured by 3 to 6 sparse keypoint correspondences plus a plane normal constraint.
    Used throughout Section IV-C; if structural alignment at sparse points is insufficient, the constraints miss important contacts.
  • domain assumption Dense correspondences pretrained on garments in near-planar states transfer to arbitrary 3D poses after projection, distillation, and fine-tuning.
    Section IV-C.2 and Appendix B; no evidence that orthographic XY projection preserves useful information for occluded or complex 3D poses.
  • domain assumption The manually annotated relative pose in the final demonstration state defines task-relevant correspondences.
    Section IV-B.2; the keypoint pairs are generated from this annotation and transferred to new instances.
  • domain assumption VLM stage decomposition and keypoint selection are reliable enough for the manipulation stages.
    Section IV-A and Appendix D; the error breakdown in Fig. 7 shows VLM errors are the second largest failure source.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sparse Meets Dense: Correspondence Guided Robotic Manipulation with Rigid-Deformable Interactions." pith.science (2026). https://pith.science/paper/JXBJRJI4

@misc{pith2026260801083,
  author       = {Pith},
  title        = {Pith review of: Sparse Meets Dense: Correspondence Guided Robotic Manipulation with Rigid-Deformable Interactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JXBJRJI4}},
  note         = {Machine review of arXiv:2608.01083}
}
read the original abstract

Manipulation involving rigid-deformable interactions, such as hanging clothes or dressing humans, is common in daily life, making it essential for household robots. Compared to single-object manipulation or interactions between rigid bodies, these tasks are particularly challenging due to the rich multi-point contacts and the complex dynamics of the deformable bodies during interaction. Therefore, object-centric representations such as 6D poses or structural points without task-specific information become insufficient for these interactions. In this work, we propose a hybrid correspondence-based representation tailored for rigid-deformable interactions. First, to capture intricate interaction information, we introduce structure-, task-, and interaction-aware sparse keypoints. The keypoints are generated based on the global structures of both rigid and deformable objects, and filtered by their local interaction contacts. However, tracking these sparse keypoints through the interaction remains difficult due to the high-dimensional dynamics of deformable objects. Therefore, we further construct dense correspondences on the deformable objects for accurate keypoint tracking throughout the manipulation. This hybrid design combines the advantages of both representations: sparse keypoints encode rich, task-specific information for fine-grained manipulation, while dense correspondences ensure efficient tracking and generalization to novel deformations, shapes, and scenarios. Together, they enable one-shot transfer to new tasks with minimal demonstrations. Extensive experiments demonstrate the effectiveness and broad applicability of our method.

Figures

Figures reproduced from arXiv: 2608.01083 by the authors.

Figure 1
Figure 1. Correspondences for Rigid-Deformable Interactions. We propose a hybrid correspondence-based representation for rigid-deformable interaction tasks. (Left) Sparse keypoint correspondences capture the interaction information between objects. (Top-middle) For long-horizon manipulation with a sequence of actions, dense correspondences can track keypoint accurately on soft bodies across deformations, and enable one-shot t… view at source ↗
Figure 2
Figure 2. Framework Overview. Given a task demonstration, VLMs decompose it into stages. For the stages with rigid￾deformable interactions, we first extract sparse keypoints as spatial constraints, then apply constraint optimization to guide the manipulation sequence, with pretrained dense correspondence to track keypoints on deformable objects. IV. METHOD Our framework consists of four components: stage decom￾position via VL… view at source ↗
Figure 3
Figure 3. Task Illustrations. Representative tasks in simulation and the real world. D. Generalization to Novel Objects To further generalize this pipeline to novel objects, it is crucial to accurately transfer the Csparse from the demon￾stration instances to new object pairs. Therefore, we use Cdense to retrieve the dense feature embedding and perform a nearest-neighbor search in the dense feature space of the novel object t… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Ablation Studies. The top and bottom rows respectively show results of replacing Csparse and Cdense with DINOv2 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Dense Correspondences Enable Generalization. Given the demonstration of table tennis racket (row 1), we can generalize to other rackets by mapping keypoints via Cdense (row 2,3). Task ReKep UniGarment Ours Insert hanger into shirt 15.6% 57.1% 65.2% Put racket into bag …
Figure 6
Figure 6. Figure 6: Real-world Environment and Assets Utilized. Insert hanger into shirt Put racket into bag Put books into backpack VLM 25.6% 29.1% 30.2% Ours 71.4% 73.6% 78.2% TABLE IV: Generalization Results. Section V-C provides detailed analysis. We then identify a set of hard negati…
Figure 7
Figure 7. Figure 7: Error breakdown of the system modules. 3) Comparison with Vision-Language Model Generated Constraints: Many existing works [7]–[9] utilize Vision￾Language Models (VLMs) to generate constraints for motion planning. However, in tasks involving rigid-deformable inter￾acti…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 42 canonical work pages

  1. [1]

    A constraint programming approach to simultaneous task allocation and motion scheduling for industrial dual-arm manipulation tasks

    Jan Kristof Behrens, Ralph Lange, and Masoumeh Mansouri. A constraint programming approach to simultaneous task allocation and motion scheduling for industrial dual-arm manipulation tasks. In2019 International Conference on Robotics and Automation (ICRA), pages 8705–8711, 2019. ii

    work page 2019

  2. [2]

    Benchmarking the sim-to-real gap in cloth manipulation.IEEE Robotics and Automation Letters, 9(3):2981–2988, 2024

    David Blanco-Mulero, Oriol Barbany, Gokhan Alcan, Adri `a Colom ´e, Carme Torras, and Ville Kyrki. Benchmarking the sim-to-real gap in cloth manipulation.IEEE Robotics and Automation Letters, 9(3):2981–2988, 2024. i

    work page 2024

  3. [3]

    Keystate: Im- proving image-based reinforcement learning with keypoint for robot control

    Ling-Chen Chen, Chi-Kai Ho, and Chung-Ta King. Keystate: Im- proving image-based reinforcement learning with keypoint for robot control. In2023 IEEE International Conference on Industrial Tech- nology (ICIT), pages 1–6, 2023. ii

    work page 2023

  4. [4]

    Florence, Lucas Manuelli, and Russ Tedrake

    Peter R. Florence, Lucas Manuelli, and Russ Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation, 2018. ii, v

    work page 2018

  5. [5]

    Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tom ´as Lozano-P ´erez

    Caelan R. Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tom ´as Lozano-P ´erez. Inte- grated task and motion planning.Annual Review of Control, Robotics, and Autonomous Systems, 4:265–293, 2021. ii

    work page 2021

  6. [6]

    Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding

    Huy Ha and Shuran Song. Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding. InConference on Robot Learning, pages 24–33. PMLR, 2022. ii

    work page 2022

  7. [7]

    Copa: General robotic manipulation through spatial constraints of parts with foundation models.arXiv preprint arXiv:2403.08248,

    Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models.arXiv preprint arXiv:2403.08248,

    arXiv

  8. [8]

    Rekep: Spatio-temporal reasoning of relational keypoint con- straints for robotic manipulation.arXiv preprint arXiv:2409.01652,

    Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei- Fei. Rekep: Spatio-temporal reasoning of relational keypoint con- straints for robotic manipulation.arXiv preprint arXiv:2409.01652,

    arXiv

Show all 50 references
  1. [9]

    V oxposer: Composable 3d value maps for robotic manipulation with language models.arXiv preprint arXiv:2307.05973,

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models.arXiv preprint arXiv:2307.05973,

    arXiv

  2. [10]

    Hierarchical task and motion planning in the now

    Leslie Pack Kaelbling and Tom ´as Lozano-P´erez. Hierarchical task and motion planning in the now. In2011 IEEE International Conference on Robotics and Automation, pages 1470–1477. IEEE, 2011. ii

    2011

  3. [12]

    Hierarchical planning in the now

    Leslie Pack Kaelbling and Tom ´as Lozano-P´erez. Hierarchical planning in the now. InProceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 1460–1467, 2010. ii

    2010

  4. [13]

    Integrated task and motion planning in belief space.The International Journal of Robotics Research, 32(9-10):1194–1227, 2013

    Leslie Pack Kaelbling and Tom ´as Lozano-P ´erez. Integrated task and motion planning in belief space.The International Journal of Robotics Research, 32(9-10):1194–1227, 2013. ii

    2013

  5. [14]

    Cotracker: It is better to track together.arXiv preprint arXiv:2307.07635, 2023

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together.arXiv preprint arXiv:2307.07635, 2023. v

    2023 arXiv

  6. [15]

    Efficiently combining task and motion planning using geometric constraints.The International Journal of Robotics Research, 33(14):1726–1747, 2014

    Fabien Lagriffoul, Dimitar Dimitrov, Julien Bidot, Alessandro Saffiotti, and Lars Karlsson. Efficiently combining task and motion planning using geometric constraints.The International Journal of Robotics Research, 33(14):1726–1747, 2014. ii

    2014

  7. [16]

    Constraint propagation on interval bounds for dealing with geometric backtracking

    Fabien Lagriffoul, Dimitar Dimitrov, Alessandro Saffiotti, and Lars Karlsson. Constraint propagation on interval bounds for dealing with geometric backtracking. In2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 957–964. IEEE, 2012. ii

    2012

  8. [17]

    Optimizing keypoint-based single-shot camera-to-robot pose estimation through shape segmentation

    Jens Lambrecht, Philipp Grosenick, and Marvin Meusel. Optimizing keypoint-based single-shot camera-to-robot pose estimation through shape segmentation. In2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13843–13849, 2021. ii

    2021

  9. [18]

    Broadcasting support relations recursively from local dynamics for object retrieval in clutters

    Yitong Li, Ruihai Wu, Haoran Lu, Chuanruo Ning, Yan Shen, Guanqi Zhan, and Hao Dong. Broadcasting support relations recursively from local dynamics for object retrieval in clutters. InRobotics: Science and Systems, 2024. i

    2024

  10. [19]

    Moka: Open-vocabulary robotic manipulation through mark-based visual prompting

    Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. InFirst Workshop on Vision-Language Models for Navi- gation and Manipulation at ICRA 2024, 2024. ii

    2024

  11. [20]

    A constraint-based method for solving sequential manipulation planning problems

    Tom ´as Lozano-P ´erez and Leslie Pack Kaelbling. A constraint-based method for solving sequential manipulation planning problems. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3684–3691. IEEE, 2014. ii

    2014

  12. [21]

    Garmentlab: A unified simulation and benchmark for garment manipulation, 2024

    Haoran Lu, Ruihai Wu, Yitong Li, Sijie Li, Ziyu Zhu, Chuanruo Ning, Yan Shen, Longzan Luo, Yuanpei Chen, and Hao Dong. Garmentlab: A unified simulation and benchmark for garment manipulation, 2024. v, x

    2024

  13. [22]

    Robot visual servoing grasping based on top-down keypoint detection network

    Junqi Luo, Liucun Zhu, Liang Li, and Peitao Hong. Robot visual servoing grasping based on top-down keypoint detection network. IEEE Transactions on Instrumentation and Measurement, 73:1–11,

  14. [23]

    kpam: Keypoint affordances for category-level robotic manipulation

    Lucas Manuelli, Wei Gao, Peter Florence, and Russ Tedrake. kpam: Keypoint affordances for category-level robotic manipulation. In The International Symposium of Robotics Research, pages 132–157. Springer, 2019. i, ii

    2019

  15. [24]

    Contact- invariant optimization for hand manipulation

    Igor Mordatch, Zoran Popovi ´c, and Emanuel Todorov. Contact- invariant optimization for hand manipulation. InProceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pages 137–144. Eurographics Association, 2012. ii

    2012

  16. [25]

    Discovery of complex behaviors through contact-invariant optimization.ACM Transactions on Graphics (TOG), 31(4):1–8, 2012

    Igor Mordatch, Emanuel Todorov, and Zoran Popovi ´c. Discovery of complex behaviors through contact-invariant optimization.ACM Transactions on Graphics (TOG), 31(4):1–8, 2012. ii

    2012

  17. [26]

    Pivot: Iterative visual prompting elicits actionable knowledge for vlms

    Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07872, 2024. ii

    2024 arXiv

  18. [27]

    Isaac Sim

    NVIDIA. Isaac Sim. v

  19. [28]

    Omnimanip: Towards general robotic manipulation via object-centric interaction primitives as spatial constraints.arXiv preprint arXiv:2307.04767, 2024

    Mingjie Pan, Jiyao Zhang, Tianshu Wu, Yinghao Zhao, Wenlong Gao, and Hao Dong. Omnimanip: Towards general robotic manipulation via object-centric interaction primitives as spatial constraints.arXiv preprint arXiv:2307.04767, 2024. i

    2024 arXiv

  20. [29]

    A direct method for trajectory optimization of rigid bodies through contact.The International Journal of Robotics Research, 33(1):69–81, 2014

    Michael Posa, Cecilia Cantu, and Russ Tedrake. A direct method for trajectory optimization of rigid bodies through contact.The International Journal of Robotics Research, 33(1):69–81, 2014. ii

    2014

  21. [30]

    Optimization and stabilization of trajectories for constrained dynamical systems

    Michael Posa, Scott Kuindersma, and Russ Tedrake. Optimization and stabilization of trajectories for constrained dynamical systems. In2016 IEEE International Conference on Robotics and Automation (ICRA), pages 1366–1373. IEEE, 2016. ii

    2016

  22. [31]

    Andrew Bagnell, and Siddhartha Srinivasa

    Nathan Ratliff, Matt Zucker, J. Andrew Bagnell, and Siddhartha Srinivasa. Chomp: Gradient optimization techniques for efficient motion planning. In2009 IEEE International Conference on Robotics and Automation, pages 489–494, 2009. ii

    2009

  23. [32]

    Self-supervised visual descriptor learning for dense correspondence.IEEE Robotics and Automation Letters, 2(2):420–427, 2016

    Tanner Schmidt, Richard Newcombe, and Dieter Fox. Self-supervised visual descriptor learning for dense correspondence.IEEE Robotics and Automation Letters, 2(2):420–427, 2016. ii

    2016

  24. [33]

    Lee, Ibrahim Awwal, Henry Bradlow, Jia Pan, Sachin Patil, Ken Goldberg, and Pieter Abbeel

    John Schulman, Yan Duan, Jonathan Ho, Alex X. Lee, Ibrahim Awwal, Henry Bradlow, Jia Pan, Sachin Patil, Ken Goldberg, and Pieter Abbeel. Motion planning with sequential convex optimization and convex collision checking.Int. J. Robotics Res., 33(9):1251–1270,

  25. [34]

    Initial results on grasping and lifting physical deformable bags with a bimanual robot

    Daniel Seita, Justin Kerr, John Canny, and Ken Goldberg. Initial results on grasping and lifting physical deformable bags with a bimanual robot. InIROS Workshop on Robotic Manipulation of Deformable Objects in Real-world Applications, volume 2, page 3, 2021. i

    2021

  26. [35]

    Skeleton merger: an unsupervised aligned keypoint detector

    Ruoxi Shi, Zhengrong Xue, Yang You, and Cewu Lu. Skeleton merger: an unsupervised aligned keypoint detector. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 43–52, 2021. ix

    2021

  27. [36]

    Neural descriptor fields: Se (3)-equivariant object representations for manipulation

    Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenen- baum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In2022 International Conference on Robotics and Automation (ICR...

    2022

  28. [37]

    curobo: Parallelized collision-free minimum-jerk robot motion generation, 2023

    Balakumar Sundaralingam, Siva Kumar Sastry Hari, Adam Fishman, Caelan Garrett, Karl Van Wyk, Valts Blukis, Alexander Millane, Helen Oleynikova, Ankur Handa, Fabio Ramos, Nathan Ratliff, and Dieter Fox. curobo: Parallelized collision-free minimum-jerk robot motion generation, 2023. ii

    2023

  29. [38]

    Robokeygen: Robot pose and joint angles estimation via diffusion-based 3d keypoint generation, 2024

    Yang Tian, Jiyao Zhang, Guowei Huang, Bin Wang, Ping Wang, Jiangmiao Pang, and Hao Dong. Robokeygen: Robot pose and joint angles estimation via diffusion-based 3d keypoint generation, 2024. ii

    2024

  30. [39]

    You only demonstrate once: Category-level manipulation from single visual demonstration.arXiv preprint arXiv:2201.12716, 2022

    Bowen Wen, Wenzhao Lian, Kostas Bekris, and Stefan Schaal. You only demonstrate once: Category-level manipulation from single visual demonstration.arXiv preprint arXiv:2201.12716, 2022. ii

    2022 arXiv

  31. [40]

    Foundation- pose: Unified 6d pose estimation and tracking of novel objects

    Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundation- pose: Unified 6d pose estimation and tracking of novel objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17868–17879, 2024. i, ii

    2024

  32. [41]

    Any-point trajectory modeling for policy learning

    Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025, 2023. ii

    2023 arXiv

  33. [42]

    Unigarmentmanip: A unified framework for category-level garment manipulation via dense visual correspondence

    Ruihai Wu, Haoran Lu, Yiyan Wang, Yubo Wang, and Hao Dong. Unigarmentmanip: A unified framework for category-level garment manipulation via dense visual correspondence. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16340–16350, 2024...

    2024

  34. [43]

    Learning foresightful dense visual affordance for deformable object manipulation, 2023

    Ruihai Wu, Chuanruo Ning, and Hao Dong. Learning foresightful dense visual affordance for deformable object manipulation, 2023. i, ii

    2023

  35. [44]

    V AT-mart: Learning visual action trajectory proposals for manipulating 3d ARTiculated objects

    Ruihai Wu, Yan Zhao, Kaichun Mo, Zizheng Guo, Yian Wang, Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas Guibas, and Hao Dong. V AT-mart: Learning visual action trajectory proposals for manipulating 3d ARTiculated objects. InInternational Conference on Learning Representations, 2022. i

    2022

  36. [45]

    Garmentpile: Point-level visual affordance guided retrieval and adaptation for cluttered garments manipulation, 2025

    Ruihai Wu, Ziyu Zhu, Yuran Wang, Yue Chen, Jiarui Wang, and Hao Dong. Garmentpile: Point-level visual affordance guided retrieval and adaptation for cluttered garments manipulation, 2025. ii

    2025

  37. [46]

    Spatialtracker: Tracking any 2d pixels in 3d space, 2024

    Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space, 2024. vi

    2024

  38. [47]

    Ruinian Xu, Fu-Jen Chu, Chao Tang, Weiyu Liu, and Patricio A. Vela. An affordance keypoint detection network for robot manipulation. IEEE Robotics and Automation Letters, 6(2):2870–2877, 2021. ii

    2021

  39. [48]

    D-cubed: Latent diffusion trajectory optimisation for dexterous de- formable manipulation, 2024

    Jun Yamada, Shaohong Zhong, Jack Collins, and Ingmar Posner. D-cubed: Latent diffusion trajectory optimisation for dexterous de- formable manipulation, 2024. ii APPENDIX A. Real-world Environment We detailed our real-world environment setup in Fig. 6. Our real-world implementa...

    2024

  40. [49]

    3: Insert Hanger into Shirt

    Task Details:We have implemented the following tasks in real-world and simulation, as shown in Fig. 3: Insert Hanger into Shirt. A shirt is laid flat on the table, while a hanger is suspended from a rod. RobotAfirst grasps the hanger and begins inserting it into the shirt afte...

  41. [50]

    To ensure a fair comparison, we adapt Rekep to our setting with several modifications

    Details on Baseline Methods:We adopt Rekep and UnigarmentManip as baseline methods, as both utilize point- based representations, making them well-suited for handling interactions involving rigid and deformable objects. To ensure a fair comparison, we adapt Rekep to our settin...

  42. [51]

    Insert hanger into shirt

    Comparison with Vision-Language Model Generated Constraints:Many existing works [7]–[9] utilize Vision- Language Models (VLMs) to generate constraints for motion planning. However, in tasks involving rigid-deformable inter- actions, it is often necessary to fully exploit the c...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.