Pith. sign in

REVIEW 5 major objections 5 minor 92 references

DR-LfD claims long-horizon manipulation needs only tens of demonstrations per skill if demonstrations are decomposed into contact-defined atomic skills and recombined by a task-and-motion planner.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-01 02:32 UTC pith:K3ERWTFL

load-bearing objection A promising TAMP-IL hybrid whose central data-complexity claim is plausible but rests on an untested unimodal contact-pose assumption; worth a careful referee, not a desk reject. the 5 major comments →

arxiv 2607.25397 v1 pith:K3ERWTFL submitted 2026-07-28 cs.RO

Decompose and Reorganize: Planning with Primitives and Visuomotor Policies Learned from Demonstrations

classification cs.RO
keywords task and motion planninglearning from demonstrationbimanual manipulationskill decompositioncontact-aware scene graphequivariant diffusionlong-horizon manipulationimitation learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DR-LfD sets out to prove that the combinatorial data cost of long-horizon manipulation can be broken by decomposition: cut demonstrations into atomic skills at contact changes, learn each skill as either a spatially generalizable object-centric primitive or a contact-rich visuomotor policy, and delegate sequencing to a task-and-motion planner. The paper argues this converts a learning problem that scales exponentially in possible skill sequences into one that scales with the number of distinct skill types, each needing roughly 20–100 demonstrations. A sympathetic reader would care because the hybrid targets both known failure modes at once: end-to-end imitation's brittleness to unseen object placements and error accumulation, and classical TAMP's brittleness in contact-rich operations. If correct, a small skill repertoire recombines into plans of 19–21 steps on real hardware, including reachability and collision constraints.

Core claim

On its own terms, the contribution is a complete 'decompose-and-reorganize' pipeline in which a vision-language model assists contact-based segmentation of demonstrations into skills, each skill is wrapped as a TAMP-compatible stream function (with learned primitives, keypose predictors, and safety checks), and an online perceive-plan-act loop re-plans locally after each step. The paper reports that this system outperforms end-to-end imitation baselines on spatially out-of-distribution tabletop tasks, reaches 100% success in peg insertion where baselines drop to 2–12%, and on a public bimanual benchmark achieves with 100 demonstrations per skill what end-to-end policies need 1000 demonstrati

What carries the argument

The load-bearing object is the contact-aware scene graph and its contact signatures: for each skill, a precondition graph and an effect graph that abstract away stochastic intermediate contact states while fixing the initiation and termination conditions. These signatures let TAMP chain skills learned from different demonstration sets, because the unimodal contact-pose assumption makes handed-off gripper–object states interchangeable. The supporting machinery is a set of network-integrated stream functions — attach/detach primitives, uni/bi-manual keypose streams, and a post-grasp bridge — that expose learned skills to the planner as conditional samplers, together with SO(3)-equivariant diff

Load-bearing premise

The framework depends on the assumption that for a common gripper–object contact edge, relative contact poses are similar across skills and demonstrations and follow a single unimodal distribution, so that skills learned from different demonstrations can be sequenced without retraining.

What would settle it

Collect demonstrations of a single skill where the gripper grasps the same object in two qualitatively different ways (e.g., top grasp versus side grasp), run the decomposition, and check whether the contact-pose clusters overlap. If they form two distinct modes, the unimodal assumption fails, and the chained primitives will hand off gripper–object states the policy never saw, producing missed handoffs or failed insertions.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • The demonstration budget for a new long-horizon task grows with the number of skill types, not with sequence length.
  • Skills can be improved in isolation: swapping a weak skill for a better-trained variant does not require recollecting demos or retraining the whole task.
  • Out-of-distribution object placements are handled by first moving objects into in-distribution states with object-centric primitives, then starting the visuomotor policy from a keypose it has seen.
  • Reachability and collision-safety constraints are enforced at plan time, with failed checks feeding back to the planner so it can insert regrasps or obstacle relocations.
  • On a public bimanual benchmark, 100 demonstrations per skill outperform end-to-end imitation trained on 1000 demonstrations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the unimodal contact-pose assumption were relaxed by adding contact-mode arguments to the grasp predicate, the same framework could handle skills that grasp the same object in qualitatively different ways; this is the paper's own stated relaxation path, but it is not implemented.
  • The decomposition principle suggests a complementary scaling route for foundation behavior models: instead of one monolithic policy memorizing full sequences, pretrained local skills can be wrapped as planner streams and chained by TAMP, which may improve long-horizon reliability without more data.
  • A direct test of the framework's central claim would be to measure success rate versus number of demonstrations per skill for a fixed composite task; the paper reports results at specific budgets but does not plot the scaling curve.
  • The object-centric equivariance advantage may shrink outside tabletop settings where objects are not stably segmented or where global context matters; extending to mobile or in-hand manipulation would clarify the boundary.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes DR-LfD, a hybrid framework that decomposes human demonstrations into atomic skills using VLM-assisted, contact-aware scene-graph analysis, learns each skill either as an SO(3)-equivariant object-centric primitive or as a visuomotor policy, and wraps all skills as PDDLStream-compatible stream functions so a sampling-based TAMP planner can recombine them for novel long-horizon tasks. The central claim is that this decomposition changes the demonstration burden from exponential in the length of the skill sequence to roughly linear in the number of distinct skill types, with limited data per skill. The manuscript supports this claim with real-world ALOHA experiments (Peg-in-Hole variants, Object Handoff, Screwdriver Packing, Cup-sleeve Insertion, and two combined long-horizon tasks) and simulation benchmarks (LIBERO and DexMimicGen), plus specialized constraint-handling scenarios for reachability and collision safety.

Significance. If the central claim holds, DR-LfD is a meaningful step toward data-efficient long-horizon manipulation: it provides a concrete mechanism for sequencing closed-loop visuomotor policies with geometric reasoning, avoids 6-DoF pose estimation in learned primitives, and introduces network-integrated stream functions that are useful abstractions for TAMP with learned skills. The paper also contains refreshingly honest and detailed failure/limitation discussions (Section VII, Appendices B and D), including the dependence on manual VLM checking, the exponential TAMP search cost, and the depth-sensing requirement. The architecture is coherent and the experiments are broad, but as discussed below, the headline scaling claim and the key composability assumption are not yet empirically established at the level the paper claims.

major comments (5)
  1. [Section III-B] The unimodal contact-pose assumption is the load-bearing premise for skill recombination, but it is stated and never tested. The paper reports no measurement of the within-edge contact-pose distribution for a common gripper-object edge across different skills. The DexMimicGen ablation (Table VIII, DR-LfD (Planned Contact): 0% and 35%) demonstrates exactly the failure mode this assumption rules out: contact poses that deviate from the demonstration distribution produce OOD wrist-camera observations and policy failure. To support the central data-complexity claim, the authors should either measure that across all skill sets used in Sections VI-B and VI-C the common gripper-object edges have overlapping, unimodal contact-pose distributions, or extend the AtGrasp predicate with explicit contact modes (as the paper itself mentions) and show that TAMP can plan over those modes.
  2. [Abstract, Section I, Table VIII] The headline claim that DR-LfD 'transforms the learning problem from one requiring exponential demonstration data over possible skill sequences to one whose demonstration burden scales with the number of distinct skill types' is asserted rather than demonstrated. No experiment varies the number of demonstrations per skill or the length/combination of skills and measures success as a function of data. Table VIII compares 100 vs 1000 demos for end-to-end baselines but not for DR-LfD's own per-skill data budget, and Section VI-C fixes 20 demos per skill without a data curve. Please add a scaling experiment (e.g., per-skill demo counts vs composite success for 1, 2, and 3 skills, or sequence length vs required demos), or clearly restrict the claim to a structural architectural argument.
  3. [Section VI-C, Fig. 10] The Three-Tapes task, one of the two headline long-horizon composition results, has no reported success rate or trial count; only the 21-step skeleton is described. The Cup-Sponge-Screwdriver task reports a 50% success rate (and later 35%/50% in the clean-background substitution) but no per-subtask failure analysis. These two tasks are the direct evidence for Q3 ('Can DR-LfD complete tasks with novel goals that require multiple steps?'), and as reported they do not quantitatively establish the composability claim. Please report per-task success over a stated number of trials, and include per-subgoal completion statistics (the paper already computes 'average completed steps' for the substitution study, so this is feasible).
  4. [Tables IV–VIII] All success rates are point estimates over 10 or 20 trials with no confidence intervals or significance tests. With 20 trials, the difference between 90% and 100% is within binomial noise; with 10 trials, 10-percentage-point differences are inconclusive. Given the strong claims of outperforming DP, SDP, ACT, and a pretrained VLA baseline, the headline comparative claims need at least Wilson or bootstrap confidence intervals, or raw per-trial outcomes, for the main comparisons. This is especially important in Table V, where DR-LfD and π_{0.5} differ by 5–10 percentage points in several settings.
  5. [Appendix D, Section IV-A1] The VLM grounding stage is not fully automated: the authors state that 'we manually check the correctness of VLM output before the next data processing step' and that task descriptions were manually crafted when not provided. This is disclosed honestly, but it means the 'VLM-assisted' decomposition pipeline requires a human in the loop, and the claimed scalability to new tasks inherits that manual cost. At minimum, the paper should state this limitation prominently in the main text (not only in Appendix D), and ideally report the effect of human-corrected vs raw VLM output on downstream success rates.
minor comments (5)
  1. [Section VII-C] The discussion of depth-sensing limitations is good, but it appears only after the experiments; consider moving a sentence about the depth requirement into Section IV-B where the object-centric primitives are introduced.
  2. [Section V-B2] The MDF safety map is built by replaying demonstrations that were also used to train the policy. This is not circular in the damaging sense, but it may be optimistic if the policy deviates from the demonstrations during deployment. Please add one sentence acknowledging this and why the min-over-time construction is still conservative in the intended use.
  3. [Section VI-A2] The real-world baseline π_{0.5} is described only as 'a stronger pretrained behavior model.' Since the comparison with this baseline is used to make a general claim about VLA models and OOD generalization, please give the exact model, checkpoint, and fine-tuning protocol.
  4. [Table XI] The VLM SGCorr scores for Screwdriver Packing (0.50) and LIBERO-long (0.53) are notably lower than the other tasks; this is informative, but the reader is left wondering how these grounding errors propagate. A short paragraph in Appendix D connecting the low SGCorr cases to the reported success rates would help.
  5. [Throughout] There are several spacing/hyphenation artifacts (e.g., 'DR-LfD(D...' in the abstract, 'aframework' in the introduction). A careful proofread is needed before publication.

Circularity Check

2 steps flagged

Framework is not circular in its central decomposition claim; two minor self-referential definitions (MDF safety envelope, keypose admissibility) are present but not load-bearing.

specific steps
  1. fitted input called prediction [Section V-B.2 (Policy Safety Constraint), Eq. (3)]
    "We replay the demonstration during T a in simulation. At each timestep t∈T a, we voxelize the workspace and record the voxels occupied by the manipulator or related objects as an occupancy map O t. ... A minimum distance field (MDF) is then obtained by aggregating across timesteps, updating each voxel in the workspace with the minimum distance value encountered."

    The IsSafePolicy test stream certifies safety by checking whether obstacles lie inside the MDF, but the MDF is constructed from the same demonstrations that trained the visuomotor policy. Thus 'safe' is defined as 'inside the policy's own demonstration swept volume'; the safety certification reduces to the training envelope rather than independently predicting the stochastic policy's collision-freeness. This is a self-referential constraint, though it is a minor component and the paper's real-world safety experiments provide external evidence.

  2. self definitional [Section IV-C (Learning Visuomotor Policies and Keyposes)]
    "To this end, we design a keypose predictor Φ kp to produce admissible initiation and termination conditions for each segmented skill. ... We learn the distribution of keyposes in both initial and terminal contact states, i.e., P(q a.Gpre , qa.Geff |λ a), conditioned on the skill description λ a."

    The keypose predictor's notion of 'admissible' is defined as being inside the distribution of keyposes extracted from the very demonstrations used to train the visuomotor policy. Therefore, ensuring in-distribution initiation is true by construction: the predictor reproduces the policy's own training distribution. Appendix B makes this explicit: the primitive and keypose predictors 'jointly drive the system observation O to follow the same distribution as that of the policy learned for a.' This is a self-consistency argument, though the paper's successful task rollouts supply independent empirical support.

full rationale

The central claim of DR-LfD is not circular. The data-complexity argument—that decomposing demonstrations into atomic skills and letting TAMP sequence them shifts the demonstration burden from exponential in sequence length to linear in skill types—follows from the architecture and is not an equation that reduces the output to a fitted constant. The learned policies, primitives, and keypose predictors are all trained from demonstrations, but the TAMP skeleton and skill sequencing are computed by a planner, and the empirical benchmarks (LIBERO, DexMimicGen, real-world ALOHA tasks, and ablation comparisons) provide independent content. The two identified items are minor self-referential definitions: the MDF safety map is built by replaying the demonstrations that define the policy, so IsSafePolicy certifies staying inside the training swept volume; and the keypose predictor's 'admissible' states are, by construction, the keypose distribution of the same demonstrations. These are honest design choices and are tested empirically, so they do not undermine the central decomposition claim. No load-bearing self-citation chain or imported uniqueness theorem is present; self-citations such as [15], [17], and [74] are peripheral. Overall, the paper's derivation is largely self-contained, with only minor circular content.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The central claim rests on a set of domain assumptions about contact-pose stability, observation factorization, and perception reliability, plus several hand-chosen thresholds that are not reported. There are no new physical entities; 'policy identifier κ' and contact signatures are data structures, not entities requiring external falsifiable evidence.

free parameters (4)
  • Contact distance threshold
    Section IV-A3: contact established/broken when closest distance is below 'a predefined threshold'; no value or sensitivity analysis is given. This threshold defines the skill timespan boundaries that all downstream training depends on.
  • In-hand velocity alignment threshold
    Section IV-A3: object attachment/detachment is detected by whether object velocity aligns with gripper velocity; the alignment threshold is not specified.
  • Reachability map thresholds
    Section V-B1: a voxel is labeled reachable if its orientation coverage fraction exceeds a threshold, and orientations are compared by a geodesic threshold; neither value is reported.
  • MDF safety distance threshold
    Section V-B2: CFreeMDF and IsSafePolicy treat voxels with 'low distance value' as unsafe; the threshold that separates safe from unsafe is not specified.
axioms (6)
  • domain assumption Unimodal, repeatable contact-pose distribution across skills
    Section III-B states that relative contact poses for a common gripper-object edge lie in a similar, unimodal distribution. This is load-bearing for sequencing skills from different demonstrations.
  • domain assumption Factorized observation model P(O|s) = P(q|s_h) P(I_wrist|s_h-o) P(I_3rd|s_h, s_o)
    Appendix B uses this factorization to argue that predicted keyposes and object-centric primitives keep wrist and third-view observations in-distribution for the visuomotor policy.
  • domain assumption Conditional independence of skill trajectories given point cloud and language
    Equation (1) factorizes the joint trajectory distribution as a product of per-skill distributions; this justifies training one unified diffusion network but ignores cross-skill dependencies.
  • domain assumption Contact signatures Gpre/Geff are sufficient for skill preconditions and effects
    The paper abstracts all stochastic intermediate behavior into start/end contact graphs and uses them as the TAMP interface; this assumes that contact changes fully determine planning-relevant state transitions.
  • domain assumption Perception pipeline accuracy
    Real-world contact detection uses GroundedSAM, XMem, and point-cloud forward kinematics; the pipeline must produce reliable masks and point clouds in contact-rich scenes. The paper notes depth noise degrades performance in the Leaky setting.
  • ad hoc to paper VLM grounding is correct after manual checking
    Section IV-A1 says 'we manually check the correctness of VLM output before the next data processing step'; the method therefore inherits a human-in-the-loop assumption not reflected in the fully-automated framing.

pith-pipeline@v1.3.0-alltime-deepseek · 31587 in / 11921 out tokens · 132593 ms · 2026-08-01T02:32:20.417125+00:00 · methodology

0 comments
read the original abstract

Successfully automating dexterous, long-horizon robotic manipulation requires frameworks capable of both high-level reasoning and fine-grained execution. Traditional task and motion planning (TAMP), while excellent at symbolic planning, is often brittle in contact-rich operations. Simultaneously, imitation learning (IL), while effective in manipulation tasks with visual feedback, is limited by its low capability in spatial generalization and multi-stage operation. To reconcile their complementary strengths and limitations, we propose DR-LfD (Decomposed and Reorganized Skills Learned from Demonstrations), a framework that seamlessly integrates visuomotor policies into a TAMP-gated decision-making system. Based on contact relationships, DR-LfD decomposes human demonstrations into atomic skills, which are reproduced as visuomotor policies or object-centric primitives. The initiation, termination, and constraints of the visuomotor policies are carefully modeled and implemented in a TAMP-compatible form, enabling reorganization of skills learned from different sources. DR-LfD transforms the learning problem from one requiring exponential demonstration data over possible skill sequences to one whose demonstration burden scales with the number of distinct skill types, with limited data for each skill. Through comprehensive real-world and simulation benchmarking across diverse scenarios, we demonstrate the strong performance of DR-LfD on tasks involving multiple steps, unseen setups, and physical constraints. Project website: https://dr-lfd.github.io/DR-LfD-website.

Figures

Figures reproduced from arXiv: 2607.25397 by Ben M. Chen, Dongjie Yu, Hang Xu, Jia Pan, Tengye Xu, Wei Zhang, Yi Ren, Yizhou Chen, Yupu Lu, Zeqing Zhang.

Figure 1
Figure 1. Figure 1: We present DR-LfD, a systematic framework that decomposes demonstrations into skills and reorganizes them for novel tasks. DR-LfD adopts a [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (a) and (c) show the start and end of the contact-rich segment of the Object Handoff operation; (b) shows an intermediate contact-rich state. Scene graphs are shown below each frame. States matching the precondition graph Gpre in (a) and the effect graph Geff in (c) are used for skill transitioning, while intermediate states are used for skill replication. Symbols are world frame ρ, object o1, left arm hl … view at source ↗
Figure 3
Figure 3. Figure 3: The overall diagram of DR-LfD, organized into offline and online phases. In the offline phase, (a) skill acquisition and decomposition extract task [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Skill detection and scene grounding. detecting any contact, which reduces hallucination. A rule￾based step then assembles the tagged operations into each skill’s contact changes. Furthermore, the VLM also captures the names and properties of task-related objects from the input Algorithm 1: GraphAbstraction(·) Input: Arm H, the world frame ρ, skill sequence {ai} n i=1, edge operations {E˜ i} n i=1 Output: U… view at source ↗
Figure 5
Figure 5. Figure 5: The primitive prediction network Φτ and the keypose prediction network Φkp. poses of each skill, as shown in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The SO(3) equivariant network for primitive learning, containing the observation encoding part and noise prediction part. The output U contains an object-centric end-effector (EEF) trajectory, the finger trajectory in a scalar array, and the index when a contact is established between a pair of entities. zinv with the equivariant feature zeqv, yielding a fused feature z ′ eqv that remains equivariant under… view at source ↗
Figure 7
Figure 7. Figure 7: (a) The reachability map constructed for the left puppet arm of the [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: An execution example for the task “(1) Apply the handoff skill; (2) Place the object inside the basket.” When encountering unseen setups, DR￾LfD leverages the learned object-picking skill to initiate the visuomotor policy in an in-distribution observation. Additionally, DR-LfD can complete test-time customized goals with skills in its repertoire. exactly as predicted by the plan. However, the object end po… view at source ↗
Figure 10
Figure 10. Figure 10: Experiments of two long-horizon tasks with customized goals. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗
Figure 9
Figure 9. Figure 9: Executions in unreachable and unsafe scenarios. Table VI presents the profiling of DR-LfD on added con￾straints with success rate and computation time. Baseline methods ACT [3] and DP [5] are omitted because they fail entirely in both unreachable and unsafe scenarios. In contrast, DR-LfD achieves non-zero success rates—55% in unreachable and 70% in unsafe. The snapshots of DR￾LfD in [PITH_FULL_IMAGE:figur… view at source ↗
Figure 11
Figure 11. Figure 11: Initial object-pose distributions in the real-world comparative studies. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Initial poses of task-related objects in the combined [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

92 extracted references · 4 linked inside Pith

  1. [1]

    An incremental constraint-based framework for task and motion planning,

    N. T. Dantam, Z. K. Kingston, S. Chaudhuri, and L. E. Kavraki, “An incremental constraint-based framework for task and motion planning,” The International Journal of Robotics Research, vol. 37, no. 10, pp. 1134–1151, 2018

  2. [2]

    Task and motion planning for execution in the real,

    T. Pan, R. Shome, and L. E. Kavraki, “Task and motion planning for execution in the real,”IEEE Transactions on Robotics, vol. 40, pp. 3356– 3371, 2024

  3. [3]

    Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware,

    T. Zhao, V . Kumar, S. Levine, and C. Finn, “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware,” inRobotics: Science and Systems XIX. Robotics: Science and Systems Foundation, Jul. 2023

  4. [4]

    GELLO: A general, low-cost, and intuitive teleoperation framework for robot manipulators,

    P. Wu, Y . Shentu, Z. Yi, X. Lin, and P. Abbeel, “GELLO: A general, low-cost, and intuitive teleoperation framework for robot manipulators,” in2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 12 156–12 163

  5. [5]

    Diffusion Policy: Visuomotor Policy Learning via Action Diffusion,

    C. Chi, S. Feng, Y . Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “Diffusion Policy: Visuomotor Policy Learning via Action Diffusion,” inRobotics: Science and Systems XIX. Robotics: Science and Systems Foundation, Jul. 2023

  6. [6]

    Implicit behavioral cloning,

    P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson, “Implicit behavioral cloning,” inConference on Robot Learning. PMLR, 2022, pp. 158– 168

  7. [7]

    What matters in learning from offline human demonstrations for robot manipulation,

    A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y . Zhu, and R. Mart ´ın-Mart´ın, “What matters in learning from offline human demonstrations for robot manipulation,” inConference on Robot Learning. PMLR, 2021, pp. 1678–1690

  8. [8]

    3D Diffuser Actor: Policy diffusion with 3d scene representations,

    T.-W. Ke, N. Gkanatsios, and K. Fragkiadaki, “3D Diffuser Actor: Policy diffusion with 3d scene representations,” inConference on Robot Learning. PMLR, 2024, pp. 1949–1974

  9. [9]

    3D Diffusion Policy: Generalizable visuomotor policy learning via simple 3D repre- sentations,

    Y . Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu, “3D Diffusion Policy: Generalizable visuomotor policy learning via simple 3D repre- sentations,” inICRA 2024 Workshop on 3D Visual Representations for Robot Manipulation, 2024

  10. [10]

    π0: A vision-language-action flow model for general robot control,

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichteret al., “π0: A vision-language-action flow model for general robot control,” inRobotics: Science and Systems, 2025

  11. [11]

    ALOHA Unleashed: A simple recipe for robot dexterity,

    T. Z. Zhao, J. Tompson, D. Driess, P. Florence, K. Ghasemipour, C. Finn, and A. Wahid, “ALOHA Unleashed: A simple recipe for robot dexterity,” inConference on Robot Learning. PMLR, 2024, pp. 1910–1924

  12. [12]

    RDT-1B: A diffusion foundation model for bimanual manipulation,

    S. Liu, L. Wu, B. Li, H. Tan, H. Chen, Z. Wang, K. Xu, H. Su, and J. Zhu, “RDT-1B: A diffusion foundation model for bimanual manipulation,” inInternational Conference on Learning Representations (ICLR), 2025

  13. [13]

    A careful exami- nation of large behavior models for multitask dexterous manipulation,

    J. Barreiros, A. Beaulieu, A. Bhat, R. Cory, E. Cousineau, H. Dai, C.-H. Fang, K. Hashimoto, M. Z. Irshad, M. Itkinaet al., “A careful exami- nation of large behavior models for multitask dexterous manipulation,” Science Robotics, vol. 11, no. 113, p. eaea6201, 2026

  14. [14]

    A Bimanual Manipulation Taxonomy,

    F. Krebs and T. Asfour, “A Bimanual Manipulation Taxonomy,”IEEE Robot. Autom. Lett., vol. 7, no. 4, pp. 11 031–11 038, Oct. 2022

  15. [15]

    BiKC: Keypose-conditioned consistency policy for bimanual robotic manipulation,

    D. Yu, H. Xu, Y . Chen, Y . Ren, and J. Pan, “BiKC: Keypose-conditioned consistency policy for bimanual robotic manipulation,” inInternational Workshop on the Algorithmic Foundations of Robotics (WAFR), 2024

  16. [16]

    Combined task and motion planning for a dual-arm robot to use a suction cup tool,

    H. Chen, W. Wan, and K. Harada, “Combined task and motion planning for a dual-arm robot to use a suction cup tool,” in2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids). IEEE, 2019, pp. 446–452

  17. [17]

    Enabling versatility and dexterity of the dual- arm manipulators: A general framework toward universal cooperative manipulation,

    Y . Ren, Z. Zhou, Z. Xu, Y . Yang, G. Zhai, M. Leibold, F. Ni, Z. Zhang, M. Buss, and Y . Zheng, “Enabling versatility and dexterity of the dual- arm manipulators: A general framework toward universal cooperative manipulation,”IEEE Transactions on Robotics, vol. 40, 2024

  18. [18]

    Efficient task/motion planning for a dual-arm robot from language instructions and cooking images,

    K. Takata, T. Kiyokawa, I. G. Ramirez-Alpizar, N. Yamanobe, W. Wan, and K. Harada, “Efficient task/motion planning for a dual-arm robot from language instructions and cooking images,” in2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 12 058–12 065

  19. [19]

    Bi-KVIL: Keypoints-based Visual Imitation Learning of Bimanual Manipulation Tasks,

    J. Gao, Z. Tao, N. Jaquier, and T. Asfour, “Bi-KVIL: Keypoints-based Visual Imitation Learning of Bimanual Manipulation Tasks,” in2024 IEEE International Conference on Robotics and Automation (ICRA), 2024

  20. [20]

    Gripper keypose and object pointflow as interfaces for bimanual robotic manipulation,

    Y . Yang, Z. Cai, Y . Tian, J. Zeng, and J. Pang, “Gripper keypose and object pointflow as interfaces for bimanual robotic manipulation,” in Robotics: Science and Systems, 2025

  21. [21]

    Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning,

    Z. Jiang, Y . Xie, K. Lin, Z. Xu, W. Wan, A. Mandlekar, L. J. Fan, and Y . Zhu, “Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning,” in2025 IEEE Interna- tional Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 16 923–16 930

  22. [22]

    Mimicgen: A data generation system for scalable robot learning using human demonstrations,

    A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y . Narang, L. Fan, Y . Zhu, and D. Fox, “Mimicgen: A data generation system for scalable robot learning using human demonstrations,” inConference on Robot Learning. PMLR, 2023, pp. 1820–1864

  23. [23]

    Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation,

    T. Chen, Z. Chen, B. Chen, Z. Cai, Y . Liu, Z. Li, Q. Liang, X. Lin, Y . Ge, Z. Guet al., “Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation,”arXiv preprint arXiv:2506.18088, 2025

  24. [24]

    Novel demonstration generation with gaussian splatting enables robust one-shot manipulation,

    S. Yang, W. Yu, J. Zeng, J. Lv, K. Ren, C. Lu, D. Lin, and J. Pang, “Novel demonstration generation with gaussian splatting enables robust one-shot manipulation,” inRobotics: Science and Systems, 2025

  25. [25]

    Equibot: SIM(3)-Equivariant diffusion policy for generalizable and data efficient learning,

    J. Yang, Z.-a. Cao, C. Deng, R. Antonova, S. Song, and J. Bohg, “Equibot: SIM(3)-Equivariant diffusion policy for generalizable and data efficient learning,” inConference on Robot Learning. PMLR, 2024

  26. [26]

    Demogen: Synthetic demonstration generation for data-efficient visuomotor policy learning,

    Z. Xue, S. Deng, Z. Chen, Y . Wang, Z. Yuan, and H. Xu, “Demogen: Synthetic demonstration generation for data-efficient visuomotor policy learning,” inRobotics: Science and Systems, 2025

  27. [27]

    You only teach once: Learn one-shot bimanual robotic manipulation from video demonstrations,

    H. Zhou, R. Wang, Y . Tai, Y . Deng, G. Liu, and K. Jia, “You only teach once: Learn one-shot bimanual robotic manipulation from video demonstrations,” inRobotics: Science and Systems, 2025

  28. [28]

    Diffusion-EDFs: Bi-Equivariant denoising generative modeling on SE(3) for visual robotic manipulation,

    H. Ryu, J. Kim, H. An, J. Chang, J. Seo, T. Kim, Y . Kim, C. Hwang, J. Choi, and R. Horowitz, “Diffusion-EDFs: Bi-Equivariant denoising generative modeling on SE(3) for visual robotic manipulation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18 007–18 018

  29. [29]

    RiEMann: Near real-time SE(3)-equivariant robot manipulation without point cloud segmentation,

    C. Gao, Z. Xue, S. Deng, T. Liang, S. Yang, L. Shao, and H. Xu, “RiEMann: Near real-time SE(3)-equivariant robot manipulation without point cloud segmentation,” inConference on Robot Learning. PMLR, 2024

  30. [30]

    3d equivariant visuomotor policy learning via spherical projection,

    B. Hu, D. Wang, D. Klee, H. Tian, X. Zhu, H. Huang, R. Platt, and R. Walters, “3d equivariant visuomotor policy learning via spherical projection,” inAdvances in Neural Information Processing Systems, 2025

  31. [31]

    A practical guide for incorporating symmetry in diffusion policy,

    D. Wang, B. Hu, S. Song, R. Walters, and R. Platt, “A practical guide for incorporating symmetry in diffusion policy,” inAdvances in Neural Information Processing Systems, 2025

  32. [32]

    Plan-seq- learn: Language model guided rl for solving long horizon robotics tasks,

    M. Dalal, T. Chiruvolu, D. Chaplot, and R. Salakhutdinov, “Plan-seq- learn: Language model guided rl for solving long horizon robotics tasks,” inInternational Conference on Learning Representations (ICLR), 2024

  33. [33]

    League: Guided skill learning and abstraction for long-horizon manipulation,

    S. Cheng and D. Xu, “League: Guided skill learning and abstraction for long-horizon manipulation,”IEEE Robotics and Automation Letters, 2023

  34. [34]

    STAP: Sequencing task- agnostic policies,

    C. Agia, T. Migimatsu, J. Wu, and J. Bohg, “STAP: Sequencing task- agnostic policies,” in2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 7951–7958

  35. [35]

    Logic-skill program- ming: An optimization-based approach to sequential skill planning,

    T. Xue, A. Razmjoo, S. Shetty, and S. Calinon, “Logic-skill program- ming: An optimization-based approach to sequential skill planning,” in Robotics: Science and Systems, 2024

  36. [36]

    Spire: Synergistic planning, imitation, and reinforcement learning for long- horizon manipulation,

    Z. Zhou, A. Garg, D. Fox, C. Garrett, and A. Mandlekar, “Spire: Synergistic planning, imitation, and reinforcement learning for long- horizon manipulation,” inConference on Robot Learning. PMLR, 2024, pp. 2347–2371. 18

  37. [37]

    NOD-TAMP: Multi- step manipulation planning with neural object descriptors,

    S. Cheng, C. R. Garrett, A. Mandlekar, and D. Xu, “NOD-TAMP: Multi- step manipulation planning with neural object descriptors,” inCoRL 2023 Workshop on Learning Effective Abstractions for Planning (LEAP), 2023

  38. [38]

    Logic learning from demonstrations for multi-step manipulation tasks in dynamic environ- ments,

    Y . Zhang, T. Xue, A. Razmjoo, and S. Calinon, “Logic learning from demonstrations for multi-step manipulation tasks in dynamic environ- ments,”IEEE Robotics and Automation Letters, 2024

  39. [39]

    Generative factor chaining: Coor- dinated manipulation with diffusion-based factor graph,

    U. A. Mishra, Y . Chen, and D. Xu, “Generative factor chaining: Coor- dinated manipulation with diffusion-based factor graph,” inICRA 2024 Workshop - Back to the Future: Robot Learning Going Probabilistic, 2024

  40. [40]

    Human-in-the-loop task and motion planning for imitation learning,

    A. Mandlekar, C. R. Garrett, D. Xu, and D. Fox, “Human-in-the-loop task and motion planning for imitation learning,” inConference on Robot Learning. PMLR, 2023, pp. 3030–3060

  41. [41]

    Skillmimicgen: Automated demonstration generation for efficient skill learning and deployment,

    C. Garrett, A. Mandlekar, B. Wen, and D. Fox, “Skillmimicgen: Automated demonstration generation for efficient skill learning and deployment,” inConference on Robot Learning. PMLR, 2024, pp. 2750–2790

  42. [42]

    Learning compositional behaviors from demonstration and language,

    W. Liu, N. Nie, R. Zhang, J. Mao, and J. Wu, “Learning compositional behaviors from demonstration and language,” in8th Annual Conference on Robot Learning, 2024

  43. [43]

    Learning a thousand tasks in a day,

    K. Dreczkowski, P. Vitiello, V . V osylius, and E. Johns, “Learning a thousand tasks in a day,”Science Robotics, vol. 10, no. 108, p. eadv7594, 2025

  44. [44]

    Learning Reusable Manipulation Strategies,

    J. Mao, T. Lozano-P ´erez, J. B. Tenenbaum, and L. P. Kaelbling, “Learning Reusable Manipulation Strategies,” in7th Annual Conference on Robot Learning, Aug. 2023

  45. [45]

    InterPreT: Interactive predicate learning from language feedback for generalizable task planning,

    M. Han, Y . Zhu, S.-C. Zhu, Y . N. Wu, and Y . Zhu, “InterPreT: Interactive predicate learning from language feedback for generalizable task planning,” inRobotics: Science and Systems, 2024

  46. [46]

    UniDomain: Pretraining a unified PDDL domain from real-world demonstrations for generalizable robot task planning,

    H. Ye, Y . Xiao, C. Lu, and P. Cai, “UniDomain: Pretraining a unified PDDL domain from real-world demonstrations for generalizable robot task planning,” inAdvances in Neural Information Processing Systems, 2025

  47. [47]

    Vlm see, robot do: Human demo video to robot action plan via vision language model,

    B. Wang, J. Zhang, S. Dong, I. Fang, and C. Feng, “Vlm see, robot do: Human demo video to robot action plan via vision language model,” in2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2025, pp. 17 215–17 222

  48. [48]

    Do as i can, not as i say: Grounding language in robotic affordances,

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausmanet al., “Do as i can, not as i say: Grounding language in robotic affordances,” inConference on Robot Learning. PMLR, 2022, pp. 287–318

  49. [49]

    Progprompt: Generating situated robot task plans using large language models,

    I. Singh, V . Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “Progprompt: Generating situated robot task plans using large language models,” in2023 IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 11 523– 11 530

  50. [50]

    Differentiable physics and stable modes for tool-use and manipulation planning,

    M. A. Toussaint, K. R. Allen, K. A. Smith, and J. B. Tenenbaum, “Differentiable physics and stable modes for tool-use and manipulation planning,” 2018

  51. [51]

    Object-centric task and motion planning in dynamic environments,

    T. Migimatsu and J. Bohg, “Object-centric task and motion planning in dynamic environments,”IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 844–851, 2020

  52. [52]

    Sampling-based methods for factored task and motion planning,

    C. R. Garrett, T. Lozano-P ´erez, and L. P. Kaelbling, “Sampling-based methods for factored task and motion planning,”The International Journal of Robotics Research, vol. 37, no. 13-14, pp. 1796–1825, 2018

  53. [53]

    PDDLStream: Integrating symbolic planners and blackbox sam- plers via optimistic adaptive planning,

    ——, “PDDLStream: Integrating symbolic planners and blackbox sam- plers via optimistic adaptive planning,” inProceedings of the interna- tional conference on automated planning and scheduling, vol. 30, 2020, pp. 440–448

  54. [54]

    Long-horizon manipulation of unknown objects via task and motion planning with estimated affordances,

    A. Curtis, X. Fang, L. P. Kaelbling, T. Lozano-P ´erez, and C. R. Garrett, “Long-horizon manipulation of unknown objects via task and motion planning with estimated affordances,” in2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022, pp. 1940–1946

  55. [55]

    Unseen object instance segmentation for robotic environments,

    C. Xie, Y . Xiang, A. Mousavian, and D. Fox, “Unseen object instance segmentation for robotic environments,”IEEE Transactions on Robotics, vol. 37, no. 5, pp. 1343–1359, 2021

  56. [56]

    High precision grasp pose detection in dense clutter,

    M. Gualtieri, A. Ten Pas, K. Saenko, and R. Platt, “High precision grasp pose detection in dense clutter,” in2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016, pp. 598–605

  57. [57]

    6-DoF Graspnet: Varia- tional grasp generation for object manipulation,

    A. Mousavian, C. Eppner, and D. Fox, “6-DoF Graspnet: Varia- tional grasp generation for object manipulation,” inProceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 2901– 2910

  58. [58]

    Gen6D: Generalizable model-free 6-DoF object pose estimation from rgb images,

    Y . Liu, Y . Wen, S. Peng, C. Lin, X. Long, T. Komura, and W. Wang, “Gen6D: Generalizable model-free 6-DoF object pose estimation from rgb images,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 298–315

  59. [59]

    POPE: 6- DoF promptable pose estimation of any object in any scene with one reference,

    Z. Fan, P. Pan, P. Wang, Y . Jiang, D. Xu, and Z. Wang, “POPE: 6- DoF promptable pose estimation of any object in any scene with one reference,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 7771–7781

  60. [60]

    FoundationPose: Unified 6D pose estimation and tracking of novel objects,

    B. Wen, W. Yang, J. Kautz, and S. Birchfield, “FoundationPose: Unified 6D pose estimation and tracking of novel objects,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 17 868–17 879

  61. [61]

    Unified task and motion planning us- ing object-centric abstractions of motion constraints,

    A. Agostini and J. Piater, “Unified task and motion planning us- ing object-centric abstractions of motion constraints,”arXiv preprint arXiv:2312.17605, 2023

  62. [62]

    Grounded SAM: Assembling open-world models for diverse visual tasks,

    T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y . Chen, F. Yanet al., “Grounded SAM: Assembling open-world models for diverse visual tasks,”arXiv preprint arXiv:2401.14159, 2024

  63. [63]

    Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model,

    H. K. Cheng and A. G. Schwing, “Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model,” inEuropean conference on computer vision. Springer, 2022, pp. 640–658

  64. [64]

    D (r, o) grasp: A unified representation of robot and object interaction for cross-embodiment dexterous grasping,

    Z. Wei, Z. Xu, J. Guo, Y . Hou, C. Gao, Z. Cai, J. Luo, and L. Shao, “D (r, o) grasp: A unified representation of robot and object interaction for cross-embodiment dexterous grasping,” in2025 IEEE International Conference on Robotics and Automation (ICRA), 2025

  65. [65]

    Vector Neurons: A general framework for SO(3)-equivariant networks,

    C. Deng, O. Litany, Y . Duan, A. Poulenard, A. Tagliasacchi, and L. J. Guibas, “Vector Neurons: A general framework for SO(3)-equivariant networks,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 12 200–12 209

  66. [66]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inPro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), 2019

  67. [67]

    EFEM: Equivariant neural field expectation maximization for 3D object seg- mentation without scene supervision,

    J. Lei, C. Deng, K. Schmeckpeper, L. Guibas, and K. Daniilidis, “EFEM: Equivariant neural field expectation maximization for 3D object seg- mentation without scene supervision,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 4902–4912

  68. [68]

    Denoising Diffusion Probabilistic Models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising Diffusion Probabilistic Models,” inAdvances in Neural Information Processing Systems, vol. 33. Curran Associates, Inc., 2020, pp. 6840–6851

  69. [69]

    Should ebms model the energy or the score?

    T. Salimans and J. Ho, “Should ebms model the energy or the score?” inEnergy Based Models Workshop-ICLR 2021, 2021

  70. [70]

    Generative modeling by estimating gradients of the data distribution,

    Y . Song and S. Ermon, “Generative modeling by estimating gradients of the data distribution,”Advances in neural information processing systems, vol. 32, 2019

  71. [71]

    Score-Based Generative Modeling through Stochastic Differential Equations,

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” inInternational Conference on Learning Rep- resentations, Oct. 2020

  72. [72]

    Coast: Constraints and streams for task and motion planning,

    B. Vu, T. Migimatsu, and J. Bohg, “Coast: Constraints and streams for task and motion planning,” in2024 IEEE International Conference on Robotics and Automation (ICRA), 2024, pp. 14 875–14 881

  73. [73]

    Schedulestream: Temporal planning with samplers for gpu-accelerated multi-arm task and motion planning & scheduling,

    C. Garrett and F. Ramos, “Schedulestream: Temporal planning with samplers for gpu-accelerated multi-arm task and motion planning & scheduling,” in2026 IEEE International Conference on Robotics and Automation (ICRA), 2026

  74. [74]

    Gpu-accelerated incremental euclidean distance transform for online motion planning of mobile robots,

    Y . Chen, S. Lai, J. Cui, B. Wang, and B. M. Chen, “Gpu-accelerated incremental euclidean distance transform for online motion planning of mobile robots,”IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 6894–6901, 2022

  75. [75]

    One demo is worth a thousand trajectories: Action-view augmentation for visuomotor policies,

    C. Pan, L. Liang, D. Bauer, E. Cousineau, B. Burchfiel, S. Feng, and S. Song, “One demo is worth a thousand trajectories: Action-view augmentation for visuomotor policies,” in9th Annual Conference on Robot Learning, 2025

  76. [76]

    Rail: Reachability-aided imitation learning for safe policy execution,

    W. Jung, D. Anthony, U. A. Mishra, N. R. Arachchige, M. Bronars, D. Xu, and S. Kousik, “Rail: Reachability-aided imitation learning for safe policy execution,” in2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 3582–3589

  77. [77]

    Libero: Benchmarking knowledge transfer for lifelong robot learning,

    B. Liu, Y . Zhu, C. Gao, Y . Feng, Q. Liu, Y . Zhu, and P. Stone, “Libero: Benchmarking knowledge transfer for lifelong robot learning,”Advances in Neural Information Processing Systems, vol. 36, pp. 44 776–44 791, 2023

  78. [78]

    Grasp pose detection in point clouds,

    A. Ten Pas, M. Gualtieri, K. Saenko, and R. Platt, “Grasp pose detection in point clouds,”The International Journal of Robotics Research, vol. 36, no. 13-14, pp. 1455–1473, 2017

  79. [79]

    M2t2: Multi-task masked transformer for object-centric pick and place,

    W. Yuan, A. Murali, A. Mousavian, and D. Fox, “M2t2: Multi-task masked transformer for object-centric pick and place,” in7th Annual Conference on Robot Learning, 2023. 19

  80. [80]

    TiPToP: A modular open-vocabulary planning system for robotic manipulation,

    W. Shen, N. Kumar, S. Chintalapudi, J. Wang, C. Watson, E. Hu, J. Cao, D. Jayaraman, L. P. Kaelbling, and T. Lozano-P ´erez, “TiPToP: A modular open-vocabulary planning system for robotic manipulation,” arXiv preprint arXiv:2603.09971, 2026

Showing first 80 references.