Pith. sign in

REVIEW 4 major objections 5 minor 44 references

VLM-TDP: VLM-guided Trajectory-conditioned Diffusion Policy for Robust Long-Horizon Manipulation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Trajectory hints from a vision-language model make diffusion policies robust for long-horizon manipulation.

desk verdict Trajectory conditioning on voxel plans is a real, useful effect, but the paper's headline numbers are attributed to the wrong variant and the VLM decomposition is untested. read the letter →

arxiv 2507.04524 v1 pith:S6F5KAB3 submitted 2025-07-06 cs.RO

Kefeng Huang , Tingguang Li , Yuzhen Liu , Zhe Zhang , Jiankun Wang , Lei Han This is my paper · ORCID
classification cs.RO
keywords diffusionpolicyvision-languagemodeltrajectoryconditioningvoxel-basedlong-horizonmanipulationrobotictaskdecompositionrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the bottleneck for diffusion policies in manipulation is not just the action-generation model but the weakness of the conditioning signal: RGB-only conditioning leaves the policy uncertain about where to go, especially in long tasks and under image noise. It proposes feeding a vision-language model's voxel-based end-effector trajectory as an extra condition, along with decomposing long tasks into gripper-bracketed sub-tasks. Across simulation and a physical robot, this raises average task success by 44%, more than doubles long-horizon success, and reduces performance loss under noisy images or altered environments by about 20%. If right, the method shows that robust long-horizon manipulation can be achieved without new sensors, by injecting a cheap spatial hint from a VLM into a standard diffusion policy.

What carries the argument

The load-bearing object is the voxel-based spatial trajectory representation: an ordered M×N×K matrix, with M=N=K=6, in which cells are labeled by their order in the end-effector path and off-path cells are 0. It encodes where the tool should go in 3D, including height, unlike a flat 2D line or a color-coded 2.5D sketch. The trajectory is produced by asking the VLM to choose cells on a grid-annotated top-down image, and it is consumed by the policy through three 3D-convolutional layers whose output is concatenated with image and state encodings before the noise predictor. A second mechanism is task decomposition into gripper-bracketed sub-tasks, which removes repeated or reversed motion segments that would make a single trajectory ambiguous. Together, they convert the VLM's coarse spatial plan into a steering signal for a trained low-level action generator.

What would settle it

Train the trajectory-conditioned policy, then evaluate it on the same stacking tasks using randomly sampled voxel trajectories drawn from the demonstration-derived distribution instead of VLM-generated ones: if success stays high, the semantic content of the VLM path is not what carries the gain, and if it collapses, the gain rests on a VLM-to-demonstration similarity that the paper never measures.

Watch

Extended reading notes

Core claim

The paper's central claim is that conditioning a denoising diffusion policy on a VLM-generated voxel trajectory makes the policy succeed where RGB-only conditioning fails, and keeps it working when images are degraded. The proposed system first asks a VLM to split a long task into sub-tasks, each starting and ending with a gripper open-close transition, then has the same VLM mark waypoints on a 6x6x6 voxel grid derived from a top-down camera view. That ordered voxel path is encoded by 3D convolutions and injected once into the denoising network alongside the RGB history and robot state. The authors argue that the spatial hint removes ambiguity about which object to target and how to approach it, so the policy degrades less under noise and visual variation and can execute long sequences by following per-sub-task trajectories. Both TDP, using ground-truth trajectories, and VLM-TDP, using VLM-generated trajectories, outperform the classical diffusion policy in all tested tasks, with the gap growing as task length increases.

Load-bearing premise

The weakest premise is that VLM-generated voxel trajectories, which are never shown to the policy during training, resemble the demonstration-derived trajectories closely enough that the same learned policy can follow both; the paper shows the fragility of this assumption when close-together blocks make VLM trajectories ambiguous.

Editorial extensions

If this is right

  • In the four-block stacking task, the plain diffusion policy never completes the episode, while VLM-TDP reaches 0.04 combined success, and every increase in task length widens the gap.
  • Under Gaussian image noise with σ=0.64, the trajectory-conditioned policy retains an absolute 15% success rate on the Pick sub-task, while the diffusion policy's Pick success drops to 0%.
  • Across the seven simulation tasks, both TDP and VLM-TDP beat the classical diffusion policy in every task, raising flat or hard-to-see tasks like Phone on Base from 0.06–0.11 to about 0.53–0.55.
  • In real-world trials the advantage grows with horizon: 85% versus 70% for one orange, 95% versus 70% for two bananas, and 70% versus 20% for three bananas.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the same trajectory condition is injected once per sub-task, the policy effectively delegates sequencing and target selection to the VLM; a natural test is to freeze the learned policy and swap in cheaper or weaker planners to see how much of the gain comes from the spatial hint itself rather than from task decomposition.
  • The 6x6x6 voxel grid is a visible resolution bottleneck: the paper reports Pick success falling from 0.88 to about 0.5 when two blocks are placed close together, and a coarse-to-fine or adaptive-resolution prompting scheme is the obvious next knob to turn.
  • Since the method needs only a front camera plus a VLM query, the same trajectory-conditioning idea should transfer to closed-loop visuomotor backbones other than diffusion policies, such as action-chunking transformers or other imitation-learning architectures.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes VLM-TDP, a diffusion-policy extension that conditions action generation on voxel-based end-effector trajectories produced by a vision-language model. A trajectory-conditioned diffusion policy (TDP) is trained on trajectories extracted from demonstrations, and VLM-TDP is the same policy evaluated with trajectories generated by GPT-4o. The authors claim a 44% average success-rate improvement over classical diffusion policy, an over-100% improvement on long-horizon tasks, a 20% reduction in performance degradation under noise or environmental changes, and real-world validation on a Franka Panda robot. Experiments are conducted on seven RLBench tasks, Stack Blocks long-horizon tasks with RLBench and Colosseum robustness variations, and three real-world pick-and-place tasks.

Significance. If the central claim were fully supported, the voxel-trajectory conditioning idea would be a practical, sensor-light way to improve diffusion-policy robustness and long-horizon performance. The paper includes useful comparisons against 2D and 3D diffusion baselines, several simulation benchmarks, real-robot experiments, and a video. The main scientific contribution, VLM-generated voxel trajectories as a conditioning modality, is clear and worth pursuing. However, the numerical headline claims are associated with TDP using oracle ground-truth trajectories rather than with the full VLM-TDP system, the long-horizon evaluation uses manually chosen sub-task splits, and the real-world experiments appear to evaluate TDP rather than VLM-TDP. These mismatches mean the significance is currently not fully established in the form stated.

major comments (4)
  1. [IV-A, Table I, Abstract] The headline '44% improvement in average task success' is supported by TDP (0.71 vs. diffusion policy 0.49 in Table I), not by VLM-TDP, whose Table I average is 0.69 (about 41% relative improvement). Similarly, the robustness claim of a '20% reduction in performance degradation' is evaluated for TDP in Section IV-C, not for VLM-TDP. Please attribute all headline numbers to the correct system, or restate them for VLM-TDP and clearly label TDP as an oracle-trajectory upper bound.
  2. [IV-B] The long-horizon experiment does not exercise the proposed VLM task decomposition: the paper states 'We manually split the task into several sub-tasks and report the results.' Therefore the claimed over-100% long-horizon improvement demonstrates the benefit of trajectory conditioning combined with manual decomposition, not the VLM-based decomposition described in Section III-B. Please either rephrase the claim or add an experiment in which the VLM generates the sub-task split for Stack Blocks.
  3. [V, Table III] The real-world experiments are labeled 'TDP (ours)' in Table III and Section V-B, and the text does not describe any VLM-generated trajectories or VLM task decomposition in the real-world setup. The Abstract and Conclusion claim real-world validation of VLM-TDP, but the reported real-world numbers support only the trajectory-conditioned policy with demonstration-derived trajectories. Please run VLM-TDP on the physical robot or explicitly qualify the real-world claim as validating TDP alone.
  4. [III-C, IV] There is an unexamined distribution shift between the trajectory conditions used at training and at deployment. Section III-C says trajectory conditions are derived from demonstrations, while Section IV states VLM-TDP is validated using trajectories generated by GPT-4o. The paper provides no statistics on VLM trajectory validity, agreement with demonstration trajectories, or prompt details, and no per-condition variance. The ambiguity example in Section IV-B, where close blocks reduce Pick success from 0.88 to about 0.5, shows that VLM trajectories can mislead the policy. Please add an analysis of the trajectory distribution shift and report per-condition results and variances.
minor comments (5)
  1. [II-C] There is a duplicated word in 'relies on the VLM solely solely for trajectory selection'; please fix the typo.
  2. [VI] The sentence 'In the presene of varying input images' contains a typo; it should be 'presence'.
  3. [IV] The evaluation protocol 'averaging the five highest success rates' over checkpoints is unusual and can overestimate performance; please also report the mean over the last several checkpoints or justify the selection protocol.
  4. [IV-B] The task name 'Put Item on Drawer' in the text appears to be a typo for 'Put Item in Drawer' used elsewhere.
  5. [III-B] The VLM prompt templates and the exact GPT-4o model configuration are not provided; without these, the trajectory generation component is not reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core evaluation uses held-out episodes and trajectories from the external GPT-4o model, so the reported successes are not equivalent to the method's inputs by construction.

full rationale

The paper's central chain is: train a trajectory-conditioned diffusion policy using trajectories extracted from demonstrations, then at deployment feed trajectories generated by GPT-4o into the policy and measure task success on held-out episodes in RLBench, Colosseum, and real-robot trials. The evaluation is therefore self-contained against external benchmarks and an external VLM, not a restatement of the training data or of a fitted parameter. The potentially suspicious elements — TDP evaluated with ground-truth demonstration trajectories, manual splitting of long-horizon tasks into sub-tasks, top-5 checkpoint averaging, and the abstract's 44% improvement matching TDP rather than VLM-TDP — are methodology or reporting concerns about what precisely is being tested, but none of them satisfies the standard of a circular reduction: no equation is equivalent to its own input by construction, no fitted parameter is renamed as a prediction, and no load-bearing claim depends on a self-citation chain or on a uniqueness theorem imported from the authors' prior work. The reported numbers come from held-out simulations and robot trials, so the central quantitative claims are not forced by the method's definitions. Accordingly, the appropriate circularity finding is no significant circularity, and the score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on two tuned numbers (grid resolution, action horizon), several domain assumptions (fixed camera, sub-task definition), and an evaluation protocol that selects the best checkpoints and manually splits long-horizon tasks. No invented physical entities are introduced.

free parameters (3)
  • Voxel grid resolution (M, N, K) = 6 x 6 x 6
    Set by hand in Section III-B. Directly controls VLM trajectory precision; the paper notes that higher resolution makes VLM generation harder, and grid coarseness causes grasp ambiguity when blocks are close (Section IV-B).
  • Action chunk horizon T and execution count N = T=12, N=8
    Chosen empirically: 'Empirically, we found T = 12 and N = 8 work well' (Section IV). Affects all methods equally but is tuned for this task set.
  • Number of training epochs and checkpoint selection = 500 epochs, best-5 average
    Inherited from Lang-o3dp [8]; the best-of-training evaluation protocol inflates reported success rates relative to the final model.
assumptions (5)
  • standard math DDPM noise-prediction objective and iterative denoising formula (Eq. 1-2) correctly model the action distribution.
    Background from Ho et al. [6] and Diffusion Policy [4]; the paper modifies the noise predictor to include trajectory conditioning.
  • domain assumption Robot base and front camera remain fixed throughout task execution.
    Section III-C: 'We assume that the robot base and the front camera remain fixed throughout the task'. Required to map end-effector positions into the fixed voxel grid via calibrated extrinsics and intrinsics.
  • domain assumption Gripper open/close delimited sub-tasks remove trajectory ambiguity.
    Section III-B: a sub-task 'begins with the opening or closing the gripper and ends with closing or opening the gripper', which 'eliminates the occurrence of repeated or reversed trajectories'. This assumption underlies the decomposition benefit.
  • ad hoc to paper Manually chosen sub-task splits for Stack Blocks are representative of VLM decomposition.
    Section IV-B: 'We manually split the task into several sub-tasks and report the results'. The long-horizon improvement claim is validated with the authors' own splits, not the VLM pipeline, so the evaluation is not fully independent of the authors' choices.
  • ad hoc to paper Best-of-training checkpoint averaging (top 5 of 20 evaluations) estimates deployment performance.
    Section IV: 'We report the result by averaging the five highest success rates'. This convention (from [8]) assumes the best checkpoints are attainable at deployment, which is not demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VLM-TDP: VLM-guided Trajectory-conditioned Diffusion Policy for Robust Long-Horizon Manipulation." pith.science (2026). https://pith.science/paper/S6F5KAB3

@misc{pith2026250704524,
  author       = {Pith},
  title        = {Pith review of: VLM-TDP: VLM-guided Trajectory-conditioned Diffusion Policy for Robust Long-Horizon Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S6F5KAB3}},
  note         = {Machine review of arXiv:2507.04524}
}
read the original abstract

Diffusion policy has demonstrated promising performance in the field of robotic manipulation. However, its effectiveness has been primarily limited in short-horizon tasks, and its performance significantly degrades in the presence of image noise. To address these limitations, we propose a VLM-guided trajectory-conditioned diffusion policy (VLM-TDP) for robust and long-horizon manipulation. Specifically, the proposed method leverages state-of-the-art vision-language models (VLMs) to decompose long-horizon tasks into concise, manageable sub-tasks, while also innovatively generating voxel-based trajectories for each sub-task. The generated trajectories serve as a crucial conditioning factor, effectively steering the diffusion policy and substantially enhancing its performance. The proposed Trajectory-conditioned Diffusion Policy (TDP) is trained on trajectories derived from demonstration data and validated using the trajectories generated by the VLM. Simulation experimental results indicate that our method significantly outperforms classical diffusion policies, achieving an average 44% increase in success rate, over 100% improvement in long-horizon tasks, and a 20% reduction in performance degradation in challenging conditions, such as noisy images or altered environments. These findings are further reinforced by our real-world experiments, where the performance gap becomes even more pronounced in long-horizon tasks. Videos are available on https://youtu.be/g0T6h32OSC8

Figures

Figures reproduced from arXiv: 2507.04524 by the authors.

Figure 1
Figure 1. The overview of VLM-TDP. We utilize VLM to decompose a task into a series of sub-tasks and then generate trajectories for each sub-task (bottom [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Examples of voxel-based spatial trajectory and their corresponding [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Relative Success Rate with Noisy Image Input. Both our model and the diffusion policy were trained on clean images and evaluated using noisy front and wrist image inputs with varying standard deviations (σ). The top figure presents the success rate relative to the result obtained with clean input. Example images of different noise level are shown at the bottom. tasks and the complete episode compared to the Diffusio… view at source ↗
Figures from the paper (2 more)
Figure 6
Figure 6. Figure 6: Task Randomization. Object positions are randomized across all trials. For banana tasks, object orientations are also randomized to increase variability and challenge policy robustness. TABLE III REAL-WORLD EXPERIMENT RESULTS. OUR METHOD CONSISTENTLY OUTPERFORMS THE BA…
Figure 5
Figure 5. Figure 5: Real-world experiment setup. We use a Franka robotic arm equipped with a parallel gripper for all manipulation tasks. Visual observa￾tions are captured using an Intel RealSense camera. 20 evaluations per task. All setups were manually verified to ensure consistent init…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 12 canonical work pages

  1. [1]

    What matters in learning from offline human demonstrations for robot manipula- tion,

    A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y . Zhu, and R. Mart´ın-Mart´ın, “What matters in learning from offline human demonstrations for robot manipula- tion,” arXiv preprint arXiv:2108.03298 , 2021

    arXiv 2021

  2. [2]

    From play to policy: Conditional behavior generation from uncurated robot data,

    Z. J. Cui, Y . Wang, N. M. M. Shafiullah, and L. Pinto, “From play to policy: Conditional behavior generation from uncurated robot data,” arXiv preprint arXiv:2210.10047 , 2022

    arXiv 2022

  3. [3]

    Self-supervised correspon- dence in visuomotor policy learning,

    P. Florence, L. Manuelli, and R. Tedrake, “Self-supervised correspon- dence in visuomotor policy learning,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 492–499, 2019

    work page 2019

  4. [4]

    Diffusion policy: Visuomotor policy learning via action diffusion,

    C. Chi, S. Feng, Y . Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” arXiv preprint arXiv:2303.04137 , 2023

    arXiv 2023

  5. [5]

    3d diffusion policy: Generalizable visuomotor policy learning via simple 3d rep- resentations,

    Y . Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu, “3d diffusion policy: Generalizable visuomotor policy learning via simple 3d rep- resentations,” in ICRA 2024 Workshop on 3D Visual Representations for Robot Manipulation , 2024

    work page 2024

  6. [6]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems , vol. 33, pp. 6840–6851, 2020

    2020

  7. [7]

    3d diffuser ac- tor: Policy diffusion with 3d scene representations,

    T.-W. Ke, N. Gkanatsios, and K. Fragkiadaki, “3d diffuser ac- tor: Policy diffusion with 3d scene representations,” arXiv preprint arXiv:2402.10885, 2024

    arXiv 2024

  8. [8]

    Language-guided object-centric diffusion policy for collision-aware robotic manipula- tion,

    H. Li, Q. Feng, Z. Zheng, J. Feng, and A. Knoll, “Language-guided object-centric diffusion policy for collision-aware robotic manipula- tion,” arXiv preprint arXiv:2407.00451 , 2024

    arXiv 2024

Show all 44 references
  1. [9]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  2. [10]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  3. [11]

    Palm: Scaling language modeling with pathways,

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al. , “Palm: Scaling language modeling with pathways,” Journal of Machine Learning Research , vol. 24, no. 240, pp. 1–113, 2023

    2023

  4. [12]

    Do as i can, not as i say: Grounding language in robotic affordances,

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman et al. , “Do as i can, not as i say: Grounding language in robotic affordances,” arXiv preprint arXiv:2204.01691, 2022

    2022 arXiv

  5. [13]

    Open-vocabulary queryable scene representations for real world planning,

    B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler, “Open-vocabulary queryable scene representations for real world planning,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 11 509–11 522

    2023

  6. [14]

    Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,

    W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,” in International conference on machine learning . PMLR, 2022, pp. 9118–9147

    2022

  7. [15]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,

    B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia, “Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 455–14 465

    2024

  8. [16]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v,

    J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao, “Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v,” arXiv preprint arXiv:2310.11441, 2023

    2023 arXiv

  9. [17]

    Code as policies: Language model programs for em- bodied control,

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for em- bodied control,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 9493–9500

    2023

  10. [18]

    V oxposer: Composable 3d value maps for robotic manipulation with language models,

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei, “V oxposer: Composable 3d value maps for robotic manipulation with language models,” arXiv preprint arXiv:2307.05973 , 2023

    2023 arXiv

  11. [19]

    Moka: Open-vocabulary robotic manipulation through mark-based visual prompting,

    F. Liu, K. Fang, P. Abbeel, and S. Levine, “Moka: Open-vocabulary robotic manipulation through mark-based visual prompting,” arXiv preprint arXiv:2403.03174, 2024

    2024 arXiv

  12. [20]

    Rlbench: The robot learning benchmark & learning environment,

    S. James, Z. Ma, D. R. Arrojo, and A. J. Davison, “Rlbench: The robot learning benchmark & learning environment,” IEEE Robotics and Automation Letters , vol. 5, no. 2, pp. 3019–3026, 2020

    2020

  13. [21]

    The colosseum: A benchmark for evaluating generalization for robotic manipulation,

    W. Pumacay, I. Singh, J. Duan, R. Krishna, J. Thomason, and D. Fox, “The colosseum: A benchmark for evaluating generalization for robotic manipulation,” arXiv preprint arXiv:2402.08191 , 2024

    2024 arXiv

  14. [22]

    Deep unsupervised learning using nonequilibrium thermodynamics,

    J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in International conference on machine learning . PMLR, 2015, pp. 2256–2265

    2015

  15. [23]

    Denoising diffusion implicit models,

    J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” arXiv preprint arXiv:2010.02502 , 2020

    2010 arXiv

  16. [24]

    Score-based generative modeling through stochastic differential equations,

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” arXiv preprint arXiv:2011.13456 , 2020

    2011 arXiv

  17. [25]

    High-resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 684–10 695

    2022

  18. [26]

    Scaling robot learning with semantically imagined experience,

    T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, J. Peralta, B. Ichter et al. , “Scaling robot learning with semantically imagined experience,” arXiv preprint arXiv:2302.11550 , 2023

    2023 arXiv

  19. [27]

    Cacti: A framework for scalable multi-task multi-scene visual imitation learning,

    Z. Mandi, H. Bharadhwaj, V . Moens, S. Song, A. Rajeswaran, and V . Kumar, “Cacti: A framework for scalable multi-task multi-scene visual imitation learning,” arXiv preprint arXiv:2212.05711 , 2022

    2022 arXiv

  20. [28]

    Genaug: Retargeting behaviors to unseen situations via generative augmentation,

    Z. Chen, S. Kiami, A. Gupta, and V . Kumar, “Genaug: Retargeting behaviors to unseen situations via generative augmentation,” arXiv preprint arXiv:2302.06671, 2023

    2023 arXiv

  21. [29]

    Dreamar- rangement: Learning language-conditioned robotic rearrangement of objects via denoising diffusion and vlm planner,

    W. Chen, C. Xiao, G. Gao, F. Sun, C. Zhang, and J. Zhang, “Dreamar- rangement: Learning language-conditioned robotic rearrangement of objects via denoising diffusion and vlm planner,” Authorea Preprints, 2024

    2024

  22. [30]

    Sculptdiff: Learning robotic clay sculpting from humans with goal conditioned diffusion policy,

    A. Bartsch, A. Car, C. Avra, and A. B. Farimani, “Sculptdiff: Learning robotic clay sculpting from humans with goal conditioned diffusion policy,” arXiv preprint arXiv:2403.10401 , 2024

    2024 arXiv

  23. [31]

    Hierarchical diffu- sion policy for kinematics-aware multi-task robotic manipulation,

    X. Ma, S. Patidar, I. Haughton, and S. James, “Hierarchical diffu- sion policy for kinematics-aware multi-task robotic manipulation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18 081–18 090

    2024

  24. [32]

    Robocat: A self- improving foundation agent for robotic manipulation,

    K. Bousmalis, G. Vezzani, D. Rao, C. Devin, A. X. Lee, M. Bauza, T. Davchev, Y . Zhou, A. Gupta, A. Raju et al. , “Robocat: A self- improving foundation agent for robotic manipulation,” arXiv preprint arXiv:2306.11706, 2023

    2023 arXiv

  25. [33]

    Learning latent plans from play,

    C. Lynch, M. Khansari, T. Xiao, V . Kumar, J. Tompson, S. Levine, and P. Sermanet, “Learning latent plans from play,” in Conference on robot learning. PMLR, 2020, pp. 1113–1132

    2020

  26. [34]

    Bc-z: Zero-shot task generalization with robotic imitation learning,

    E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, “Bc-z: Zero-shot task generalization with robotic imitation learning,” in Conference on Robot Learning. PMLR, 2022, pp. 991–1002

    2022

  27. [35]

    Rt-2: Vision- language-action models transfer web knowledge to robotic control,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choro- manski, T. Ding, D. Driess, A. Dubey, C. Finn et al. , “Rt-2: Vision- language-action models transfer web knowledge to robotic control,” arXiv preprint arXiv:2307.15818 , 2023

    2023 arXiv

  28. [36]

    Learning language-conditioned robot behavior from offline data and crowd- sourced annotation,

    S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn et al. , “Learning language-conditioned robot behavior from offline data and crowd- sourced annotation,” in Conference on Robot Learning . PMLR, 2022, pp. 1303–1315

    2022

  29. [37]

    Open-world object manipulation using pre-trained vision-language models,

    A. Stone, T. Xiao, Y . Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong, P. Wohlhart, S. Kirmani, B. Zitkovich, F. Xia et al. , “Open-world object manipulation using pre-trained vision-language models,” arXiv preprint arXiv:2303.00905, 2023

    2023 arXiv

  30. [38]

    Learning generalizable manipulation policies with object-centric 3d representations,

    Y . Zhu, Z. Jiang, P. Stone, and Y . Zhu, “Learning generalizable manipulation policies with object-centric 3d representations,” arXiv preprint arXiv:2310.14386, 2023

    2023 arXiv

  31. [39]

    Rt-trajectory: Robotic task generalization via hindsight trajectory sketches,

    J. Gu, S. Kirmani, P. Wohlhart, Y . Lu, M. G. Arenas, K. Rao, W. Yu, C. Fu, K. Gopalakrishnan, Z. Xu et al., “Rt-trajectory: Robotic task generalization via hindsight trajectory sketches,” arXiv preprint arXiv:2311.01977, 2023

    2023 arXiv

  32. [40]

    Rt-1: Robotics transformer for real-world control at scale,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu et al. , “Rt-1: Robotics transformer for real-world control at scale,” arXiv preprint arXiv:2212.06817, 2022

    2022 arXiv

  33. [41]

    A robotic skill learning system built upon diffusion policies and foundation models,

    N. Ingelhag, J. Munkeby, J. van Haastregt, A. Varava, M. C. Welle, and D. Kragic, “A robotic skill learning system built upon diffusion policies and foundation models,” arXiv preprint arXiv:2403.16730 , 2024

    2024 arXiv

  34. [42]

    Affordance- guided reinforcement learning via visual prompting,

    O. Y . Lee, A. Xie, K. Fang, K. Pertsch, and C. Finn, “Affordance- guided reinforcement learning via visual prompting,” arXiv preprint arXiv:2407.10341, 2024

    2024 arXiv

  35. [43]

    V-rep: A versatile and scalable robot simulation framework,

    E. Rohmer, S. P. Singh, and M. Freese, “V-rep: A versatile and scalable robot simulation framework,” in 2013 IEEE/RSJ international conference on intelligent robots and systems . IEEE, 2013, pp. 1321– 1326

    2013

  36. [44]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo et al. , “Segment anything,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 4015–4026

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.