Pith. sign in

REVIEW 4 major objections 5 minor 48 references

Bridging Perception and Action: Spatially-Grounded Mid-Level Representations for Robot Generalization

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that attention-gated mid-level spatial representations (object boxes, motion traces, grasp plans, depth traces) raise bimanual dexterous success rates by 11–24 points over language and plain diffusion baselines.

desk verdict Promising multi-expert mid-level conditioning idea, but the headline numbers are not statistically anchored and the paper has internal sign and heading inconsistencies that need fixing. read the letter →

arxiv 2506.06196 v1 pith:HBYWHNRU submitted 2025-06-06 cs.RO

Jonathan Yang , Chuyuan Kelly Fu , Dhruv Shah , Dorsa Sadigh , Fei Xia , Tingnan Zhang This is my paper
classification cs.RO
keywords spatialgroundingmid-levelrepresentationsrobotgeneralizationdiffusionpolicymixture-of-expertsbimanualmanipulationdexterousweightedimitationlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that robot policies for bimanual, dexterous manipulation generalize better when they are conditioned on interpretable mid-level spatial representations — bounding boxes, trajectory traces, grasp plans, and depth-aware traces — than on language or on raw pixels alone. The authors build a diffusion policy whose decoder attends to four specialist encoders, each trained to emit one of these representations, and evaluate it on eight manipulation tasks in simulation and the real world. They report 69.6% average success across tasks, against 45.1% for a plain diffusion policy, 51.5% for a language-conditioned baseline, and 58% for a keypoints-conditioned baseline. A self-consistency weighting scheme that up-weights demonstrations whose trajectories already follow the mid-level plan adds about 10% more. If the reported numbers hold, the lesson is that no single grounding works for every task, but an attention-based combination of spatial groundings does.

What carries the argument

The carrying mechanism is an attention-based mixture-of-experts diffusion policy. Four frozen specialist encoders emit mid-level spatial representations — bounding boxes, trajectory traces, grasp plans, and depth-aware traces — and a multi-head attention block performs early gating over those embeddings, followed by cross-attention between the gated representation and the image embeddings, so the policy can decide at each state which representation to trust. The second mechanism is self-consistency weighting: a hand-defined adherence metric measures how closely a demonstrated trajectory follows each expert's prediction, and each demonstration is re-weighted by the negative exponential of that adherence error before behavioral cloning. This turns mid-level representations into a pseudo-reward, so the weighted imitation step sharpens the policy's sensitivity to the representations while the attention design preserves robustness to their noise.

What would settle it

Run the main real-world evaluation (or the simulation suite) with multiple seeds per method, for example 20 independent policy rollouts per task, and compare per-seed success-rate distributions; the central claim would be unsupported if the standard error across seeds is comparable to the reported 11–24 point gaps. Even simpler, a reader with access to the raw per-trial logs could compute exact binomial confidence intervals for 69.6% versus 45.1% and check whether the intervals overlap.

Watch

Extended reading notes

Core claim

The paper's central claim is that spatially grounded mid-level representations are a better bridge between perception and action for dexterous bimanual manipulation than either high-level language conditioning or no grounding. It claims that different tasks need different axes of grounding — motion-centric trajectory traces for insertion and cup stacking, object-centric bounding boxes for arranging fruit or kitchen objects, pose-aware grasp plans for assembly and shirt hanging, and depth-aware traces for handover and shoelace tying — and that no single representation wins everywhere. It then claims that an attention-based mixture-of-experts policy, which gates and cross-attends over all four specialist encoders, can harvest the task-specific benefits and beat every single-representation variant on average. Finally, it claims that training with self-consistency weights, defined by the adherence of each demonstration to the mid-level expert outputs, makes the policy follow its representations more precisely and raises success by about 10%. The central discovery, in the authors' framing, is that the sensitivity-robustness tradeoff is the right lens for choosing how to inject mid-level structure into a policy.

Load-bearing premise

The load-bearing premise is that the reported aggregate success rates — 69.6% versus 45.1%, 51.5%, 58%, and the 10% weighting gain — are stable, meaningful differences between methods, because the paper gives no trial counts, seeds, standard deviations, or confidence intervals for the main experiments.

Editorial extensions

If this is right

  • Across varied bimanual tasks, a policy conditioned on multiple spatial mid-level representations should beat any single-representation policy on average, because the best representation differs by task.
  • Language conditioning underperforms granular spatial grounding on dexterous bimanual tasks in this benchmark, despite language's flexibility.
  • Attention-based gating is more robust to noise in the representations than concatenation or early fusion, with reported robustness indices of 0.86 versus 0.75 and 0.80.
  • Without the attention architecture, self-consistency weighting can backfire: it lowers the robustness index enough that weighted concatenation drops 26% in average success.
  • The sensitivity score and robustness index are useful diagnostics: architectures and training schemes with lower sensitivity and higher robustness on these tasks also had higher average success.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that the four representation axes form a reusable checklist: future work could use the sensitivity score to automatically select which mid-level representation a new task needs, rather than training all four experts and testing.
  • The self-consistency weighting scheme is not tied to spatial representations in principle; a similar adherence metric for language or keypoint annotations could extend the same pseudo-reward weighting to those modalities, though the paper only demonstrates it for its four spatial experts.
  • A natural testable extension would be to distill the four asynchronous experts into a single fast policy after training, since the authors note that asynchronous expert execution is a limitation for high-frequency control and that smaller distilled versions of the experts lose some generality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes conditioning a diffusion policy on multiple spatially-grounded mid-level representations (bounding boxes, grasp plans, 2D and depth-aware trajectory traces) via a mixture-of-experts architecture with attention-based gating. The authors report real-world and simulation experiments on bimanual dexterous tasks, claim a 24% improvement over an ungrounded diffusion policy and an 11% improvement over a language-grounded baseline, and introduce a self-consistency weighting scheme for imitation learning. The core conceptual claim is that different mid-level representations are task-dependent and that a mixture-of-experts policy can combine them to improve generalization.

Significance. If the reported results are stable, the paper makes a useful empirical contribution by systematically comparing four mid-level spatial representations and showing that an attention-gated mixture of experts can combine their task-specific benefits. The two-stage design (off-the-shelf experts frozen, then policy trained) and the self-consistency weighting idea are practical and likely to be of interest to the robot learning community. The paper also ships a real-world evaluation and a zero-shot generalization appendix, which strengthens its scope. However, the quantitative support for the central claim is currently incomplete: the main experiments lack trial counts, seeds, and confidence intervals, and there is an internal contradiction in the definition of the training objective. These issues must be resolved before the empirical claims can be accepted.

major comments (4)
  1. [Section VI.C, Figures 7 and 9] The main experimental comparisons are reported without trial counts, number of seeds, standard deviations, or confidence intervals. The headline claims of 11%, 24%, and 10% improvements are therefore not anchored to any measure of run-to-run variability. If the unquantified variance is comparable to these gaps, the central claim is unsupported. The zero-shot appendix reports 20 trials per task, but the main tables and figures give no equivalent information; please provide per-task trial counts and at least standard deviations or confidence intervals for all reported averages.
  2. [Abstract vs. Section VII.B] The abstract states that the method achieves an average success rate '11% higher than a language-grounded baseline.' Section VII.B reports 69.6% for Mid-Level MoE and 51.5% for the language baseline, which is an 18.1-percentage-point difference, not 11%. The 11.6-point gap matches the keypoints baseline (58%), not the language baseline. Please clarify which comparison the abstract refers to, and ensure the abstract, introduction, and results section report the same numbers.
  3. [Algorithm 1 line 5 vs. Appendix C] Algorithm 1 defines the self-consistency weight as w_i = exp(-(1/|E|) Σ λ_E Adherence(E(s_i), τ)), which is positive and upweights consistent data. Appendix C defines w(s,a) = -exp(-(1/|E|) Σ λ_E Adherence(E(s), τ)), which is negative. Since the policy is trained by minimizing the weighted BC loss L_BC = (1/B) Σ w_i ℓ(π_θ(s_i), a_i), negative weights turn the minimization into maximization of the BC loss. The statement in Appendix C that the sign is 'up to a constant sign' is incorrect: a constant sign flips the gradient direction and changes the optimization entirely. Please specify the exact objective and correct the inconsistency.
  4. [Section VII.D heading and text] The heading 'Self-Consistency Leads to Higher Sensitivity Scores' conflicts with the body text, which reports that weighted imitation learning gives a 25.5% lower average sensitivity score than unweighted training. Table II also shows lower sensitivity scores for all weighted architectures. Lower sensitivity is defined in Section IV as closer adherence to the representations, so the heading appears to state the opposite of the reported result. Please align the heading, text, and table interpretation.
minor comments (5)
  1. [Section V.C] The sentence 'The exact method to compute the weights can be viewed in Appendix??.' contains an unresolved placeholder reference; please replace it with the correct appendix citation.
  2. [Section VII.C] The caption for Table III says 'Table III records the average success rates...' and then 'Meanwhile, Table III records the sensitivity scores...'; the second reference should be Table II. This makes the table's intended content ambiguous.
  3. [Appendix A] The description of 'Bounding Box 3D' states that it is a set of normalized coordinates 'in image space' but includes a z-coordinate; please clarify the coordinate frame (image-plus-depth or camera frame) to avoid confusion.
  4. [Section V.C] The analogy between self-consistency weights and an advantage function is informal; the pseudo-reward is derived from the same demonstrations used for behavioral cloning, so the paper should explicitly discuss what prevents this weighting from simply memorizing idiosyncrasies of the demonstration set.
  5. [Appendix B] The Sensitivity Score definition in Section IV states that 'A lower SS indicates that the policy closely follows the representations,' but the Bounding Box adherence metric uses distance to the four corners only, which may not capture whether the trajectory actually passes through the box; this is a minor definitional point but worth clarifying.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical comparisons against external baselines and a training-loss design; no derivation reduces by construction to its own inputs.

full rationale

The paper is an empirical robotics study rather than a derivation, and the central claims (Mid-Level MoE vs a no-representation diffusion policy, a language-grounded RT-H baseline, a MOKA-style keypoints baseline, and weighted vs unweighted training) are evaluated on external tasks with measured success rates. The headline improvements therefore do not reduce by construction to the paper's inputs. The self-consistency weighting in Algorithm 1 computes weights from adherence between demonstrations and mid-level representations and then reweights the same demonstrations in a behavioral-cloning loss; this is a standard weighted-imitation training signal rather than a fitted parameter renamed as a prediction, and the reported 10% gain is an external performance measurement, not a quantity forced by the weight definition. The lambda values in Table IV are hand-set with an ablation in Figure 10, which is hyperparameter selection and not circularity, and the Limitations section explicitly acknowledges that the adherence metrics are hand-designed. Self-citations appear (e.g., RT-H [2], SpatialVLM [5], RoboCLIP [35]) but only as baselines or related work; no load-bearing claim is justified by an unverified self-citation or by an imported uniqueness theorem. Separately, and outside the circularity category, Appendix C defines w(s,a) with a negative sign that conflicts with the positive weights in Algorithm 1, and the statement that this is equivalent 'up to a constant sign' is mathematically incorrect for a minimization objective; Section VI.C also omits trial counts, seeds, and confidence intervals, and the abstract's '11% higher than a language-grounded baseline' does not match the paper's own 69.6% vs 51.5% point estimates. These are correctness and statistical-reporting concerns, not circularity, so the circularity score remains 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities. Its central empirical claim rests on hand-tuned lambda weights, accurate calibration, reliable expert encoders, and the assumption that the evaluation numbers are stable. The self-consistency reweighting is the most fragile modeling choice because it uses the same demonstrations for both the weights and the BC loss.

free parameters (5)
  • lambda_BoundingBox_2D = 4.0
    Hand-chosen weight in self-consistency pseudo-reward (Appendix C, Table IV); directly shapes the weighted-IL training signal and the reported 10% gain.
  • lambda_GraspPlan = 1.0
    Hand-chosen weight for grasp-plan adherence in the same pseudo-reward; affects the weighted-IL ablation results.
  • lambda_TrajectoryTrace_2D = 2.0
    Hand-chosen weight for trajectory-trace adherence in self-consistency weighting.
  • lambda_TrajectoryTrace_Depth = 2.0
    Hand-chosen weight for depth-aware trace adherence in self-consistency weighting.
  • perturbation_sigma = 0.1
    Gaussian noise standard deviation used to compute the Robustness Index (Section IV, Table II); no sensitivity analysis on this choice is reported.
assumptions (5)
  • domain assumption Camera calibration (intrinsics/extrinsics, AprilTag) is accurate enough that projected trajectory and object representations are correct.
    Invoked in Section V.B for real-world trajectory projection; if wrong, all mid-level expert outputs are corrupted.
  • domain assumption Distilled ResNet34 expert encoders recover the relabeled mid-level outputs with sufficient accuracy and speed when run asynchronously at inference.
    Section V.B states experts are frozen, distilled, and executed asynchronously; the policy depends on these outputs for grounding.
  • domain assumption Relabeled 'ground-truth' trajectory traces estimated from proprioception and object pose are unbiased proxies for true future trajectories.
    Table I labels these as ground truth but they are model-based estimates; biased estimates affect the representation comparison.
  • domain assumption Measured success-rate differences are not within run-to-run variance.
    No trial counts, seeds, or error bars are reported for the main experiments (Section VI.C, Figures 7 and 9).
  • ad hoc to paper Self-consistency weights can be treated as an advantage-like training signal despite being computed from the same demonstrations used for behavioral cloning.
    Section V.C and Appendix C; this is the load-bearing premise of the 10% weighted-IL claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bridging Perception and Action: Spatially-Grounded Mid-Level Representations for Robot Generalization." pith.science (2026). https://pith.science/paper/HBYWHNRU

@misc{pith2026250606196,
  author       = {Pith},
  title        = {Pith review of: Bridging Perception and Action: Spatially-Grounded Mid-Level Representations for Robot Generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HBYWHNRU}},
  note         = {Machine review of arXiv:2506.06196}
}
read the original abstract

In this work, we investigate how spatially grounded auxiliary representations can provide both broad, high-level grounding as well as direct, actionable information to improve policy learning performance and generalization for dexterous tasks. We study these mid-level representations across three critical dimensions: object-centricity, pose-awareness, and depth-awareness. We use these interpretable mid-level representations to train specialist encoders via supervised learning, then feed them as inputs to a diffusion policy to solve dexterous bimanual manipulation tasks in the real world. We propose a novel mixture-of-experts policy architecture that combines multiple specialized expert models, each trained on a distinct mid-level representation, to improve policy generalization. This method achieves an average success rate that is 11% higher than a language-grounded baseline and 24 percent higher than a standard diffusion policy baseline on our evaluation tasks. Furthermore, we find that leveraging mid-level representations as supervision signals for policy actions within a weighted imitation learning algorithm improves the precision with which the policy follows these representations, yielding an additional performance increase of 10%. Our findings highlight the importance of grounding robot policies not only with broad perceptual tasks but also with more granular, actionable representations. For further information and videos, please visit https://mid-level-moe.github.io.

Figures

Figures reproduced from arXiv: 2506.06196 by the authors.

Figure 1
Figure 1. Bimanual, dexterous manipulation requires task-specific grounding. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. The sensitivity-robustness tradeoff. Policies need to follow their mid-level representations while being robust in erroneous noise to these representations. spatial grounding for robot policies. In addition, 1) Object-Centric Representations: These experts focus on extracting information pertinent to each object in the scene, such as object poses, sizes, shapes, and potential interaction points (e.g., grasp points, … view at source ↗
Figure 4
Figure 4. Policy Architecture. Four images are passed into a transformer encoder. In addition, an image is fed into each individual mid-level expert. The results embeddings are passed into the transformer decoder through cross-attention. we have separate object-centric, motion-centric, pose￾aware, and depth-aware representations. These corre￾spond to bounding boxes, trajectory traces, grasp plans, and depth-aware traces. 2) A… view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Self-Consistency. On the left image, the robot’s achieved trajectory doesn’t match its mid-level representation, which leads to a lower weight. On the right, the robot follows its representation, leading to a higher weight. signals serve as mid-level “expert outputs” t…
Figure 6
Figure 6. Figure 6: Simulation Tasks. arm, and a block containing a hole with another. Then, it must align the peg with the block and insert it into the hole. • FMB Assembly: The robot must pick up a multiple blocks and place it into its appropriate slots. Each of the blocks must be place…
Figure 7
Figure 7. Figure 7: Simulation Results. Mid-level MoE achieves a 24% higher success rate over a standard diffusion policy baseline. It performs consistently well over different tasks by leveraging different representations [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Real-World Results. There are clear differences in the benefits that different representations provide for tasks in the real world. such as trajectory traces, outperform object-centric representa￾tions (94% average success rate vs 85%. Meanwhile, in FMB, pose-aware rep…
Figure 9
Figure 9. Figure 9: Architecture and Self-Consistency Ablation. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Self-Consistency Weighting Strategies. Weighting demon￾strations by scale leads to a higher improvement in success rate over setting λ equal to 1. Under this choice, we have R(s, a) = − 1 |E| X E∈E λE Adherence E(s), τ  , and by definition w(s, a) = − exp − 1 |E| X …
Figure 11
Figure 11. Figure 11: Comprehensive Simulation Results. F. Ablation: Image-Space versus Lower-Level Embedding We find that explicitly attending to lower-level embeddings tends to perform slightly better for our task than image-space embebddings. The following table records our results: Tas…
Figure 12
Figure 12. Figure 12: Comprehensive Real-World Results [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Zero-Shot Generalization. (Left) Kitchen Stack with two unseen cups. (Right) Pen Handover using an unseen marker. a consistent stacking trajectory, allowing the policy to adapt its grasp point dynamically. • Pen Handover: Despite the marker’s body being 30 % thicker t…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 12 canonical work pages

  1. [1]

    Do as i can, not as i say: Grounding language in robotic affordances, 2022

    Michael Ahn et al. Do as i can, not as i say: Grounding language in robotic affordances, 2022. URL https://arxiv. org/abs/2204.01691

    arXiv 2022

  2. [2]

    Rt-h: Action hier- archies using language, 2024

    Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, De- bidatta Dwibedi, and Dorsa Sadigh. Rt-h: Action hier- archies using language, 2024. URL https://arxiv.org/abs/ 2403.01823

    arXiv 2024

  3. [3]

    Zero-shot robotic manipulation with pretrained image-editing diffusion models, 2023

    Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models, 2023. URL https://arxiv. org/abs/2310.10639

    arXiv 2023

  4. [4]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control, 2023

    Anthony Brohan et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control, 2023. URL https://arxiv.org/abs/2307.15818

    arXiv 2023

  5. [5]

    Spatialvlm: Endowing vision- language models with spatial reasoning capabilities,

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision- language models with spatial reasoning capabilities,

    work page

  6. [6]

    Towards human-level bimanual dexterous manipulation with reinforcement learning, 2022

    Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuang Jiang, Stephen Marcus McAleer, Yiran Geng, Hao Dong, Zongqing Lu, Song-Chun Zhu, and Yaodong Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning, 2022. URL https://arxiv.org/abs/2206.08686

    arXiv 2022

  7. [7]

    Open x- embodiment: Robotic learning datasets and rt-x models,

    Open X-Embodiment Collaboration et al. Open x- embodiment: Robotic learning datasets and rt-x models,

    work page

  8. [8]

    Robonet: Large-scale multi-robot learning, 2020

    Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning, 2020. URL https://arxiv.org/abs/ 1910.11215

    arXiv 2020

Show all 48 references
  1. [9]

    URL https://arxiv.org/abs/2310.08864

    arXiv

  2. [10]

    Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint...

    2023 arXiv

  3. [11]

    Goal-conditioned imitation learning,

    Yiming Ding, Carlos Florensa, Mariano Phielipp, and Pieter Abbeel. Goal-conditioned imitation learning,

  4. [12]

    Bridge data: Boosting generalization of robotic skills with cross- domain datasets, 2021

    Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Dani- ilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross- domain datasets, 2021. URL https://arxiv.org/abs/2109. 13396

    2021

  5. [13]

    Mooney, and Scott Niekum

    Prasoon Goyal, Raymond J. Mooney, and Scott Niekum. Zero-shot task adaptation using natural language, 2021. URL https://arxiv.org/abs/2106.02972

    2021 arXiv

  6. [14]

    Tenenbaum, Dale Schuurmans, and Pieter Abbeel

    Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation, 2023. URL https://arxiv.org/abs/2302. 00111

    2023

  7. [15]

    Scaling up and distilling down: Language-guided robot skill acquisition,

    Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-guided robot skill acquisition,

  8. [16]

    Hierarchical few- shot imitation with skill transition models, 2022

    Kourosh Hakhamaneshi, Ruihan Zhao, Albert Zhan, Pieter Abbeel, and Michael Laskin. Hierarchical few- shot imitation with skill transition models, 2022. URL https://arxiv.org/abs/2107.08981

    2022 arXiv

  9. [17]

    Rt-trajectory: Robotic task generalization via hindsight trajectory sketches,

    Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montserrat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, Priya Sundaresan, Peng Xu, Hao Su, Karol Hausman, Chelsea Finn, Quan Vuong, and Ted Xiao. Rt-trajectory: Robotic task generaliza...

  10. [18]

    Inner monologue: Embodied reasoning through planning with language models, 2022

    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning th...

    2022 arXiv

  11. [19]

    Davi- son

    Stephen James, Michael Bloesch, and Andrew J. Davi- son. Task-embedded control networks for few-shot imitation learning, 2018. URL https://arxiv.org/abs/1810. 03237

    2018

  12. [20]

    URL https://arxiv.org/abs/2307.14535

    arXiv

  13. [21]

    Egomimic: Scaling imitation learning via egocentric video, 2024

    Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoffman, and Danfei Xu. Egomimic: Scaling imitation learning via egocentric video, 2024. URL https://arxiv.org/abs/2410. 24221

    2024

  14. [22]

    Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,

  15. [23]

    Openvla: An open-source vision-language-action model,

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla...

  16. [24]

    Code as policies: Language model programs for embod- ied control, 2023

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embod- ied control, 2023. URL https://arxiv.org/abs/2209.07753

    2023 arXiv

  17. [25]

    Moka: Open-world robotic manipulation through mark-based visual prompting, 2024

    Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-world robotic manipulation through mark-based visual prompting, 2024. URL https://arxiv. org/abs/2403.03174

    2024 arXiv

  18. [26]

    Bc-z: Zero-shot task generalization with robotic imitation learning, 2022

    Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning, 2022. URL https://arxiv.org/abs/2202. 02005

    2022

  19. [27]

    Learning latent plans from play

    Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Ku- mar, Jonathan Tompson, Sergey Levine, and Pierre Ser- manet. Learning latent plans from play. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Proceedings of the Conference on Robot Learning, vol- ume 100 ofP...

    2020

  20. [28]

    Droid: A large-scale in-the- wild robot manipulation dataset, 2024

    Alexander Khazatsky et al. Droid: A large-scale in-the- wild robot manipulation dataset, 2024. URL https://arxiv. org/abs/2403.12945

    2024 arXiv

  21. [29]

    Visual reinforcement learning with imagined goals, 2018

    Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals, 2018. URL https://arxiv. org/abs/1807.04742

    2018 arXiv

  22. [30]

    URL https://arxiv.org/abs/2406.09246

    arXiv

  23. [31]

    Llarva: Vision-action instruction tuning enhances robot learning, 2024

    Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, and Roei Herzig. Llarva: Vision-action instruction tuning enhances robot learning, 2024. URL https://arxiv.org/ abs/2406.11815

    2024 arXiv

  24. [32]

    Ashutosh Saxena, Justin Driemeyer, Justin Kearns, and A. Ng. Robotic grasping of novel objects. InNeural Information Processing Systems, 2006. URL https://api. semanticscholar.org/CorpusID:8682350

    2006

  25. [33]

    Rdt-1b: a diffusion foundation model for bimanual manipulation, 2024

    Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation, 2024. URL https://arxiv.org/abs/ 2410.07864

    2024 arXiv

  26. [34]

    Skill induction and planning with latent language, 2022

    Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language, 2022. URL https://arxiv.org/abs/2110.01517

    2022 arXiv

  27. [35]

    Dex-net 2.0: Deep learning to plan ro- bust grasps with synthetic point clouds and analytic grasp metrics, 2017

    Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea, and Ken Goldberg. Dex-net 2.0: Deep learning to plan ro- bust grasps with synthetic point clouds and analytic grasp metrics, 2017. URL https://arxiv.org/abs/1703.09312

    2017 arXiv

  28. [36]

    Kite: Keypoint-conditioned policies for semantic manipulation, 2023

    Priya Sundaresan, Suneel Belkhale, Dorsa Sadigh, and Jeannette Bohg. Kite: Keypoint-conditioned policies for semantic manipulation, 2023. URL https://arxiv.org/abs/ 2306.16605

    2023 arXiv

  29. [37]

    Robocasa: Large-scale simulation of everyday tasks for generalist robots, 2024

    Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Man- dlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots, 2024. URL https://arxiv.org/abs/2406.02523

    2024 arXiv

  30. [38]

    Robotic control via embodied chain-of-thought reasoning, 2024

    Michał Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thought reasoning, 2024. URL https://arxiv.org/abs/2407.08693

    2024 arXiv

  31. [39]

    Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn

    Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manip- ulation with low-cost hardware, 2023. URL https://arxiv. org/abs/2304.13705

    2023 arXiv

  32. [40]

    Discovering motor programs by re- composing demonstrations

    Tanmay Shankar, Shubham Tulsiani, Lerrel Pinto, and Abhinav Gupta. Discovering motor programs by re- composing demonstrations. InProceedings of (ICLR) International Conference on Learning Representations, April 2020

    2020

  33. [41]

    left", "right

    Hongkuan Zhou, Xiangtong Yao, Oier Mees, Yuan Meng, Ted Xiao, Yonatan Bisk, Jean Oh, Edward Johns, Mohit Shridhar, Dhruv Shah, Jesse Thomason, Kai Huang, Joyce Chai, Zhenshan Bing, and Alois Knoll. Bridging language and action: A survey of language-conditioned robot manipulati...

    2024 arXiv

  34. [42]

    Sumedh A Sontakke, Jesse Zhang, Sébastien M. R. Arnold, Karl Pertsch, Erdem Bıyık, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. Roboclip: One demon- stration is enough to learn robot policies, 2023. URL https://arxiv.org/abs/2310.07899

    2023 arXiv

  35. [44]

    Robust imitation of diverse behaviors, 2017

    Ziyu Wang, Josh Merel, Scott Reed, Greg Wayne, Nando de Freitas, and Nicolas Heess. Robust imitation of diverse behaviors, 2017. URL https://arxiv.org/abs/1707. 02747

    2017

  36. [47]

    Zhao, Jonathan Tompson, Danny Driess, Pete Florence, Kamyar Ghasemipour, Chelsea Finn, and Ayzaan Wahid

    Tony Z. Zhao, Jonathan Tompson, Danny Driess, Pete Florence, Kamyar Ghasemipour, Chelsea Finn, and Ayzaan Wahid. Aloha unleashed: A simple recipe for robot dexterity, 2024. URL https://arxiv.org/abs/2410. 13126

    2024

  37. [2020]

    URL https://arxiv.org/abs/1906.05838

    1906 arXiv

  38. [2022]

    URL https://arxiv.org/abs/2201.07207

    arXiv

  39. [2023]

    URL https://arxiv.org/abs/2311.01977

    arXiv

  40. [2024]

    URL https://arxiv.org/abs/2401.12168

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.