REVIEW 4 major objections 6 minor 13 cited by
DexterityGen: Foundation Controller for Unprecedented Dexterity
T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read A pretrained generative controller turns unsafe hand commands into safe dexterous actions.
desk verdict Serious systems paper with a genuinely new integration of RL pretraining and diffusion guidance for dexterous teleoperation; headline stability numbers are from in-distribution simulation and the real-world tool use success is modest, but the core idea deserves peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery has three parts. First, the Anygrasp-to-Anygrasp pretraining pipeline uses RL to learn transitions between nearby grasp configurations on diverse objects, with random wrist poses and heavy domain randomization, then rolls the policies out to build a dataset of about $10^{10}$ transitions; a separate precision-rotation task adds fine-grained motions. Second, a diffusion model learns the conditional distribution of future finger keypoint offsets $\Delta x$ (positions of eight finger keypoints over a 0.2-second horizon in the hand frame) given proprioceptive history and a one-hot mode label, making keypoint motion the intermediate action representation. Third, during inference the external command enters only as gradient guidance, steering the denoised sample toward the user's commanded motion while staying inside the learned safe-action distribution, and an inverse-dynamics network maps the sampled keypoint motion to target joint positions for the 16-DOF hand.
What would settle it
Run the zero-shot teleoperation protocol on objects with contact properties far outside the pretraining range (for example, thin flexible strips, highly slippery or very heavy items) and measure time-to-fall; if holding time does not clearly exceed the raw-teleoperation baseline, the coverage premise collapses.
Extended reading notes
Core claim
On its own terms, the paper establishes that a conditional generative model $p_\theta(\Delta x \mid o)$, trained on $10^{10}$ simulation transitions collected from RL policies solving grasp-to-grasp and related in-hand tasks, acts as a foundation controller: at inference it samples finger keypoint motions from the learned distribution while using gradient guidance to keep the sample close to an external motion prompt, and an inverse-dynamics model turns the keypoint motion into joint-position commands. The central discovery is that this projection preserves the human's intention while rejecting unsafe behavior, so a human teleoperator who cannot reliably control contact-rich finger motions can nonetheless guide the hand through long-horizon tasks. The paper reports that this raises object-holding duration by 10-100x on noisy policies and, for the first time in its setup, enables dexterous tool use such as a syringe and screwdriver under teleoperation.
Load-bearing premise
The load-bearing premise is that the simulated transitions, despite coming from a limited set of random objects with randomized wrist poses, cover the hand-object states and command directions a human produces when teleoperating novel real objects like pens, syringes, and screwdrivers.
Editorial extensions
If this is right
- A single pretrained controller can serve as a reusable low-level dexterity layer for many downstream tasks, without task-specific rewards or real-world fine-tuning.
- Humans who cannot reliably teleoperate contact-rich finger motions can still complete long-horizon tasks, because dangerous commands are corrected while intended directions are preserved.
- The same guided-sampling interface can accept prompts from other high-level sources, such as a noisy RL policy, with stability gains of 10-100x in simulated evaluation.
- Stage-wise tool-use results indicate that the remaining bottleneck is chaining skills reliably, not the low-level stabilizer, so improving high-level stage policies should improve whole-task success.
Reading between the lines
- If the same keypoint-motion interface were kept, the controller could be retrained on a different robot hand or even a gripper, suggesting that the transferable unit may be the motion prior, not the embodiment.
- The 10-100x claim measures holding duration; a natural next test is whether the projection degrades precision tasks, where intention preservation is harder than safety, or image-based commands.
- A natural extension is to add a learned release or handover mode instead of disabling the controller when object placement is needed, which would make the controller cover open-loop phases of tool use.
- Since the dataset is proprioceptive only, adding simulated touch or vision tokens to the conditioning state could push the same architecture toward robust hand-eye coordination without changing the guided-sampling interface.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DexterityGen (DexGen), a generative controller trained on a large simulated dataset of dexterous in-hand manipulation trajectories generated by reinforcement learning. The controller uses a diffusion model over finger keypoint motions and an inverse dynamics model to produce joint commands. At inference, a coarse external motion command, e.g. from human teleoperation, is injected via gradient guidance so that samples from the learned action distribution approximately preserve the commanded motion while remaining in the learned safe distribution. The authors report simulated experiments in which DexGen improves the object-holding duration of noise-corrupted expert policies by 10-100x, and real-world teleoperation experiments on reorientation, functional grasping, regrasping, and long-horizon tasks such as screwdriver and syringe use. The central claim is that a single pretrained controller can serve as a general-purpose low-level dexterity layer that realizes input manipulation commands without task-specific rewards or real-world fine-tuning.
Significance. If the claims are established, the work would be a useful step toward reusable low-level dexterity controllers: it combines the scalability of simulation-based RL pretraining with the flexibility of human-provided coarse motion commands, and it does so with a generative model that can be prompted by continuous finger-level commands rather than discrete skill labels. The paper includes extensive simulation data collection, a concrete model architecture, real-hardware experiments, and an unusually candid limitations section. The main value is the demonstration that a learned conditional action distribution can act as a stabilizing filter for imperfect teleoperation. However, the load-bearing evidence for cross-task generality and for the headline 'unprecedented dexterity' is currently thin, for the reasons detailed in the major comments.
major comments (4)
- [Section IV-B with Section III-B] The simulated evaluation is in-distribution and does not substantiate the generalization claim. The noisy policies are corruptions of an Anygrasp-to-Anygrasp expert, and Anygrasp-to-Anygrasp is exactly the task used to generate the pretraining dataset in Section III-B. Uniform additive or multiplicative noise around the expert action remains largely inside the support of the learned conditional distribution, so the 10-100x duration improvement in Figure 8 chiefly measures the ability to denoise perturbations within the training distribution, not the ability to handle novel objects, novel commands, or out-of-support states. Please add held-out object and task evaluations, commands that are intentionally outside the training distribution, or a quantitative coverage diagnostic such as estimated log-likelihood of recorded real-world states under p_theta.
- [Tables I-II and Section IV-C] The real-world evidence for the headline claims is statistically thin. Only two users are reported, with no per-user breakdown, no confidence intervals, and no statistical test. The end-to-end tool-use success rates are 3/20 for the screwdriver and 4/20 for the syringe, with the largest drop in the final use stage. Section VI states that vision, which the controller does not use, is necessary for accurate tool motions, so the current data support a promising assistive effect on stage-wise skills rather than 'unprecedented' tool use or a general-purpose controller. Please report confidence intervals, per-user and per-object breakdowns, the raw holding-time values behind the normalized TTF metric, and a comparison with a reasonable shared-autonomy or wrist-stabilization baseline, or soften the claims accordingly.
- [Section VI, Limitations 1-3] The paper's own stated limitations directly undercut the central claim that DexGen is a safe, general-purpose controller that realizes input dexterous manipulation commands. The controller has no vision, relies on imperfect implicit touch sensing, is deployed zero-shot, and the authors acknowledge that real-world fine-tuning may be necessary. The method has no out-of-distribution detection or uncertainty mechanism: when a user command lies outside the support of p_theta, the gradient guidance in Section III-D will move toward the nearest learned mode or follow the command without any way to indicate that the request is unsafe. The authors should either provide evidence that real-world teleoperation states and commands fall in high-likelihood regions of the learned distribution, or introduce an explicit coverage/safety mechanism and temper the 'safe projection' claim.
- [Section III-D and Figure 8] The guidance strength alpha is a critical free parameter, and the paper's only sensitivity study is the simulation sweep in Figure 8, which is on the same Anygrasp task used for pretraining. The optimal alpha visibly varies with the noise level, and no selection procedure or real-world sensitivity analysis is reported for the results in Tables I and II. Since the behavior of the deployed system depends strongly on this parameter, please report how alpha was chosen for the real-world experiments and provide a robustness study over alpha, the reward weights in Eq. (4), and the threshold F_thresh in Algorithm 4.
minor comments (6)
- [Eq. (3)] The dimension of the input command Delta x_input is written as R^{K x 3}, but the predicted motion is a sequence of length T, so the command should be R^{T x K x 3} or the distance function should be defined accordingly.
- [Table I] The TTF metric is reported only as a percentage normalized by episode length, with no raw durations or episode length stated. Please give the actual episode length and raw holding times so the 10-100x stability claim can be compared against real-world numbers.
- [Figure 8] The legend and line styles are hard to parse: the figure shows 'Ours (Duration)', 'Ours (# Goals)', and two baseline curves, but the text only describes duration and goals in general. Please make clear which curves correspond to which quantity and whether the reported improvement is on the same trial set.
- [Section II] There is a typo: 'boardly' should be 'broadly'.
- [Algorithm 4] The grasp-analysis threshold F_thresh is never given a value or a sensitivity analysis; since it controls the grasp set used for pretraining coverage, please state the numerical value and the criterion used to choose it.
- [Section IV-C] The evaluation protocol describes 30 minutes of familiarization and two users, but it does not state the number of trials per user, whether task ordering was randomized, or whether the users were the authors. Please report these details.
Circularity Check
Simulation '10-100x' stability claim is an in-distribution denoising check on the same expert policy that generated the pretraining data; real-world results provide the external evidence.
-
fitted input called prediction
[Section III-B (dataset generation) and Section IV-B (simulated experiments)]
"After training, we use this anygrasp-to-anygrasp policy to rollout grasp transition sequences to cover all the possible hand-object interaction modes. ... We first test the capability of DexGen in assisting suboptimal policies in solving the Anygrasp-to-Anygrasp task in simulation. We simulate 2 kinds of suboptimal policies with an expert RL policy πexp. The first one is πnoisy(a|s) = πexp(a|s) + U(−α, α)"
Pretraining set D comes from rolling out the trained Anygrasp-to-Anygrasp RL policy (Algorithm 1: 'Collect training dataset D = ∪i Rollout(πi)'). The simulated evaluation takes the same expert πexp, corrupts it with uniform noise, and measures duration/goals on the exact Anygrasp-to-Anygrasp task. DexGen is trained to maximize likelihood/MSE on D's action distribution, so at states in the expert's rollout support, sampling from pθ reproduces the expert's stable behavior. The 10-100x duration improvement thus measures how well DexGen denoises/autoencodes its training distribution, not generalization to novel commands or objects. If pθ fits D, correcting a noisy version of D's generator is statistically forced.
full rationale
Score 4 reflects one partial circularity. The simulated '10-100x' stability result is an in-distribution denoising check: the expert policy corrupted in Section IV-B is the same policy whose rollouts form the pretraining dataset of Section III-B, so the improvement is expected once pθ fits that distribution. This does not, by itself, establish a foundation controller for unseen commands, objects, or embodiments. The real-world experiments are external and support the method, though tool-use success is modest and the paper's own limitations (Section VI) concede that vision and touch, absent from the controller, are needed for accurate tool motions. Self-citations to Yin et al. are background, not load-bearing; no uniqueness theorem or ansatz is smuggled in via citation. The 'safe means in-distribution' phrasing in Section III-D is an operational definition rather than a derived guarantee.
Assumptions & free parameters
free parameters (4)
- guidance strength alpha =
not reported (tuned)
- RL reward weights (w_goal, w_style, w_reg) =
not reported quantitatively
- grasp analysis threshold F_thresh =
not reported
- motion horizon T and keypoint count K =
T=2 (0.2s), K=8
assumptions (5)
- domain assumption IsaacGym simulator with domain randomization is an adequate proxy for real-world Allegro hand contact dynamics
- domain assumption The RL policies trained on Anygrasp-to-Anygrasp and auxiliary tasks generate a representative distribution of all useful in-hand manipulation behaviors
- domain assumption The inverse dynamics model can accurately map generated keypoint motions to executable joint commands
- ad hoc to paper Diffusion model guidance via gradient of a simple L2 distance is sufficient to align samples with external intent
- domain assumption Human teleoperation retargeting (confidential method) provides a usable coarse motion prompt
Cite this review
Pith. "Pith review of DexterityGen: Foundation Controller for Unprecedented Dexterity." pith.science (2026). https://pith.science/paper/ABSUDAB4
@misc{pith2026250204307,
author = {Pith},
title = {Pith review of: DexterityGen: Foundation Controller for Unprecedented Dexterity},
year = {2026},
howpublished = {\url{https://pith.science/paper/ABSUDAB4}},
note = {Machine review of arXiv:2502.04307}
}
read the original abstract
Teaching robots dexterous manipulation skills, such as tool use, presents a significant challenge. Current approaches can be broadly categorized into two strategies: human teleoperation (for imitation learning) and sim-to-real reinforcement learning. The first approach is difficult as it is hard for humans to produce safe and dexterous motions on a different embodiment without touch feedback. The second RL-based approach struggles with the domain gap and involves highly task-specific reward engineering on complex tasks. Our key insight is that RL is effective at learning low-level motion primitives, while humans excel at providing coarse motion commands for complex, long-horizon tasks. Therefore, the optimal solution might be a combination of both approaches. In this paper, we introduce DexterityGen (DexGen), which uses RL to pretrain large-scale dexterous motion primitives, such as in-hand rotation or translation. We then leverage this learned dataset to train a dexterous foundational controller. In the real world, we use human teleoperation as a prompt to the controller to produce highly dexterous behavior. We evaluate the effectiveness of DexGen in both simulation and real world, demonstrating that it is a general-purpose controller that can realize input dexterous manipulation commands and significantly improves stability by 10-100x measured as duration of holding objects across diverse tasks. Notably, with DexGen we demonstrate unprecedented dexterous skills including diverse object reorientation and dexterous tool use such as pen, syringe, and screwdriver for the first time.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 13 Pith papers
-
Dexonomy: Synthesizing All Dexterous Grasp Types in a Grasp Taxonomy
A two-stage optimization pipeline produces 9.5 million validated grasps across 31 GRASP taxonomy types and enables a type-conditional generative model to grasp novel objects from single-view point clouds with 82.3% re...
-
UniCross: Unified Cross-Skill Dexterous Manipulation Synthesis
Grasping, relocation, in-hand rotation, and translation share one relational state/action/reward structure, so ten expert policies distill into one cross-skill controller that chains and generalizes in simulation.
-
Towards Human-level Dexterous Teleoperation
A single-stage RL co-tracking controller trained on consecutive human-derived hand–object subgoals achieves ~75% real-robot success on long-horizon dexterous teleoperation where baselines fail.
-
Play2Perfect: What Matters in Dexterous Play Pretraining for Precise Assembly?
Task-agnostic RL play pretraining on diverse objects yields a reusable dexterous prior that makes sparse-reward assembly learning ~33× more sample-efficient and enables zero-shot sim-to-real transfer on tight insertio...
-
PTLD: Sim-to-real Privileged Tactile Latent Distillation for Dexterous Manipulation
PTLD distills real privileged tactile data into a state estimator to boost sim-to-real performance of proprioceptive dexterous manipulation policies, yielding 182% improvement on in-hand rotation and 57% on reorientat...
-
Scaling Cross-Embodiment World Models for Dexterous Manipulation
A single particle-based world model trained on many simulated robot hands and real human hands can plan dexterous manipulation on robot hands it never trained on.
-
DEXOP: A Device for Robotic Transfer of Dexterous Human Manipulation
DEXOP implements perioperation with a passive exoskeleton linked to a sensorized robot hand, and DEXOP-collected demonstrations train robot policies more efficiently per unit time than teleoperation.
-
LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrations
LodeStar combines automatic skill segmentation with simulation-based reinforcement learning augmentation and a learned routing transformer to let a robotic hand complete long-horizon dexterous tasks from a few human demos.
-
TypeTele: Releasing Dexterity in Teleoperation by Dexterous Manipulation Types
A type-guided teleoperation system that selects predefined dexterous hand poses with a language model outperforms retargeting-based teleoperation on nine real-world tasks and improves imitation learning success.
-
FlashBack: Consistency Model-Accelerated Shared Autonomy
Consistency model distillation enables one-step denoising of user actions for shared autonomy, achieving faster assistance than DDPM-based methods with comparable or better task success.
-
DexCtrl: Towards Sim-to-Real Dexterity with Adaptive Controller Learning
DexCtrl jointly predicts joint actions and adaptive PD controller gains from proprioceptive history, improving sim-to-real transfer for dexterous manipulation tasks.
-
The Latent Color Subspace: Emergent Order in High-Dimensional Chaos
FLUX.1’s VAE latent space contains an interpretable Hue–Saturation–Lightness structure that enables training-free color prediction and control via closed-form latent edits.
-
Going with the Flow: Koopman Behavioral Models as Pseudo Planners for Visuo-Motor Dexterity
A single learned linear Koopman model over coupled visual and proprioceptive states generates full-horizon dexterous manipulation plans and triggers replanning when its own visual predictions diverge from reality.
Reference graph
Works this paper leans on
-
[1]
Adap- tive virtual fixtures for machine-assisted teleoperation tasks
Daniel Aarno, Staffan Ekvall, and Danica Kragic. Adap- tive virtual fixtures for machine-assisted teleoperation tasks. In International Conference on Robotics and Automation (ICRA), 2005
work page 2005
-
[2]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023
arXiv 2023
-
[3]
Ananye Agarwal, Shagun Uppal, Kenneth Shaw, and Deepak Pathak. Dexterous functional grasping. In Conference on Robot Learning (CoRL) , 2023
work page 2023
-
[4]
Learning dexterous in-hand manipula- tion
OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pa- chocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipula- tion. The International Journal of Robotics Research , 39 (1):3–20, 2020
2020
-
[5]
Highly parallelized data-driven mpc for minimal in- tervention shared control
Alexander Broad, Todd Murphey, and Brenna Argall. Highly parallelized data-driven mpc for minimal in- tervention shared control. In Robotics: Science and Systems (RSS), 2019
work page 2019
-
[6]
Rt-1: Robotics transformer for real-world control at scale
Anthony Brohan, Noah Brown, Justice Carbajal, Yev- gen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022
arXiv 2022
-
[7]
Rt-2: Vision-language-action models transfer web knowledge to robotic control
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818 , 2023
arXiv 2023
-
[8]
Human-wheelchair collaboration through prediction of intention and adaptive assistance
Tom Carlson and Yiannis Demiris. Human-wheelchair collaboration through prediction of intention and adaptive assistance. In International Conference on Robotics and Automation (ICRA), 2008
work page 2008
Show all 65 references
-
[9]
Sequential dexterity: Chaining dexterous policies for long-horizon manipulation
Yuanpei Chen, Chen Wang, Li Fei-Fei, and C Karen Liu. Sequential dexterity: Chaining dexterous policies for long-horizon manipulation. In Conference on Robot Learning (CoRL), 2023
2023
-
[10]
Expressive whole-body control for humanoid robots
Xuxin Cheng, Yandong Ji, Junming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang. Expressive whole-body control for humanoid robots. In Robotics: Science and Systems (RSS), 2024
2024
-
[11]
Open-television: Teleoperation with immersive active visual feedback
Xuxin Cheng, Jialong Li, Shiqi Yang, Ge Yang, and Xiaolong Wang. Open-television: Teleoperation with immersive active visual feedback. In Conference on Robot Learning (CoRL) , 2024
2024
-
[12]
Bunny-visionpro: Real-time bimanual dexterous teleoperation for imitation learning
Runyu Ding, Yuzhe Qin, Jiyue Zhu, Chengzhe Jia, Shiqi Yang, Ruihan Yang, Xiaojuan Qi, and Xiaolong Wang. Bunny-visionpro: Real-time bimanual dexterous teleoperation for imitation learning. arXiv preprint arXiv:2407.03162, 2024
2024 arXiv
-
[13]
Learning universal policies via text-guided video generation
Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. In Neural Information Processing Systems (NeurIPS), 2024
2024
-
[14]
Ave: Assistance via empowerment
Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan. Ave: Assistance via empowerment. In Neural Information Processing Sys- tems (NeurIPS), 2020
2020
-
[15]
Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system
Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, and Dieter Fox. Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system. In International Conference on Robotics and Automation (ICRA) , 2020
2020
-
[16]
Dextreme: Transfer of agile in-hand manipulation from simulation to reality
Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In International ...
2023
-
[17]
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Neural Information Processing Systems (NeurIPS) , 2020
2020
-
[18]
Dynamic handover: Throw and catch with bi- manual hands
Binghao Huang, Yuanpei Chen, Tianyu Wang, Yuzhe Qin, Yaodong Yang, Nikolay Atanasov, and Xiaolong Wang. Dynamic handover: Throw and catch with bi- manual hands. In Conference on Robot Learning (CoRL), 2023
2023
-
[19]
Imitation learning: A survey of learning methods
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR) , 50 (2):1–35, 2017
2017
-
[20]
Planning with diffusion for flexible behavior synthesis
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning (ICML) , 2022
2022
-
[21]
Shared autonomy via hindsight optimization
Shervin Javdani, Siddhartha S Srinivasa, and J Andrew Bagnell. Shared autonomy via hindsight optimization. In Robotics: Science and Systems (RSS) , 2015
2015
-
[22]
Planning motions compliant to complex contact states
Xuerong Ji and Jing Xiao. Planning motions compliant to complex contact states. The International Journal of Robotics Research, 20(6):446–465, 2001
2001
-
[23]
Deft: Dexterous fine- tuning for real-world hand policies
Aditya Kannan, Kenneth Shaw, Shikhar Bahl, Pragna Mannam, and Deepak Pathak. Deft: Dexterous fine- tuning for real-world hand policies. In Conference on Robot Learning (CoRL) , 2023
2023
-
[24]
Sampling-based exploration for reinforcement learning of dexterous manipulation
Gagan Khandate, Siqi Shang, Eric T Chang, Tristan Luca Saidi, Yang Liu, Seth Matthew Dennis, Johnson Adams, and Matei Ciocarlie. Sampling-based exploration for reinforcement learning of dexterous manipulation. In Robotics: Science and Systems (RSS) , 2023
2023
-
[25]
Droid: A large-scale in-the-wild robot manipulation dataset
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ash- win Balakrishna, Sudeep Dasari, Siddharth Karam- cheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. In Robotics: Science an...
2024
-
[26]
Continuous shared control for stabilizing reaching and grasping with brain-machine in- terfaces
Hyun K Kim, J Biggs, W Schloerb, M Carmena, Mikhail A Lebedev, Miguel AL Nicolelis, and Man- dayam A Srinivasan. Continuous shared control for stabilizing reaching and grasping with brain-machine in- terfaces. IEEE Transactions on Biomedical Engineering , 53(6):1164–1173, 2006
2006
-
[27]
Open- vla: An open-source vision-language-action model
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Open- vla: An open-source vision-language-action model. In Conference on Robot Learning (CoRL) , 2024
2024
-
[28]
Adam: A method for stochastic optimization
Diederik P Kingma. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2014
2014
-
[29]
Segment anything
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In IEEE/CVF International Conference on Computer Vision (ICCV) , 2023
2023
-
[30]
Rapidly- exploring random trees: Progress and prospects
Steven M LaValle and James J Kuffner. Rapidly- exploring random trees: Progress and prospects. Al- gorithmic and Computational Robotics , pages 303–307, 2001
2001
-
[31]
Offline reinforcement learning: Tutorial, review, and perspectives on open problems
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020
2005 arXiv
-
[32]
Twisting lids off with two hands
Toru Lin, Zhao-Heng Yin, Haozhi Qi, Pieter Abbeel, and Jitendra Malik. Twisting lids off with two hands. In Conference on Robot Learning (CoRL) , 2024
2024
-
[33]
Learning visuotactile skills with two multifingered hands
Toru Lin, Yu Zhang, Qiyang Li, Haozhi Qi, Brent Yi, Sergey Levine, and Jitendra Malik. Learning visuotactile skills with two multifingered hands. In International Conference on Robotics and Automation (ICRA) , 2025
2025
-
[34]
Decoupled weight decay regularization
I Loshchilov. Decoupled weight decay regularization. In International Conference on Learning Representa- tions (ICLR), 2018
2018
-
[35]
Dextrah-g: Pixels- to-action dexterous arm-hand grasping with geometric fabrics
Tyler Ga Wei Lum, Martin Matak, Viktor Makoviy- chuk, Ankur Handa, Arthur Allshire, Tucker Hermans, Nathan D Ratliff, and Karl Van Wyk. Dextrah-g: Pixels- to-action dexterous arm-hand grasping with geometric fabrics. In Conference on Robot Learning (CoRL) , 2024
2024
-
[36]
Isaac gym: High performance gpu-based physics simulation for robot learning
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470, 2021
2021 arXiv
-
[37]
An overview of dexterous manipulation
Allison M Okamura, Niels Smaby, and Mark R Cutkosky. An overview of dexterous manipulation. In International Conference on Robotics and Automation (ICRA) , 2000
2000
-
[38]
Open x-embodiment: Robotic learning datasets and rt-x models
Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abra- ham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864 , 2023
-
[39]
Film: Visual reasoning with a general conditioning layer
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI conference on Artificial Intelligence (AAAI) , 2018
2018
-
[40]
In-hand object rotation via rapid motor adaptation
Haozhi Qi, Ashish Kumar, Roberto Calandra, Yi Ma, and Jitendra Malik. In-hand object rotation via rapid motor adaptation. In Conference on Robot Learning (CoRL) , 2023
2023
-
[41]
Anyteleop: A general vision-based dexterous robot arm- hand teleoperation system
Yuzhe Qin, Wei Yang, Binghao Huang, Karl Van Wyk, Hao Su, Xiaolong Wang, Yu-Wei Chao, and Dieter Fox. Anyteleop: A general vision-based dexterous robot arm- hand teleoperation system. In Robotics: Science and Systems (RSS), 2023
2023
-
[42]
Robot learning with sensorimotor pre-training
Ilija Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. Robot learning with sensorimotor pre-training. In Conference on Robot Learning (CoRL), 2023
2023
-
[43]
Shared autonomy via deep reinforcement learning
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Shared autonomy via deep reinforcement learning. In Robotics: Science and Systems (RSS) , 2018
2018
-
[44]
First contact: Unsupervised human-machine co-adaptation via mutual information maximization
Siddharth Reddy, Sergey Levine, and Anca Dragan. First contact: Unsupervised human-machine co-adaptation via mutual information maximization. In Neural Information Processing Systems (NeurIPS) , 2022
2022
-
[45]
U-net: Convolutional networks for biomedical image segmentation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medi- cal Image Computing and Computer-Assisted Interven- tion (MICCAI), 2015
2015
-
[46]
An autonomous robotic assistant for drinking
Sebastian Schr ¨oer, Ingo Killmann, Barbara Frank, Mar- tin V ¨olker, Lukas Fiederer, Tonio Ball, and Wolfram Burgard. An autonomous robotic assistant for drinking. In International Conference on Robotics and Automa- tion (ICRA), 2015
2015
-
[47]
Proximal policy optimization algorithms
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017
2017 arXiv
-
[48]
Parallel autonomy in automated vehicles: Safe motion generation with minimal intervention
Wilko Schwarting, Javier Alonso-Mora, Liam Pauli, Ser- tac Karaman, and Daniela Rus. Parallel autonomy in automated vehicles: Safe motion generation with minimal intervention. In International Conference on Robotics and Automation (ICRA) , 2017
2017
-
[49]
Leap hand: Low-cost, efficient, and anthropomorphic hand for robot learning
Kenneth Shaw, Ananye Agarwal, and Deepak Pathak. Leap hand: Low-cost, efficient, and anthropomorphic hand for robot learning. In Robotics: Science and Systems (RSS), 2023
2023
-
[50]
Bimanual dexterity for complex tasks
Kenneth Shaw, Yulong Li, Jiahui Yang, Mohan Kumar Srirama, Ray Liu, Haoyu Xiong, Russell Mendonca, and Deepak Pathak. Bimanual dexterity for complex tasks. In Conference on Robot Learning (CoRL) , 2024
2024
-
[51]
Learn- ing purely tactile in-hand manipulation with a torque- controlled hand
Leon Sievers, Johannes Pitz, and Berthold B ¨auml. Learn- ing purely tactile in-hand manipulation with a torque- controlled hand. In International Conference on Robotics and Automation (ICRA) , 2022
2022
-
[52]
Robotic telekinesis: Learning a robotic hand imitator by watching humans on youtube
Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak. Robotic telekinesis: Learning a robotic hand imitator by watching humans on youtube. In Robotics: Science and Systems (RSS), 2022
2022
-
[53]
De- noising diffusion implicit models
Jiaming Song, Chenlin Meng, and Stefano Ermon. De- noising diffusion implicit models. In International Con- ference on Learning Representations (ICLR) , 2020
2020
-
[54]
Octo: An open-source generalist robot policy
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. In Robotics: Science and Systems (RSS) , 2024
2024
-
[55]
Domain ran- domization for transferring deep neural networks from simulation to the real world
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain ran- domization for transferring deep neural networks from simulation to the real world. In International Conference on Intelligent Robots and Systems (IROS) , 2017
2017
-
[56]
Llama: Open and efficient foundation language models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Bap- tiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023
2023 arXiv
-
[57]
Trinkle and Richard P
Jeffrey C. Trinkle and Richard P. Paul. Planning for dexterous manipulation with sliding contacts. The Inter- national Journal of Robotics Research, 9(3):24–48, 1990
1990
-
[58]
Dexcap: Scalable and portable mocap data collection system for dexterous manipulation
Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, and C Karen Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation. In Robotics: Science and Systems (RSS) , 2024
2024
-
[59]
Lessons from learning to spin” pens”
Jun Wang, Ying Yuan, Haichuan Che, Haozhi Qi, Yi Ma, Jitendra Malik, and Xiaolong Wang. Lessons from learning to spin” pens”. In Conference on Robot Learn- ing (CoRL), 2024
2024
-
[60]
Anyrotate: Gravity- invariant in-hand object rotation with sim-to-real touch
Max Yang, Chenghua Lu, Alex Church, Yijiong Lin, Chris Ford, Haoran Li, Efi Psomopoulou, David AW Barton, and Nathan F Lepora. Anyrotate: Gravity- invariant in-hand object rotation with sim-to-real touch. In Conference on Robot Learning (CoRL) , 2024
2024
-
[61]
Offline imitation learning through graph search and retrieval
Zhao-Heng Yin and Pieter Abbeel. Offline imitation learning through graph search and retrieval. In Robotics: Science and Systems (RSS) , 2024
2024
-
[62]
Rotating without seeing: Towards in-hand dexterity through touch
Zhao-Heng Yin, Binghao Huang, Yuzhe Qin, Qifeng Chen, and Xiaolong Wang. Rotating without seeing: Towards in-hand dexterity through touch. In Robotics: Science and Systems (RSS) , 2023
2023
-
[63]
To the noise and back: Diffusion for shared autonomy
Takuma Yoneda, Luzhe Sun, Bradly Stadie, Matthew Walter, et al. To the noise and back: Diffusion for shared autonomy. In Robotics: Science and Systems (RSS), 2023
2023
-
[64]
Aloha unleashed: A simple recipe for robot dexterity
Tony Z Zhao, Jonathan Tompson, Danny Driess, Pete Florence, Kamyar Ghasemipour, Chelsea Finn, and Ayzaan Wahid. Aloha unleashed: A simple recipe for robot dexterity. In Conference on Robot Learn- ing (CoRL), 2024. APPENDIX A. DexGen Training Pipeline We provide an overview of ...
2024
-
[65]
10: Diffusion Model in DexGen Controller
We experimented with 8 and 12 DDIM steps during the State Encoder UNet Encoder UNet Encoder UNet Decoder UNet Decoder Noised Motion Robot State Diffusion Timestep Timestep Encoder Task/Mode Projection FiLM conditioning Noise Prediction Diffusion Model Embedding Fig. 10: Diffus...
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.