Pith. sign in

REVIEW 4 major objections 6 minor 91 references

A single diffusion transformer, trained on RoboTokens, can generate complete robot designs, predict their dynamics, and control them, optimizing for unseen trajectories and rewards zero-shot.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 01:26 UTC pith:ZEQPHG3R

load-bearing objection A single DiT over variable-length RoboTokens is a genuinely new way to unify robot design generation, dynamics, and control—but the abstract's blanket claim of beating CMA-ES fails in the quadruped space, and the 'zero-shot' guidance is hand-tuned per reward. the 4 major comments →

arxiv 2607.25798 v1 pith:ZEQPHG3R submitted 2026-07-28 cs.RO

Transformer Transformer: A Unified Model for Motion-Conditioned Robot Co-design

Huy Ha , C. Karen Liu , Shuran Song This is my paper
classification cs.RO
keywords robot co-designmotion-conditioned generationdiffusion transformerRoboTokencross-embodiment controlDynamics Self-Guidanceembodiment optimizationzero-shot reward optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether the best robot embodiment for a manipulation task can be found by a single learned model rather than by searching over designs with a simulator. It proposes Transformer Transformer, a diffusion transformer that operates on RoboTokens—a unified tokenization of a robot's links, joints, motors, states, and actions—and learns to model their joint distribution. The central claim is that one reward-agnostic dynamics model can, at inference time, be steered by gradients of any user-specified reward to generate high-value robot designs zero-shot, while also serving as a cross-embodiment controller to validate those designs. If true, this collapses co-design into a single GPU-parallelizable diffusion pass, with rewards comparable to or better than the evolutionary baseline CMA-ES while being orders of magnitude faster, plus a 73% tracking-error reduction on a fabricated ALOHA arm.

Core claim

The paper's central claim is that a single DiT trained on RoboTokens unifies robot embodiment generation, dynamics prediction, and control. Given target end-effector trajectories, the model diffuses complete robot embodiments together with their state and action trajectories; because the predictions are reward-agnostic, user-defined rewards can be evaluated on them and their gradients backpropagated through the denoiser to steer embodiment diffusion—a procedure the authors call Dynamics Self-Guidance. They show this zero-shot optimizer matches or outperforms CMA-ES across fixed-base, quadruped, and bimanual mobile design spaces, and that the same model, when conditioned on an embodiment, act

What carries the argument

RoboToken: a tokenization scheme that represents any rigid articulated robot as a variable-length sequence of continuous vectors for links, joints, motors, states, and actions, with IDs encoding connectivity and timesteps, and canonicalized transforms. Transformer Transformer is a diffusion transformer (DiT) trained to denoise these tokens under different masking schemes, so one model can generate embodiments (motion-to-robot) and predict actions (cross-embodiment control). Dynamics Self-Guidance converts the model's reward-agnostic state/action predictions into reward-specific value predictions by backpropagating user-reward gradients through the denoiser at each diffusion step (classifier-

Load-bearing premise

The load-bearing premise is that the model's predicted states and actions for a given embodiment and trajectory are faithful enough that rewards computed on those predictions rank designs the way the real simulator and controller would, and that reward gradients backpropagated through the denoiser point toward designs that also score well when actually simulated.

What would settle it

On a held-out set of optimized designs, compute both the model's predicted rewards and the rewards from the real simulator/controller (e.g., the RL experts or Mink). The central claim fails if the rank correlation between predicted and actual rewards is not significantly positive across all three design spaces, or if designs selected by Dynamics Self-Guidance score no better on average than random designs when evaluated with the oracle controller.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Robot co-design can be done with one GPU-parallelized diffusion model instead of a separate generator, critic, controller, and evolutionary search; reported optimization times drop from minutes/hours to under a second to tens of seconds.
  • Unseen reward functions and unseen target trajectories can be optimized zero-shot at inference time, because the model is trained reward-agnostically and rewards are injected only as gradients during sampling.
  • The same model can act as a cross-embodiment whole-body controller, enabling generated designs to be validated directly without retraining per embodiment; on bimanual robots, the learned controller's tracking reward aligns closely with an oracle controller (Fig. 11), and on quadrupeds the correlation is positive (r = 0.53, Fig. 8).
  • Generated designs inherit diffusion properties: they lie on the training manifold, respect cross-attribute coherence (mass, inertia, motor sizing), and improve with test-time compute, providing a natural test-time scaling curve.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The method's success hinges on how faithfully the model's predicted dynamics reproduce the simulator; Figure 7 already shows noticeably lower predicted-vs-actual reward correlation for the quadruped space, suggesting that off-distribution designs—especially those involving learned whole-body controllers and contacts—are where the approach is most likely to over-optimize to the model.
  • The 8-timestep context (§2.2, §9.2) is a surprisingly aggressive compression of long-horizon dynamics; it implies the model can reason about an episode from a handful of representative instants, a claim that could be tested directly by ablating context length to see when predicted rewards diverge from simulated ones.
  • The authors suggest evolutionary algorithms could use the diffusion model as a high-quality sampler at inference time; a natural extension is to interleave Dynamics Self-Guidance with an outer evolutionary loop, which might fix the local-minima issue the paper observes in single-sample guidance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes Transformer Transformer, a diffusion-transformer-based generative model over RoboTokens, a tokenization of robot embodiments, states, and actions. The same architecture is trained with masked modeling to perform motion-to-robot generation (diffusing embodiment and dynamics tokens conditioned on target end-effector trajectories) and cross-embodiment control. At inference, reward-agnostic dynamics predictions are converted into reward predictions, and their gradients steer the embodiment diffusion process ('Dynamics Self-Guidance'). The authors evaluate on three procedurally generated design spaces (fixed-base ViperX, quadruped manipulator, mobile bimanual), compare against random, CMA-ES, and zeroth-order diffusion sampling, and validate a fabricated ALOHA design in cloth-flinging experiments. The main claims are zero-shot optimization of unseen rewards/trajectories, improved performance and runtime over the evolutionary baseline, and unification of generator/critic/controller in one model.

Significance. If the claims held, the contribution would be significant: a single learned dynamics model that can generate, evaluate, and control robot designs would remove the need for separate learned generator, critic, controller, and evolutionary search, and the GPU-parallelized non-autoregressive evaluation could offer large speedups. The RoboToken representation is thoughtful, and the authors provide large-scale datasets, three diverse design spaces, and real-hardware validation (73% tracking-error reduction on ALOHA). However, the evaluation as presented does not fully support the strongest claims: the evolutionary baseline outperforms the proposed method in the quadruped single-trajectory settings, the 'zero-shot' optimization relies on per-space/per-reward tuned guidance scales, and no experiment trains a single model jointly on multiple embodiment spaces. These gaps are fixable but currently limit the paper's conclusions.

major comments (4)
  1. [Abstract; Tables 4, 8, 11] The abstract states that the method improves 'performance and runtime over the evolutionary baseline.' The runtime advantage is clear, but the performance claim is contradicted by the quadruped single-trajectory results. In Table 4 (Tracking Only), CMA-ES achieves position error 1.9 cm and 100% survival, while Zeroth and DGS achieve 2.6 cm and 99.4%; Table 8 (Tracking Torque) shows 2.0 cm vs 2.5 cm; Table 11 (Tracking Velocity) shows 2.1 cm vs 2.8/3.5 cm and lower velocity penalty. Thus the blanket performance claim is not supported in one of the three design spaces. The conclusion should be qualified or additional evidence supplied.
  2. [§1 Contributions; §3 Approaches] The paper claims a 'single model to span diverse robot embodiment spaces' and a 'one-stop shop' unifying embodiment generation, control, and evaluation. However, §3 states 'we train a model for each design space, and on only the motion-to-robot task.' No experiment trains one set of weights on multiple embodiment spaces; the cross-embodiment controller experiment (§7.1) is trained and evaluated on the bimanual space only. The 11-robot tokenization demonstrates representation generality, but not a single model spanning multiple spaces. The 'unified' claim is therefore an architectural/tokenization claim, not an empirical one, and should be stated as such or supported by a joint-training experiment.
  3. [§2.3; Fig. 7; Tables 4, 8, 11] Dynamics Self-Guidance backpropagates gradients of reward predictions through the denoiser. This is only sound if predicted rewards rank designs the way the simulator does. The paper's own Fig. 7 caption acknowledges that predicted-actual reward correlation is lower for Quadruped, and Tables 4, 8, and 11 show DGS fails to improve over the Zeroth-Order baseline in that space (and both underperform CMA-ES). No quantitative correlation values are reported, and no analysis of off-distribution prediction error is provided. Moreover, the guidance scale is tuned per design space and per reward (§9.4: 50/100/0.2/500; §8.4-8.5: alpha_size=0.005, alpha_mass=0.005 for guidance vs 0.1/10 for the actual reward), and eta=1.0 is used only for DGS. This means the 'zero-shot' claim is underspecified for an unseen reward: the user must choose a guidance scale and stochasticity. Please provide quantitative
  4. [§3; Fig. 8; Table 1] The claim that the model can 'faithfully self-validate' designs is weaker than suggested. Fig. 8 reports Pearson r=0.53, which is moderate correlation, with no confidence intervals or per-space breakdown; §7.1 shows the learned controller still trails the oracle Mink controller (Table 1: 5.9 cm vs 4.8 cm position error; reward 957 vs 1064). Since the main co-design results use oracle RL experts/Mink for validation rather than the model's own controller, the 'self-validation' capability is not actually exercised in the reported optimization pipeline. This should be presented as a promising initial result with appropriate caveats, not as an established replacement for the expert controllers.
minor comments (6)
  1. [§8.4, §8.5] The symbols alpha_size and alpha_mass are used for both the actual reward weight and the guidance-scale weight, with different values; this is confusing and should be renamed or clearly distinguished.
  2. [Fig. 7] Axis labels, units, and correlation coefficients are not stated in the main text. Please add quantitative values, not just the qualitative 'higher/lower' description.
  3. [§2.4, §3] Data counts are inconsistent across sections: the main text says 76 trajectories with 56/20 train/validation split, but bimanual experiments use 26 UMI dishwashing trajectories. Clarify which trajectories are used for each design space and how the split is applied to the bimanual space.
  4. [§3, 'Diffusion Composition'] The statement that composed diffusion generates designs 'favorable over 20/26 trajs' lacks a definition of favorable. Specify the threshold or metric used to count a trajectory as favorable.
  5. [Table 11] DGS has higher position error than Zeroth in Table 11 (3.5 vs 2.8 cm). If DGS is claimed to improve each individual sample, this result needs an explanation or a caveat.
  6. [§9.3.2, Fig. 5] The runtime comparison excludes the substantial one-time cost of training 128 RL experts (16 hours each on an A100) and collecting 1.3M quadruped episodes. While reporting optimization time is standard, the paper should explicitly acknowledge this exclusion and its implications for adopting the method on a new design space.

Circularity Check

0 steps flagged

No significant circularity; central claims are validated against external MuJoCo benchmarks.

full rationale

Transformer Transformer's derivation chain is not circular. The model is trained with a dynamics objective on RoboTokens (state/action prediction), not on the rewards used at inference; rewards are computed post hoc from the model's predicted tokens, and the same reward definitions are then evaluated in MuJoCo with independent controllers (Mink or RL experts), so the reported optimized-design rewards are not fitted to the model's own output. Dynamics Self-Guidance backpropagates reward gradients through the denoiser, but this is standard classifier-guided DDIM and the validation is external. The per-space/per-reward guidance scales (e.g., 50/100/0.2, alpha_size=0.005) are chosen to keep samples on-manifold/valid, and although they are hyperparameters that must be set for a new reward, this is an underspecification/robustness concern rather than a reduction of the prediction to a fit. Self-citations ([13], [23], [31], [49], [51]) are used for data collection, the WBC formulation, and related-work contrast; none is invoked as a uniqueness theorem or as the source of the central prediction. The paper's own admission that the quadruped space has weaker predicted-vs-actual reward correlation (Fig. 7) and that DGS underperforms CMA-ES on some quadruped metrics (Tables 4/8/11) is a correctness/scope caveat, not a circularity. The 'generated designs lie on the training manifold' section explicitly attributes the behavior to standard diffusion theory and labels it 'by construction,' so it is not a concealed prediction derived from itself.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The central claim rests on learned-dynamics fidelity, user-defined reward scope, and simulator/controller ground truth, plus a hand-tuned guidance protocol. No new physical entities are introduced; RoboToken is a representation, and Dynamics Self-Guidance is an algorithm.

free parameters (4)
  • Dynamics Self-Guidance scale = 50 / 100 / 0.2 per design space; 500 for multi-trajectory ViperX
    Hand-tuned as 'highest guidance scale that does not push the diffusion model out of distribution' (§9.4); directly determines how strongly reward gradients alter generated designs, so DGS comparisons depend on it.
  • Guidance vs reward weight for size/mass = α=0.005 for guidance, α=0.1 for reported reward
    After invalid outputs were observed, the guidance weight was set 20× lower than the reward weight (§8.4/§8.5). This post-hoc calibration changes which designs DGS selects.
  • Reward definition constants = σ_p=0.01, σ_o=0.5, α_torque=5e-5, α_velocity=0.1/0.5/0.005, α_size=0.1, α_mass=10, targets per space
    Defined by authors per design space; the paper's headline numbers are the optima under these hand-set objectives (§8).
  • DDIM stochasticity η = 1.0 for DGS, 0.0 otherwise
    Stochastic sampling was found to significantly improve DGS but not Zeroth-Order (§9.4); another hand-set knob affecting measured performance.
axioms (6)
  • domain assumption Learned diffusion predictions are faithful enough for reward evaluation and gradient guidance.
    §2.3 uses classifier-guided DDIM; Fig. 7 shows lower predicted-actual reward correlation for Quadruped, so this is a nontrivial load-bearing premise.
  • domain assumption End-effector trajectories plus differentiable rewards over embodiment/states/actions are a sufficient task specification.
    Scope paragraph in §2; excludes structural strength, appearance, tactile/sensor design.
  • domain assumption MuJoCo simulation with Mink/RL experts is valid ground truth for design quality.
    §2.4 and §3 evaluate all optimized designs with these controllers and one hardware build; if simulation is wrong, rankings change.
  • ad hoc to paper Uniform density plus parallel-axis splitting of lumped inertias preserves physical fidelity.
    §9.1 'Inertia Splitting' assumes uniform density per link and distributes lumped inertia; real links with mixed materials/voids violate this.
  • domain assumption SE(2) planar augmentation is sufficient because gravity breaks SE(3) symmetry.
    §2.2; humanoid/biped designs may need more than planar augmentations, but those are not central design spaces here.
  • domain assumption Action labels from tracking-optimized experts define the dynamics distribution, so the model is 'reward-agnostic' only up to the expert policy bias.
    §9.3.2 trains RL experts on tracking reward; action supervision is unperturbed expert actions, so predicted actions inherit tracking bias.

pith-pipeline@v1.3.0-alltime-deepseek · 20653 in / 17182 out tokens · 167704 ms · 2026-08-01T01:26:22.850594+00:00 · methodology

0 comments
read the original abstract

An often overlooked factor of robot manipulation performance is the embodiment of the robot itself. Motivated by this problem, we study motion-conditioned robot co-design, where the goal is to generate complete robot designs that track target end-effector trajectories (from human demonstrations) while optimizing user-defined rewards. We introduce Transformer Transformer, a diffusion transformer trained on RoboTokens, a unified tokenization of robot embodiments, states, and actions. The same architecture can be used across embodiment spaces (e.g., wheeled bimanual, quadrupeds, humanoids) and use cases (embodiment generation, cross embodiment controller). Rather than overfitting to one reward function, Transformer Transformer is a dynamics model, whose reward-agnostic state and action predictions can be converted into reward-specific value predictions. These value predictions are used to steer embodiment diffusion towards high value robot designs, through a procedure we call Dynamics Self-Guidance. Experiments across multiple design spaces show zero-shot optimization of unseen rewards and trajectories, improving performance and runtime over the evolutionary baseline. Finally, we fabricated an optimized ALOHA design, which reduced tracking error by over 70% compared to the original design.

Figures

Figures reproduced from arXiv: 2607.25798 by C. Karen Liu, Huy Ha, Shuran Song.

Figure 1
Figure 1. Figure 1: Demonstrate, Generate, Validate. From target end-effector motions and unseen reward functions, Transformer Transformer generates optimized robot designs that maximize rewards, then validates generated robots by controlling them to track the given motions. Trained over robot embodiment, state, and action tokens, it unifies embodiment optimization and cross-embodiment control into one model, providing a one-… view at source ↗
Figure 2
Figure 2. Figure 2: A Unified Robot Representation. RoboTo￾ken is designed to be a complete, learning-ready token representation of rigid articulated robots. Our tokenizer can convert any robot description into a sequence of RoboTokens, consisting of time-invariant embodiment tokens and time-varying dynamics tokens. Overview: The overarching goal of Trans￾former Transformer is to model the joint dis￾tribution of robots’ embod… view at source ↗
Figure 3
Figure 3. Figure 3: RoboToken Diffusion. Transformer Transformer diffuses noise into RoboTokens (to￾ken count in blue), 27-110× more compact than MJCF text tokens (token count in grey). Flexible. To encompass diverse robot kinemat￾ics, RoboToken supports a variable number of em￾bodiment tokens representing arbitrary connectivity. Here, joints point to the two links they connect via two link IDs, while motors point to the join… view at source ↗
Figure 4
Figure 4. Figure 4: Unified Architecture for Robot Co-Design. Transformer Transformer is a diffusion transformer, trained to jointly diffuse embodiment and dynamics RoboTokens, where different conditioning in the same model enables (a) motion-to-robot optimization and (b) cross-embodiment control. At inference time, the model uses unseen rewards to turn its embodiment and dynamics predictions into a reward prediction, and inc… view at source ↗
Figure 5
Figure 5. Figure 5: Better Designs with Test-Time Scaling. Across three design spaces and all rewards (§ 8), both our Zeroth Order and Dynamics Self-Guidance tend to generate higher-reward designs when allowed more parallel samples. Our model also zero-shot optimizes multiple trajectories (b,d).Our approach is significantly faster than CMA-ES, achieving similar or better performance while taking orders of magnitude less time.… view at source ↗
Figure 6
Figure 6. Figure 6: Diffusion Generates Diverse Designs. Our model diffuses embodiments with discrete (e.g., leg designs, degrees of freedom) and continuous (e.g., arm length, mounting points) variations that maximizes the specified reward functions. Changing the target trajectory (e.g., “Floor Scrub” → “Dynamic Toss”) or the reward (e.g., “size” penalization) changes the optimization landscape, leading to different embodimen… view at source ↗
Figure 7
Figure 7. Figure 7: Predicted Reward Accuracy. Learning legged robot dynamics means reasoning about how learned whole-body controllers navigate discontinu￾ous contacts and termination risks from falling over. Thus, we observe higher correlation between predicted and actual rewards for ViperX and Bimanual than Quadruped design spaces. Diffusion Composition for Multi-Trajectory Optimization. We apply diffusion compo￾sition [46,… view at source ↗
Figure 8
Figure 8. Figure 8: Whole-body Control of Generated Quadrupeds. After gen￾erating optimized robots, the same model can directly control its designs. Guidance Substitutes for Search. Dynamics Self-Guidance improves each individual sample, whereas Zeroth Order im￾proves by selecting the best of many, making the two substi￾tutes rather than complements. Their gap is therefore largest where search has least to work with: with a s… view at source ↗
Figure 9
Figure 9. Figure 9: Dynamic Cloth Unfolding. To optimize ALOHA for high speed flings, our model found bet￾ter mounting point and link dimensions, reducing tracking error by 73% and max joint speed by 30%. Real-world design validation. We fabricated an ALOHA setup [60, 61] optimized by our model for the “Tracking Velocity” reward ( [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Real World Unfolding Sequence. Compared to the original ALOHA design (a), we observe a lower max joint speed and less peaks in our model’s optimized design (b). 7 Additional Experiments In the context of robot co-design, a learned cross-embodiment controller is practically useful only for validating robot embodiments that benefit from costly learned controllers, such as legged robots. However, robots with… view at source ↗
Figure 11
Figure 11. Figure 11: Bimanual Mobile Control Valida￾tion. For each embodiment, we control it using our cross-embodiment controllers (small/large) and the oracle Mink [27] controller. Our con￾trollers have strong correlation with the oracle’s performance. In other words, using our cross￾embodiment controllers to evaluate robot designs is faithful to the oracle robot evaluation. 0 1 4 16 64 256 Optimize Time (s) 700 800 900 100… view at source ↗
Figure 12
Figure 12. Figure 12: Model Capacity Ablation. Al￾though larger models perform better, their inference time also takes longer. dataset simultaneously, and we repeat for 9 inference seeds. Unlike in the main paper, we use our learned controller (§ 7.1) instead of the oracle controller. We focus these results on the “Tracking Only” reward. Results. From [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

91 extracted references · 32 linked inside Pith

  1. [1]

    C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. InProceedings of Robotics: Science and Systems (RSS), 2024

    2024

  2. [2]

    Z. Liu, C. Chi, E. Cousineau, N. Kuppuswamy, B. Burchfiel, and S. Song. Maniwav: Learn- ing robot manipulation from in-the-wild audio-visual data.arXiv preprint arXiv:2406.19464, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    F. Lin, Y . Hu, P. Sheng, C. Wen, J. You, and Y . Gao. Data scaling laws in imitation learning for robotic manipulation.arXiv preprint arXiv:2410.18647, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Z. Wu, T. Wang, C. Guan, Z. Jia, S. Liang, H. Song, D. Qu, D. Wang, Z. Wang, N. Cao, et al. Fast-umi: A scalable and hardware-independent universal manipulation interface.arXiv preprint arXiv:2409.19499, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    M. Seo, H. A. Park, S. Yuan, Y . Zhu, , and L. Sentis. Legato: Cross-embodiment imitation using a grasping tool.IEEE Robotics and Automation Letters (RA-L), 2025

    2025

  6. [6]

    F. Liu, C. Li, Y . Qin, A. Shaw, J. Xu, P. Abbeel, and R. Chen. Vitamin: Learning contact-rich tasks through robot-free visuo-tactile manipulation interface.arXiv preprint arXiv:2504.06156, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    M. Xu, H. Zhang, Y . Hou, Z. Xu, L. Fan, M. Veloso, and S. Song. Dexumi: Using hu- man hand as the universal manipulation interface for dexterous manipulation.arXiv preprint arXiv:2505.21864, 2025

    arXiv 2025

  8. [8]

    T. Tao, M. K. Srirama, J. J. Liu, K. Shaw, and D. Pathak. Dexwild: Dexterous human interac- tions for in-the-wild robot policies.Robotics: Science and Systems (RSS), 2025

    2025

  9. [9]

    X. Zhu, B. Huang, and Y . Li. Touch in the wild: Learning fine-grained manipulation with a portable visuo-tactile gripper.arXiv preprint arXiv:2507.15062, 2025

    arXiv 2025

  10. [10]

    G. Lee, Y . Lee, K. Kim, S. Lee, S. Noh, S. Back, and K. Lee. Manipforce: Force-guided policy learning with frequency-aware representation for contact-rich manipulation.arXiv preprint arXiv:2509.19047, 2025

    arXiv 2025

  11. [11]

    Rayyan, J

    O. Rayyan, J. Abanes, M. Hafez, A. Tzes, and F. Abu-Dakka. Mv-umi: A scalable multi-view interface for cross-embodiment learning.arXiv preprint arXiv:2509.18757, 2025

    arXiv 2025

  12. [12]

    Y . Xu, L. Wei, P. An, Q. Zhang, and Y .-L. Li. exumi: Extensible robot teaching system with action-aware task-agnostic tactile representation. InConference on Robot Learning, pages 2536–2554. PMLR, 2025

    2025

  13. [13]

    Gupta, X

    H. Gupta, X. Guo, H. Ha, C. Pan, M. Cao, D. Lee, S. Sherer, S. Song, and G. Shi. Umi-on- air: Embodiment-aware guidance for embodiment-agnostic visuomotor policies, 2025. URL https://arxiv.org/abs/2510.02614

    Pith/arXiv arXiv 2025

  14. [14]

    Peebles and S

    W. Peebles and S. Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  15. [15]

    Furuta, Y

    H. Furuta, Y . Iwasawa, Y . Matsuo, and S. S. Gu. A system for morphology-task generalization via unified representation and behavior distillation.arXiv preprint arXiv:2211.14296, 2022

    Pith/arXiv arXiv 2022

  16. [16]

    Schaff, D

    C. Schaff, D. Yunis, A. Chakrabarti, and M. R. Walter. Jointly learning to construct and control agents using deep reinforcement learning. In2019 international conference on robotics and automation (ICRA), pages 9798–9805. IEEE, 2019

    2019

  17. [17]

    Kurin, M

    V . Kurin, M. Igl, T. Rockt¨aschel, W. Boehmer, and S. Whiteson. My body is a cage: the role of morphology in graph-based incompatible control.arXiv preprint arXiv:2010.01856, 2020. 10

    Pith/arXiv arXiv 2010

  18. [18]

    Gupta, L

    A. Gupta, L. Fan, S. Ganguli, and L. Fei-Fei. Metamorph: Learning universal controllers with transformers.arXiv preprint arXiv:2203.11931, 2022

    Pith/arXiv arXiv 2022

  19. [19]

    H. Lu, Z. Wu, J. Xing, J. Li, R. Li, Z. Li, and Y . Shi. Bodygen: Advancing towards efficient embodiment co-design.arXiv preprint arXiv:2503.00533, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Patel and S

    A. Patel and S. Song. Get-zero: Graph embodiment transformer for zero-shot embodiment generalization. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 14262–14269. IEEE, 2025

    2025

  21. [21]

    T. Wang, Y . Zhou, S. Fidler, and J. Ba. Neural graph evolution: Towards efficient automatic robot design.arXiv preprint arXiv:1906.05370, 2019

    Pith/arXiv arXiv 1906

  22. [22]

    J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. InInternational Con- ference on Learning Representations

  23. [23]

    X. Xu, H. Ha, and S. Song. Dynamics-guided diffusion model for robot manipulator design. arXiv preprint arXiv:2402.15038, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    T.-H. J. Wang, J. Zheng, P. Ma, Y . Du, B. Kim, A. Spielberg, J. Tenenbaum, C. Gan, and D. Rus. Diffusebot: Breeding soft robots with physics-augmented generative diffusion models. Advances in Neural Information Processing Systems, 36:44398–44423, 2023

    2023

  25. [25]

    Dhariwal and A

    P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis.Advances in neural information processing systems, 34:8780–8794, 2021

    2021

  26. [26]

    H. Choi, Y . Hou, C. Pan, S. Hong, A. Patel, X. Xu, M. R. Cutkosky, and S. Song. In-the-wild compliant manipulation with umi-ft.arXiv preprint arXiv:2601.09988, 2026

    arXiv 2026

  27. [27]

    K. Zakka. Mink: Python inverse kinematics based on MuJoCo, Dec. 2025. URLhttps: //github.com/kevinzakka/mink

    2025

  28. [28]

    Todorov, T

    E. Todorov, T. Erez, and Y . Tassa. Mujoco: A physics engine for model-based control. In2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012

    2012

  29. [29]

    R. S. Sutton, A. G. Barto, et al.Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  30. [30]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  31. [31]

    H. Ha, Y . Gao, Z. Fu, J. Tan, and S. Song. UMI on legs: Making manipulation policies mobile with manipulation-centric whole-body controllers. InProceedings of the 2024 Conference on Robot Learning, 2024

    2024

  32. [32]

    Z. Fu, X. Cheng, and D. Pathak. Deep whole-body control: learning a unified policy for manipulation and locomotion. InConference on Robot Learning, pages 138–149. PMLR, 2023

    2023

  33. [33]

    M. Ji, X. Peng, F. Liu, J. Li, G. Yang, X. Cheng, and X. Wang. Exbody2: Advanced expressive humanoid whole-body control.arXiv preprint arXiv:2412.13196, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    M. Liu, Z. Chen, X. Cheng, Y . Ji, R.-Z. Qiu, R. Yang, and X. Wang. Visual whole-body control for legged loco-manipulation.arXiv preprint arXiv:2403.16967, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Portela, A

    T. Portela, A. Cramariuc, M. Mittal, and M. Hutter. Whole-body end-effector pose tracking. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 11205– 11211. IEEE, 2025

    2025

  36. [36]

    Portela, G

    T. Portela, G. B. Margolis, Y . Ji, and P. Agrawal. Learning force control for legged manip- ulation. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 15366–15372. IEEE, 2024

    2024

  37. [37]

    H. Xue, C. Pan, Z. Yi, G. Qu, and G. Shi. Full-order sampling-based mpc for torque-level locomotion control via diffusion-style annealing. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 4974–4981. IEEE, 2025

    2025

  38. [38]

    Zakka, Y

    K. Zakka, Y . Tassa, and MuJoCo Menagerie Contributors. MuJoCo Menagerie: A collec- tion of high-quality simulation models for MuJoCo, 2022. URLhttp://github.com/ google-deepmind/mujoco_menagerie. 11

    2022

  39. [39]

    Song and S

    Y . Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019

    2019

  40. [40]

    C. He, X. Liu, G. M. S. Camps, J. Bruno, G. A. Sartoretti, and M. Schwager. Demystifying robot diffusion policies: Action memorization and a simple lookup table alternative. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  41. [41]

    Hansen and A

    N. Hansen and A. Ostermeier. Completely derandomized self-adaptation in evolution strate- gies.Evolutionary computation, 9(2):159–195, 2001

    2001

  42. [42]

    Hansen, Y

    N. Hansen, Y . Akimoto, and P. Baudis. CMA-ES/pycma on Github. Zen- odo, DOI:10.5281/zenodo.2559634, Feb. 2019. URLhttps://doi.org/10.5281/ zenodo.2559634

    doi:10.5281/zenodo.2559634 2019

  43. [43]

    T. Chen, Z. He, and M. Ciocarlie. Hardware as policy: Mechanical and computational co- optimization using deep reinforcement learning.arXiv preprint arXiv:2008.04460, 2020

    Pith/arXiv arXiv 2008

  44. [44]

    J. Xu, T. Chen, L. Zlokapa, M. Foshey, W. Matusik, S. Sueda, and P. Agrawal. An end-to-end differentiable framework for contact-aware robot design.arXiv preprint arXiv:2107.07501, 2021

    Pith/arXiv arXiv 2021

  45. [45]

    Rajani, K

    C. Rajani, K. Arndt, D. Blanco-Mulero, K. S. Luck, and V . Kyrki. Co-imitation: learning design and behaviour by imitation. InProceedings of the AAAI Conference on Artificial Intel- ligence, volume 37, pages 6200–6208, 2023

    2023

  46. [46]

    N. Liu, S. Li, Y . Du, A. Torralba, and J. B. Tenenbaum. Compositional visual generation with composable diffusion models. InEuropean conference on computer vision, pages 423–439. Springer, 2022

    2022

  47. [47]

    Y . Du, C. Durkan, R. Strudel, J. B. Tenenbaum, S. Dieleman, R. Fergus, J. Sohl-Dickstein, A. Doucet, and W. S. Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. InInternational conference on machine learning, pages 8489–8510. PMLR, 2023

    2023

  48. [48]

    Kodnongbua, I

    M. Kodnongbua, I. G. Y . Lou, J. Lipton, and A. Schulz. Computational design of passive grippers.arXiv preprint arXiv:2306.03174, 2023

    Pith/arXiv arXiv 2023

  49. [49]

    H. Ha, S. Agrawal, and S. Song. Fit2form: 3d generative model for robot gripper form design. InConference on Robot Learning, pages 176–187. PMLR, 2021

    2021

  50. [50]

    M. Li, R. Antonova, D. Sadigh, and J. Bohg. Learning tool morphology for contact-rich manipulation tasks with differentiable simulation.arXiv preprint arXiv:2211.02201, 2022

    Pith/arXiv arXiv 2022

  51. [51]

    R. Liu, J. Liang, S. Sudhakar, H. Ha, C. Chi, S. Song, and C. V ondrick. Paperbot: Learning to design real-world tools using paper, 2024

    2024

  52. [52]

    K. R. Allen, T. Lopez-Guevara, K. Stachenfeld, A. Sanchez-Gonzalez, P. Battaglia, J. Ham- rick, and T. Pfaff. Physical design using differentiable learned simulators.arXiv preprint arXiv:2202.00728, 2022

    Pith/arXiv arXiv 2022

  53. [53]

    Khazatsky, K

    A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y . J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y . Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. ...

    2024

  54. [54]

    Learning to reason with LLMs, Sept

    OpenAI. Learning to reason with LLMs, Sept. 2024. URLhttps://openai.com/ index/learning-to-reason-with-llms/

    2024

  55. [55]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 12

    Pith/arXiv arXiv 2025

  56. [56]

    Comanici, E

    G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: Pushing the frontier with advanced rea- soning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  57. [57]

    Brown, J

    B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V . Le, C. R ´e, and A. Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787, 2024

    Pith/arXiv arXiv 2024

  58. [58]

    Ballon, A

    M. Ballon, A. Algaba, and V . Ginis. The relationship between reasoning and performance in large language models–o3 (mini) thinks harder, not longer.arXiv preprint arXiv:2502.15631, 2025

    Pith/arXiv arXiv 2025

  59. [59]

    Y . Wu, Z. Sun, S. Li, S. Welleck, and Y . Yang. Inference scaling laws: An empirical analy- sis of compute-optimal inference for problem-solving with language models.arXiv preprint arXiv:2408.00724, 2024

    Pith/arXiv arXiv 2024

  60. [60]

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware.arXiv preprint arXiv:2304.13705, 2023

    Pith/arXiv arXiv 2023

  61. [61]

    T. Z. Zhao, J. Tompson, D. Driess, P. Florence, K. Ghasemipour, C. Finn, and A. Wahid. Aloha unleashed: A simple recipe for robot dexterity.arXiv preprint arXiv:2410.13126, 2024

    Pith/arXiv arXiv 2024

  62. [62]

    Ha and S

    H. Ha and S. Song. Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding. InConference on Robot Learning, pages 24–33. PMLR, 2022

    2022

  63. [63]

    S. Ha, S. Coros, A. Alspach, J. M. Bern, J. Kim, and K. Yamane. Computational design of robotic devices from high-level motion specifications.IEEE Transactions on Robotics, 34(5): 1240–1251, 2018

    2018

  64. [64]

    H. T. Suh, M. Simchowitz, K. Zhang, T. Pang, and R. Tedrake. Pathologies and challenges of using differentiable simulators in policy optimization for contact-rich manipulation. InICRA 2022 Workshop: Reinforcement Learning for Contact-Rich Manipulation, 2022

    2022

  65. [65]

    J. Xu, V . Makoviychuk, Y . Narang, F. Ramos, W. Matusik, A. Garg, and M. Macklin. Acceler- ated policy learning with parallel differentiable simulation.arXiv preprint arXiv:2204.07137, 2022

    Pith/arXiv arXiv 2022

  66. [66]

    S. Ha, S. Coros, A. Alspach, J. Kim, and K. Yamane. Computational co-optimization of design parameters and motion trajectories for robotic systems.The International Journal of Robotics Research, 37(13-14):1521–1536, 2018

    2018

  67. [67]

    K ¨ulz, S

    J. K ¨ulz, S. Ha, and M. Althoff. A design co-pilot for task-tailored manipulators.arXiv preprint arXiv:2509.13077, 2025

    arXiv 2025

  68. [68]

    Rudin, D

    N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. InConference on robot learning, pages 91–100. PMLR, 2022

    2022

  69. [69]

    G. Authors. Genesis: A generative and universal physics engine for robotics and beyond, December 2024. URLhttps://github.com/Genesis-Embodied-AI/Genesis

    2024

  70. [70]

    Kumar, Z

    A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.04034, 2021

    Pith/arXiv arXiv 2021

  71. [71]

    Zhuang, Z

    Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning.arXiv preprint arXiv:2309.05665, 2023

    Pith/arXiv arXiv 2023

  72. [72]

    Z. Fu, A. Kumar, J. Malik, and D. Pathak. Minimizing energy consumption leads to the emer- gence of gaits in legged robots.arXiv preprint arXiv:2111.01674, 2021

    Pith/arXiv arXiv 2021

  73. [73]

    G. B. Margolis and P. Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. InConference on Robot Learning, pages 22–31. PMLR, 2023

    2023

  74. [74]

    Mandi, Y

    Z. Mandi, Y . Weng, D. Bauer, and S. Song. Real2code: Reconstruct articulated objects via code generation. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  75. [75]

    K. Sims. Evolving virtual creatures. InSeminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 699–706. 2023

    2023

  76. [76]

    Lipson and J

    H. Lipson and J. B. Pollack. Automatic design and manufacture of robotic lifeforms.Nature, 406(6799):974–978, 2000. 13

    2000

  77. [77]

    K. De Jong. Evolutionary computation: a unified approach. InProceedings of the Genetic and Evolutionary Computation Conference Companion, pages 373–388, 2017

    2017

  78. [78]

    C. G. Langton. Artificial life: An overview. 1997

    1997

  79. [79]

    Cheney, R

    N. Cheney, R. MacCurdy, J. Clune, and H. Lipson. Unshackling evolution: evolving soft robots with multiple materials and a powerful generative encoding.ACM SIGEVOlution, 7(1):11–23, 2014

    2014

  80. [80]

    Y . Yuan, Y . Song, Z. Luo, W. Sun, and K. Kitani. Transform2act: Learning a transform-and- control policy for efficient agent design.arXiv preprint arXiv:2110.03659, 2021

    Pith/arXiv arXiv 2021

Showing first 80 references.