Shared Modular Recurrence in Contextual MDPs for Universal Morphology Control

Caroline Horsch; Daan Brinks; Laurens Engwegen; Max Weltevrede; Wendelin B\"ohmer

REVIEW 4 major objections 5 minor 29 references

Shared Modular Recurrence in Contextual MDPs for Universal Morphology Control

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding a shared recurrent memory to modular control policies substantially improves zero-shot generalization to unseen robot morphologies in four MuJoCo environments.

desk verdict A clean empirical study showing modular recurrence helps zero-shot morphology generalization, though the mechanism remains unverified and no code is released. read the letter →

arxiv 2506.08630 v3 pith:26M6KYFR submitted 2025-06-10 cs.AI cs.RO

Laurens Engwegen , Max Weltevrede , Caroline Horsch , Daan Brinks , Wendelin Böhmer This is my paper · ORCID

classification cs.AIcs.RO

keywords modularrecurrencezero-shotgeneralizationuniversalmorphologycontrolcontextualMDPpartiallyobservablecontexttransformerpolicyrecurrentPPOMuJoCorobots

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a universal robot controller can generalize better to unseen robots if its modular policy is given memory. It treats the robot's context—mass, topology, damping, and other properties—as only partially observable, and hypothesizes that a shared recurrent network over each limb can infer the missing context through interaction. The authors add modular recurrence to two existing universal-controller architectures, MetaMorph and ModuMorph, and report consistent zero-shot gains on robots with unseen dynamics, kinematics, and topologies across four MuJoCo environments. If the claim holds, a single policy can control new robot bodies without retraining, which would save the compute and data currently spent on per-robot training.

What carries the argument

The central mechanism is shared modular recurrence: one RNN, with parameters shared across all limbs, ingests each limb's observation and previous action and maintains a per-limb hidden state that encodes that limb's local action-observation history; a transformer then attends over these per-limb encodings to model inter-limb dependencies. This preserves the modularity of the original MetaMorph (a transformer-based universal controller) and ModuMorph (which adds hypernetwork-generated contextual modulation), while adding memory. The recurrent policies are trained with recurrent PPO and recurrent experience replay using chunked episodes with a burn-in period.

What would settle it

Train R-MoMo and ModuMorph with matched parameter counts, or replace the RNN hidden state with a zero vector at test time while keeping the rest of the network intact: if zeroing the memory does not reduce performance, the improvement is not from recurrence inferring context. A complementary check is to probe the hidden states after context changes and ask whether they encode robot identity or environment conditions.

Watch

Extended reading notes

Core claim

On the paper's own account, the central discovery is that treating a contextual MDP as partially observable and giving each limb a shared recurrent encoder of its local action-observation history improves both multi-robot training and zero-shot generalization. In the main comparison, R-MoMo (recurrent ModuMorph) outperforms ModuMorph on all four environments on the 70 unseen test topologies, for example 1829 versus 1455 mean return on Flat Terrain, and the per-robot return differences show R-MoMo ahead on a majority of test robots rather than on a few outliers. Recurrent versions also beat their non-recurrent baselines on the 2400 robots with modified dynamics and kinematics. The authors interpret this as evidence that the memory mechanism lets the policy infer relevant unobservable context, while acknowledging that whether the RNN actually performs such inference is not yet verified.

Load-bearing premise

The load-bearing premise is that the performance gain comes from the recurrent memory inferring unobservable context rather than from the extra parameters or expressive power the RNN adds, which the paper does not control for and does not verify—the authors state that whether the agent actually uses the RNN to infer relevant contextual features remains to be investigated.

Editorial extensions

If this is right

A single policy with shared recurrence can control robots whose bodies were never seen during training, reducing the need for per-robot training or fine-tuning.
Recurrent memory helps across different environments and across both baseline architectures, so the effect is not tied to one network design or one terrain.
Because the advantage holds on a majority of individual test robots, it is a systematic generalization gain rather than a few lucky bodies.
The need to store a hidden state per limb means scaling to very large robot morphologies will require a more memory-efficient recurrence mechanism.
Combining recurrence with hypernetwork-based context modulation yields the strongest results, indicating that explicit context and inferred memory can coexist.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: The gains may reflect recurrence acting as a within-episode adaptive controller (e.g., reacting to terrain or gait changes) rather than as a context estimator; a test that changes context mid-episode would separate the two.
Inference: Because the paper does not match parameter counts, part of the improvement could be capacity; ablating the RNN while increasing feedforward width to match parameters would test this.
Inference: The same per-node shared recurrence could transfer to other graph-structured control problems, such as manipulation of deformable objects or multi-agent teams, where node-level context is also incomplete.
Inference: Training R-MoMo with the provided context features removed would show how much of the generalization depends on observable context versus inferred memory; if recurrence alone nearly matches the full model, explicit context engineering matters less than claimed.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know. The empirical claim is solid: on 70 unseen topologies, R-MoMo beats ModuMorph in all four environments, with non-overlapping standard deviations, and the per-robot difference plots show the gains are spread across many robots, not driven by a few outliers. That is a real result. The explanation, though, is not established. The authors concede in Section 6 that whether the RNN actually infers contextual features remains to be investigated, and I think the missing capacity-matched baseline is a more serious gap than they let on.

What is actually new: framing morphology generalization as a partially observable CMDP and adding a shared per-limb RNN before the transformer in MetaMorph and ModuMorph. That combination is new, and the evaluation covers four environments, three generalization settings, and five seeds. The use of a validation set to pick the one hyperparameter they tuned (the KL threshold) is clean, and the per-robot breakdowns are a nice answer to the usual 'averages hide outliers' problem.

Where it is soft. The main issue is that R-MoMo differs from ModuMorph in two ways at once: it adds recurrence and it adds parameters. Without a feedforward baseline matched for parameter count, or an ablation that keeps the architecture fixed and disables the hidden state, the performance gain cannot be attributed to the proposed mechanism. The authors are upfront about not verifying the inference story, but they do not draw the stronger conclusion that the comparison is confounded. I would call that the paper's one load-bearing weakness. A second, smaller issue is that no code or data are released, which limits reproducibility. Neither issue invalidates the headline performance numbers.

Who this is for: anyone working on universal morphology control or multi-robot RL. The result, that adding modular recurrence improves zero-shot generalization, is worth knowing even if the mechanism is unresolved. I would not cite it for the mechanism, but I would cite it as empirical evidence that recurrence helps.

Recommendation: this deserves a serious peer review. The request to the authors should be to add a capacity-matched baseline and to release code and data. If that is too much for the venue, the paper still stands as a preliminary result, but the mechanism claim should be softened further.

Referee Report

4 major / 5 minor

Summary. The paper proposes adding a shared per-limb recurrent network (an RNN encoding each limb's action-observation history) to two universal morphology control architectures, MetaMorph and ModuMorph, in order to handle partial observability of robot context. The resulting methods, R-MeMo and R-MoMo, are evaluated on a set of 100 training robots from the UNIMAL design space, and on zero-shot generalization to unseen dynamics/kinematics variations and 70 unseen topologies across four MuJoCo environments (Flat Terrain, Incline, Variable Terrain, Obstacles). The central empirical claim is that R-MoMo substantially improves zero-shot generalization over ModuMorph, e.g., 1829 vs 1455 mean return on Flat Terrain, with per-robot difference plots showing consistent gains. The authors attribute this improvement to the recurrent module's ability to infer unobservable contextual features through interaction, while acknowledging that this mechanism remains unconfirmed.

Significance. If the performance gains hold up, the work offers a simple and potentially effective architectural modification for universal morphology control, with relevance beyond locomotion. The paper has clear strengths: experiments use 5 seeds, report both aggregate and per-robot comparisons, evaluate on 70 unseen topologies and 2400 dynamics/kinematics variants, and compare against two strong baselines (MetaMorph and ModuMorph). The authors are also transparent about the unresolved mechanism in Section 6. However, the significance is currently tempered by the lack of a capacity-matched control and the absence of direct evidence that the RNN infers context; these are needed to distinguish the proposed partial-observability explanation from a generic increase in model capacity. The reported performance numbers could be entirely correct while the paper's conceptual contribution remains unsupported.

major comments (4)

[§4.2, Table 1] The central attribution of the improvement to recurrence is confounded by parameter count. R-MoMo adds a per-limb RNN with trainable parameters to ModuMorph, so the comparison varies both the recurrence mechanism and the number of trainable parameters. There is no feedforward baseline that adds a comparable number of non-recurrent parameters to ModuMorph, nor an ablation that keeps the RNN architecture but removes the hidden state (e.g., resetting the hidden state at every timestep). Without such controls, the reported gains (e.g., 1829 vs 1455 on Flat Terrain) cannot be specifically attributed to temporal inference of partially observable context. This gap is load-bearing because the abstract and Section 2.2 frame the contribution as recovering unobservable context through modular interactions; if the gain comes from capacity or regularization instead, the proposed mechanism would not be supported. The authors' own discussion in Section 6 concedes this: "Whether the agent actually uses the RNN to infer relevant contextual features remains to be investigated."
[§5.4, Table 1] The statement that R-MoMo "significantly outperforms" ModuMorph is based on non-overlapping one-standard-deviation intervals of the mean over 5 seeds. For a zero-shot comparison across 70 test robots, a paired or per-robot statistical test (e.g., Wilcoxon signed-rank on the 70 per-robot differences, or a bootstrap with a stated significance level) would be more informative and would also address the multiple comparisons across four environments. The per-robot difference plots in Figure 6 and Appendix B are visually helpful, but they are not accompanied by any quantitative test; the reader cannot assess whether the consistent-looking gains are statistically reliable or how many robots would need to be flipped to change the conclusion.
[§2.2 / §5.3 / §6] The partial-observability mechanism is only hypothesized; no experiment in the paper demonstrates that the RNN's hidden state actually encodes the unobserved context. For instance, one could train a decoder to predict context variables (such as limb friction or damping) from the hidden state, or compare against a variant that receives the full context oracle. This is related to the capacity-confounding issue but is a distinct evidential gap: even with a capacity-matched baseline, the inference story would require direct evidence that the recurrent state carries information beyond the current observation. As written, the paper presents improved test returns but no internal analysis of what the RNN has learned, which is exactly the load-bearing claim of the paper.
[§5.3, Figure 5] The text says that R-MoMo "obtains a higher average return than ModuMorph, consistently throughout the different environments" for dynamics/kinematics changes, while acknowledging the differences are "albeit not always significantly." This wording is appropriately cautious, but the abstract's broader claim of "substantial improvement" may overstate the evidence in this setting, since many of the individual error bars in Figure 5 overlap. Please clarify which specific comparisons (which parameter changes and environments) actually show non-overlapping intervals or statistically significant gains, and distinguish the strength of evidence for the topology generalization results (Table 1) from the dynamics/kinematics results (Figure 5).

minor comments (5)

[Abstract] The phrase "substantial improved performance" should be "substantial improvement in performance" or "substantially improved performance." Also, consider adding one sentence to the abstract describing what "modular recurrence" means for readers unfamiliar with the architecture.
[§2.2] The notation for the emission function ϕ:S→O maps a state to an observation that includes s'_t and c+, but it is not specified whether c+ is time-dependent or identical at every step; this is relevant because the RNN is supposed to infer the unprovided part of the context. Please clarify that c+ is static within an episode.
[§4.2] The RNN update h_t^i = RNN(o_t^i, a_{t-1}^i, h_{t-1}^i) relies on limb-wise observations o_t^i and actions a_{t-1}^i, but the preceding sections define observations as pairs (s'_t, c+). Please define how these are decomposed per limb, and state the hidden-state dimensionality used in the experiments.
[§5.1] The validation/test split of the 70 unseen topologies is described as "randomly split" but no random seed or split procedure is given; please provide the split details (or the seed) to ensure reproducibility, especially since only two hyperparameter values are evaluated on this validation set.
[Figure 4 and Figure 5] The y-axis label "Return" would be clearer if the caption stated that these are undiscounted episode returns averaged over 5 seeds and, for Figure 5, over the test robots in each parameter-change category. Currently the reader must infer this from the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the results are an empirical comparison against external baselines; the unverified mechanism is a confound, not a circular derivation.

full rationale

The paper's central claims are empirical, not derived. Table 1 reports zero-shot returns of R-MoMo and R-MeMo against the external MetaMorph and ModuMorph baselines on held-out UNIMAL robots, and no equation in the paper reduces the recurrent model's output to its baseline by construction. The only hyperparameter tuning is a KL-divergence threshold selected on a validation set (Appendix A), which does not make the test-set predictions fitted or self-predictive. The proposed mechanism, inference of partially observable context through shared modular recurrence, is admittedly unverified: Section 6 states 'Whether the agent actually uses the RNN to infer relevant contextual features remains to be investigated.' That is an unsupported attribution, and the lack of a capacity-matched feedforward baseline is a confound, but neither is circularity. The one citation involving a co-author (Kurin et al., 2021) is background on graph-based morphology control and is not load-bearing for the present empirical comparison. No uniqueness theorem, ansatz, or fitted parameter is smuggled in, and no known result is renamed as a prediction. Hence the paper's derivation chain is self-contained with respect to its measured outcomes.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The empirical claim rests on standard RL/ML machinery (PPO, transformers, RNNs, MuJoCo) and on domain assumptions about the simulator and the UNIMAL split. The one ad hoc hypothesis is that missing context can be recovered from local histories; this is the load-bearing conceptual premise and it is not independently verified. The only fitted number is the KL-divergence threshold, selected on the validation set. No new physical entities are introduced.

free parameters (2)

KL-divergence regularization threshold = 3 or 5 (selected per method on 32-robot validation set)
The only hyperparameter tuned in this paper; the reported models are the ones with the best validation performance (Appendix A).
Recurrent chunk size and burn-in length = m=80, l=20
Taken from Kapturowski et al. 2018 without fitting; these values shape the recurrent training but are not fitted here.

assumptions (3)

domain assumption MuJoCo's physics simulation accurately represents robot dynamics for evaluating morphology control.
All training and test performance is measured in the MuJoCo UNIMAL simulator (Section 5.1); the conclusions are stated for this simulation setting.
ad hoc to paper Unobservable context can be inferred from per-limb action-observation history within an episode.
This is the paper's core hypothesis (Sections 2.2 and 4.2), not proven; the authors explicitly note in Section 6 that whether the RNN actually infers context remains to be investigated.
domain assumption The 100 training robots and the validation/test splits are representative of the UNIMAL design space.
Generalization is measured on this specific split from Gupta et al. 2022 (Section 5.1); results may not transfer to other robot distribution.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Shared Modular Recurrence in Contextual MDPs for Universal Morphology Control." pith.science (2026). https://pith.science/paper/26M6KYFR

@misc{pith2026250608630,
  author       = {Pith},
  title        = {Pith review of: Shared Modular Recurrence in Contextual MDPs for Universal Morphology Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/26M6KYFR}},
  note         = {Machine review of arXiv:2506.08630}
}

read the original abstract

A universal controller for any robot morphology would greatly improve computational and data efficiency. Steps have been made towards such multi-robot control by utilizing contextual information about the properties of individual robots and exploiting their modular structure in the architecture of deep reinforcement learning agents. When the robots have highly dissimilar morphologies, however, this becomes a challenging problem, especially when the agent must generalize to new, unseen robots. In this paper, we posit that contextual features are often only partially available, but that they can be recovered through modular interactions. This can allow for better multi-robot control and generalization to contexts that are not seen during training. To this extent, we implement a transformer-based architecture with shared modular recurrence and evaluate its (generalization) performance on a large set of MuJoCo robots. The results show a substantial improvement in zero-shot generalization performance on robots with unseen dynamics, kinematics, and topologies, in four different environments.

Figures

Figures reproduced from arXiv: 2506.08630 by the authors.

**Figure 1.** Example of robots that can be found in the UNIMAL design space [Gupta et al., 2021]. This is not only challenging because robots can have different action and state spaces, but also because robots with different morphologies and/or dynamics might learn tasks in different ways. The multi-task framework allows us to evaluate the performance of an agent during multi-robot training, and its zero-shot generalization per… view at source ↗

**Figure 2.** Illustration of the recurrent MetaMorph (R-MeMo) architecture. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** The ModuMorph architecture with added recurrence (R-MoMo). [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Training performance on the 100 training robots in the different environments. Mean and [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Test performance on training robot topologies with changes in contextual features that result [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: The difference in return between R-MoMo and ModuMorph for each of the 70 unseen test [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Test performance on the 32 robots in the validation set for different values of the regulariza [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

**Figure 8.** Figure 8: The difference in return between R-MoMo and ModuMorph for each of the 70 unseen test [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

**Figure 9.** Figure 9: The difference in return between ModuMorph and MetaMorph for each of the 70 unseen [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 17 canonical work pages

[1]

End-to-end training of deep visuomotor policies

Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17 0 (39): 0 1--40, 2016

work page 2016
[2]

Scalable deep reinforcement learning for vision-based robotic manipulation

Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning, pages 651--673. PMLR, 2018

work page 2018
[3]

Learning dexterous in-hand manipulation

OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39 0 (1): 0 3--20, 2020

work page 2020
[4]

Rt-1: Robotics transformer for real-world control at scale

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022

arXiv 2022
[5]

Rt-2: Vision-language-action models transfer web knowledge to robotic control

Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165--2183. PMLR, 2023

work page 2023
[6]

Open x-embodiment: Robotic learning datasets and rt-x models

Quan Vuong, Sergey Levine, Homer Rich Walke, Karl Pertsch, Anikait Singh, Ria Doshi, Charles Xu, Jianlan Luo, Liam Tan, Dhruv Shah, et al. Open x-embodiment: Robotic learning datasets and rt-x models. In Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition@ CoRL2023, 2023

2023
[7]

Scaling cross-embodied learning: One policy for manipulation, navigation, locomotion and aviation

Ria Doshi, Homer Walke, Oier Mees, Sudeep Dasari, and Sergey Levine. Scaling cross-embodied learning: One policy for manipulation, navigation, locomotion and aviation. arXiv preprint arXiv:2408.11812, 2024

arXiv 2024
[8]

Octo: An open-source generalist robot policy

Octo Model Team , Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science a...

2024

Show all 29 references

[9]

Embodied intelligence via learning and evolution

Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature Communications, 12 0 (1): 0 5721, 2021

2021
[10]

A survey of multi-task deep reinforcement learning

Nelson Vithayathil Varghese and Qusay H Mahmoud. A survey of multi-task deep reinforcement learning. Electronics, 9 0 (9): 0 1363, 2020

2020
[11]

A survey of zero-shot generalisation in deep reinforcement learning

Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rockt \"a schel. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research, 76: 0 201--264, 2023

2023
[12]

Metamorph: Learning universal controllers with transformers

Agrim Gupta, Linxi Fan, Surya Ganguli, and Li Fei-Fei. Metamorph: Learning universal controllers with transformers. International Conference on Learning Representations, 2022

2022
[13]

Universal morphology control via contextual modulation

Zheng Xiong, Jacob Beck, and Shimon Whiteson. Universal morphology control via contextual modulation. In International Conference on Machine Learning, pages 38286--38300. PMLR, 2023

2023
[14]

Contextual markov decision processes, 2015

Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes, 2015

2015
[15]

Mujoco: A physics engine for model-based control

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026--5033. IEEE, 2012

2012
[16]

One policy to control them all: Shared modular policies for agent-agnostic control

Wenlong Huang, Igor Mordatch, and Deepak Pathak. One policy to control them all: Shared modular policies for agent-agnostic control. In International Conference on Machine Learning, pages 4455--4464. PMLR, 2020

2020
[17]

a schel, Wendelin B \

Vitaly Kurin, Maximilian Igl, Tim Rockt \" a schel, Wendelin B \" o hmer, and Shimon Whiteson. My body is a cage: the role of morphology in graph-based incompatible control. International Conference on Learning Representations, 2021

2021
[18]

Why generalization in rl is difficult: Epistemic pomdps and implicit partial observability

Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P Adams, and Sergey Levine. Why generalization in rl is difficult: Epistemic pomdps and implicit partial observability. Advances in Neural Information Processing Systems, 34: 0 25502--25515, 2021

2021
[19]

Partially observable markov decision processes

Matthijs TJ Spaan. Partially observable markov decision processes. In Reinforcement learning: State-of-the-art, pages 387--414. Springer, 2012

2012
[20]

Deep recurrent q-learning for partially observable mdps

Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 AAAI Fall Symposium Series, 2015

2015
[21]

Learning to control self-assembling morphologies: a study of generalization via modularity

Deepak Pathak, Christopher Lu, Trevor Darrell, Phillip Isola, and Alexei A Efros. Learning to control self-assembling morphologies: a study of generalization via modularity. Advances in Neural Information Processing Systems, 32, 2019

2019
[22]

The graph neural network model

Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20 0 (1): 0 61--80, 2008

2008
[23]

Nervenet: Learning structured policy with graph neural networks

Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In International conference on learning representations, 2018

2018
[24]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017

2017
[25]

Hypernetworks

David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016

2016 arXiv
[26]

Long short-term memory

Sepp Hochreiter and J \"u rgen Schmidhuber. Long short-term memory. Neural Computation, 9 0 (8): 0 1735--1780, 1997

1997
[27]

The sensory neuron as a transformer: Permutation-invariant neural networks for reinforcement learning

Yujin Tang and David Ha. The sensory neuron as a transformer: Permutation-invariant neural networks for reinforcement learning. Advances in Neural Information Processing Systems, 34: 0 22574--22587, 2021

2021
[28]

Proximal policy optimization algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

2017 arXiv
[29]

Recurrent experience replay in distributed reinforcement learning

Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International Conference on Learning Representations, 2018

2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

End-to-end training of deep visuomotor policies

Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17 0 (39): 0 1--40, 2016

work page 2016

[2] [2]

Scalable deep reinforcement learning for vision-based robotic manipulation

Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning, pages 651--673. PMLR, 2018

work page 2018

[3] [3]

Learning dexterous in-hand manipulation

OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39 0 (1): 0 3--20, 2020

work page 2020

[4] [4]

Rt-1: Robotics transformer for real-world control at scale

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022

arXiv 2022

[5] [5]

Rt-2: Vision-language-action models transfer web knowledge to robotic control

Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165--2183. PMLR, 2023

work page 2023

[6] [6]

Open x-embodiment: Robotic learning datasets and rt-x models

Quan Vuong, Sergey Levine, Homer Rich Walke, Karl Pertsch, Anikait Singh, Ria Doshi, Charles Xu, Jianlan Luo, Liam Tan, Dhruv Shah, et al. Open x-embodiment: Robotic learning datasets and rt-x models. In Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition@ CoRL2023, 2023

2023

[7] [7]

Scaling cross-embodied learning: One policy for manipulation, navigation, locomotion and aviation

Ria Doshi, Homer Walke, Oier Mees, Sudeep Dasari, and Sergey Levine. Scaling cross-embodied learning: One policy for manipulation, navigation, locomotion and aviation. arXiv preprint arXiv:2408.11812, 2024

arXiv 2024

[8] [8]

Octo: An open-source generalist robot policy

Octo Model Team , Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science a...

2024