Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A paired variational encoder-decoder extracts a shared, morphology-invariant control subspace from paired state-action trajectories, and uses a KL reward-shaping term to transfer a skill from a source robot body to a morphologically…

desk verdict PVED is a plausible step beyond deterministic subspace transfer, but the load-bearing paired-trajectory alignment is unvalidated and the theory is more of a consistency check than a guarantee. read the letter →

arxiv 1908.05265 v2 pith:7WB6D7GT submitted 2019-08-14 cs.LG stat.ML

Yang Hu , Giovanni Montana This is my paper · ORCID
classification cs.LGstat.ML
keywords skilltransferdeepreinforcementlearningmorphologicalheterogeneityvariationalencoder-decodersharedlatentsubspacedisentanglementrewardshapingpairedtrajectories
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper addresses transfer between morphologically different reinforcement-learning agents. It claims that a paired variational encoder-decoder (PVED) can learn, from paired state-action trajectories of a common training task, a shared latent subspace representing morphology-invariant control stages and an individual subspace per agent. At transfer time, a KL-divergence reward-shaping term in that shared subspace lets the target agent's policy be shaped to match the source's control stages, so a skill like forward locomotion learned by a biped can bootstrap training of a hopper or ant. The paper also proves a bound: the expected-reward gap between target and source policies is at most $m + \frac{b-a}{2}\sqrt{2\delta}$, where $\delta$ is the KL mismatch in the shared subspace and $m$ bounds the similarity of the reward functions. If correct, the result means skills can be reused across different robot bodies using only one training task and two agents.

What carries the argument

The central object is the paired variational encoder-decoder (PVED), a probabilistic model that splits the control of two agents into shared latent factors $c$ (morphology-invariant control stages) and individual factors $y$ (body-specific control). It is built from a pair of variational encoder-decoders that approximate each agent's posterior from the other agent's observations, joined by a KL consistency term on the shared factors and two reconstruction bounds that force information through the shared subspace. The transfer mechanism is reward shaping by the negative KL divergence between the two shared-factor posteriors, added to the target agent's environmental reward while its policy is trained.

What would settle it

Train PVED with the standard subsampled pairing, then retrain with deliberately corrupted pairings such as randomly shuffling the target trajectory relative to the source or shifting it by a large phase offset. If the target agent's transferred performance stays as good as with the correct pairing, the shared subspace is not actually using the paired correspondence and the central mechanism is not load-bearing; a complementary check is to measure phase alignment (for example via gait phase or contact events) on the paired trajectories and test whether transfer success correlates with that measure.

Watch

Extended reading notes

Core claim

The central claim is that PVED learns a shared latent subspace that captures stages of a control task independently of body morphology, and that this subspace is sufficient to transfer skills. The model encodes $(s_S,a_S)$ and $(s_T,a_T)$ into shared factors $c$ and individual factors $y$, using two paired variational encoder-decoders: one encodes target latent factors from source control and decodes to target observations, the other encodes source latent factors from target control. A consistency KL term aligns the shared-factor distributions, and two information-flow terms prevent the encoders from collapsing all information into the individual subspaces. At transfer, the target agent's reward is shaped by $-\beta D_{\mathrm{KL}}(q_{\varphi_S}(c_T|s_S,a_S)\|q_{\varphi_T}(c_S|s_T,a_T))$, so the target policy is pushed to produce shared-factor states matching the source's. The paper's Theorem 1 states that, under an injectivity assumption on the encoders and a reward-similarity bound $|R'_T(c)-R'_S(c)|\le m$ with $R'_S(c)\in[a,b]$, the expected-reward gap is at most $m + \frac{b-a}{2}\sqrt{2\delta}$; the corollary extends this to discounted returns by the factor $1/(1-\gamma)$.

Load-bearing premise

The load-bearing assumption is that subsampling the longer trajectory along the timesteps creates valid paired correspondences between the two agents' states and actions, so that equal numbers of subsampled steps represent the same phase of the control task; this correspondence is never validated against any measure of phase alignment.

Editorial extensions

If this is right

  • A skill learned on a source body can bootstrap training of a target body with a different shape, action space, and dynamics, using only paired trajectories from a single simpler training task rather than from the target task itself.
  • The performance gap between target and source policies is controlled by two measurable quantities: the KL mismatch $\delta$ between the shared-factor distributions and the reward-similarity bound $m$ when rewards are written in shared coordinates.
  • Because transfer works through a stochastic shared subspace, the method does not require a direct mapping between state spaces or a known correspondence in action spaces, which are problematic for agents with large morphological differences.
  • The learned shared subspace is interpretable: visualisations in the paper show that nearby shared-factor points correspond to similar control stages across two different bodies, such as swinging, approaching, pushing, accelerating, balancing, and jumping.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the phase-correspondence assumption is the bottleneck, a natural extension would replace proportional subsampling with dynamic time warping or phase-based alignment; one would then predict that transfer improves or becomes more robust when paired trajectories are aligned by gait phase rather than by equal timestep counts.
  • The bound suggests a practical diagnostic: monitoring $\delta$ during target-agent training should predict whether transfer will succeed, and a persistently large $\delta$ could be used as a stopping or intervention criterion.
  • The same shared-subspace mechanism could in principle transfer between simulated and physical robots or between robots with different degrees of freedom, because the shared factors are learned from state-action pairs rather than from hand-defined correspondences.
  • A testable consequence of the injectivity assumption is that decoding a shared factor should recover a recognizable state-action pair for each agent; if decoding is poor for either agent, the bound's assumption fails and transfer performance should degrade.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes PVED, a paired variational encoder-decoder for transferring reinforcement learning skills between agents with different morphologies. The model represents each agent's control mechanism through joint state-action distributions and disentangles them into a shared latent subspace and an agent-specific individual subspace. Consistency between the shared subspaces is enforced with a KL-divergence term, and skill transfer is performed by adding a KL-based reward-shaping term to the target agent's reward. The authors derive a theorem bounding the performance gap between the transferred target policy and the source policy by a reward-similarity term m and a shared-subspace mismatch term delta. The method is evaluated on four MuJoCo agent pairs (two robot-arm pairs, two locomotion pairs) against a baseline, a prior method (Gupta et al. 2017), and four ablations, with additional t-SNE visualizations of the learned subspaces. The central claim is that PVED enables skill transfer under morphological heterogeneity.

Significance. If the claims hold, the paper addresses a practically important and technically open problem: transferring skills across agents with different bodies, state spaces, and action spaces. The model is principled, the ablations are well designed (state-only, shared-only, deterministic-shared, VAE-posterior), and the derivation of Theorem 1 is explicit and checkable. The qualitative visualizations are a useful attempt to interpret the learned subspaces. However, the paired-trajectory alignment that underpins both the representation learning and the transfer objective is not validated, the main theorem's injectivity assumption is inconsistent with the stochastic encoders used in the model, and the empirical comparisons are reported without error bars or significance tests. These issues are load-bearing for the paper's central claims, so the manuscript needs substantive revision.

major comments (4)
  1. [V-A, Figs. 3-7] The paired-trajectory correspondence is load-bearing and is implemented by 'align[ing] the trajectories between the two agent by subsampling the longer trajectories along the timesteps' (Section V-A). This assumes that proportional time-scaling maps movement phases across different bodies and gaits. For Walker2d versus Hopper (walking versus hopping) and Humanoid versus Ant, gait cycles have different periods, so equal timestep indices need not correspond to the same control stage (e.g., stance, swing, foot contact). Because the paired data are used both for the PVED training objectives in Eqs. (2), (6), and (7) and for the t-SNE interpretation in Figs. 4-7, a spurious time-index alignment could produce the reported shared-subspace structure and transfer gains without any morphology-invariant control stages. The paper provides no validation of the alignment against any phase measure, and the t-SNE analysis cannot independently confirm phase correspondence because it relies on the same time-aligned trajectories. This issue should be addressed by comparing against a phase-based or dynamic-time-warping alignment, or by an ablation that randomizes the alignment.
  2. [III-E, Assumption 1] Assumption 1 states that phi_S and phi_T define injective mappings from (sS,aS) and (sT,aT) to unique shared-feature representations c. However, the PVED encoders are stochastic: Section III-F specifies q_phiS(cT|sS,aS) and q_phiT(cS|sT,aT) as multivariate Gaussian conditional distributions, and Section III-C explicitly motivates stochasticity. A Gaussian encoder assigns positive probability to many c values for a given (s,a), so no deterministic injective mapping exists in the trained model. Consequently, the premise of Theorem 1 is not satisfied by the actual PVED model, and the bound in Eq. (13) is not a performance guarantee for the proposed method. Please either state the theorem for the deterministic means of the encoders with a correspondingly modified KL condition, restrict the model to a low-variance regime, or explicitly characterize Theorem 1 as applying only to an idealized deterministic version of the model.
  3. [Fig. 3 and V-A] The main empirical claims are reported as curves without error bars, confidence intervals, or significance tests, although 15 runs per method are performed. Statements such as 'PVED achieves better performance compared to the baseline' and 'PVED performs consistently better than PVED-state only on all tasks' are not supported statistically as presented. Moreover, the authors concede that on the Hopper curved-plane climbing task, PVED-det shared subspace performs better than PVED, which weakens the 'consistently better' characterization. Please report mean curves with shaded confidence regions and include pairwise significance tests or effect sizes for the main comparisons against the baseline and the Gupta et al. 2017 method.
  4. [III-E, Eqs. (13)-(14)] The theorem's bound is conditional on quantities that the method itself is trained to minimize: delta is the same KL divergence used as the consistency objective in Eq. (7) and as the shaping reward in Eq. (10), and m is the reward mismatch measured inside the learned shared subspace rather than an externally fixed morphology similarity. As a result, the bound does not provide an external performance guarantee; it states that if the learned subspace makes the rewards similar and the policies close in that subspace, then performance is close. This is a weaker theoretical contribution than the text suggests when it describes the theorem as providing 'a performance guarantee for the proposed method.' Please clarify that m and delta are internal to the learned model, and discuss whether a bound in terms of an externally defined state/action correspondence is possible.
minor comments (5)
  1. [Throughout] There are several typos and grammatical errors, including 'morpholigical' in Section III-C, 'Similaly' and 'MHAs' in Section III-E, 'refered' in Section V-B, and 'exept' in Section V-A; a careful proofread is needed.
  2. [III-E and Eq. (7)] The KL divergence in Eq. (7) is written between q_phiS(cT|sS,aS) and q_phiT(cS|sT,aT), but the notation C = C_S union C_T in Section III-E does not make explicit that these distributions must share a common measurable space for the KL divergence to be well defined; please clarify how cS and cT are identified.
  3. [Eqs. (9)-(10)] The hyperparameter beta is introduced only in the transfer objective Eq. (10), while the PVED objective Eq. (9) uses alpha1, alpha2, alpha3; please specify how beta is tuned relative to the other hyperparameters and whether it affects the PVED training phase.
  4. [Fig. 3] The figure caption does not state whether the plotted curves are means, medians, or single representative runs; please clarify and, ideally, add dispersion information such as interquartile ranges or shaded confidence bands.
  5. [References] Reference [61] duplicates reference [25] (both are Joshi and Chowdhary, 'Cross-domain transfer in reinforcement learning using target apprentice'); the duplicate should be removed or merged.

Circularity Check

2 steps flagged · score 6.0 of 10

Theorem 1's δ is the same KL divergence used as the Eq. 10 reward-shaping objective, and m is defined inside the learned shared subspace; the theoretical guarantee partly restates the training objective.

  1. self definitional [Section III-D Eq. 10 and Section III-E Theorem 1 (Eq. 13)]
    "R =−βDKL (qφS (cT|sS,aS)||qφT (cS|sT,aT )) (10) ... Theorem 1 . Given (sT,aT )∼ πT and (sS,aS)∼ πS, such that DKL (qφS (cT|sS,aS)||qφT (cS|sT,aT )) ⩽δ ... |Ec∼πT R′ T (c)− Ec∼πS R′ S (c)| ⩽m + (b−a) 2 √ 2δ (13)"

    The δ in Theorem 1 is exactly the KL divergence that Eq. 10 uses as the shaped reward during skill transfer. The target policy is trained to minimize DKL(qφS(cT|sS,aS)||qφT(cS|sT,aT)); therefore the theorem's conclusion that a smaller δ yields closer performance is not an independent characterization of morphology-invariant control stages but a restatement of the training objective. The dependence on the shared subspace is enforced by Eq. 10 by construction, not derived from the paired data, the morphology, or the task. If the encoders collapse to a constant, δ=0 and the bound reduces to the difference of average rewards, which is the degenerate solution the paper itself warns about in Section III-C.

  2. self definitional [Section III-E Assumption 1, Theorem 1, and the discussion after Eq. 13]
    "With Assumption 1, θ′ T in Eq. 8 actually defines an inverse mapping (sS,aS) = fθ′ T (c). Thus, the reward function of the source agent can be written as RS (sS,aS) = RS(fθ′ T (c)) = R′ S (c). Similaly we have RT (sT,aT ) = RT(fθ′ S (c)) = R′ T (c). ... Given the similarity of the source and target reward functions: |R′ T (c)−R′ S (c)| ⩽ m"

    The reward-similarity parameter m is defined only after mapping both agents' rewards through the learned decoders into the learned shared subspace c. Hence m is not a fixed property of the two morphologies or the task; it is a function of the same PVED parameters being trained. The statement that performance depends on morphology via m is therefore partly definitional: a model that maps corresponding state-action pairs to nearby c values will automatically make m small, subject only to the injectivity assumption. Thus the bound is measuring the model's own fitted geometry rather than an independently given morphological distance.

full rationale

The paper's empirical contribution is not circular: PVED is compared against baselines on MuJoCo tasks with held-out seeds, and the paired-trajectory alignment procedure is an unvalidated but non-circular assumption about phase correspondence. The circularity I find is confined to the theoretical performance guarantee. The δ term in Theorem 1 is exactly the KL divergence used as the shaped reward in Eq. 10, so the inequality restates the training objective as a performance bound; smaller δ is achieved by optimizing Eq. 10, not by discovering an independent morphological invariant. Likewise, the reward-similarity m is defined inside the learned shared subspace via the learned inverse mappings, so it is not an exogenous quantity. These features make the theorem's explanatory claims partially definitional. They do not by themselves generate the experimental results, and no load-bearing self-citation chain appears in the paper, so the score is 6 rather than higher.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The empirical claim rests on the paired-trajectory correspondence (created by subsampling alignment, Section V-A), the postulate that shared morphology-invariant control factors exist (Section III-B), and per-environment tuned weights alpha1, alpha2, alpha3, beta chosen on target-task performance (Section V-A, Table I). The theory additionally requires injectivity of the encoders into a 10-dimensional shared subspace, which is dimensionally impossible for the tested agents, plus a reward-similarity bound m defined inside the learned subspace that is never checked. The shared and individual latent subspaces are postulated entities whose only evidence is in-paper transfer gains and selected t-SNE frames.

free parameters (4)
  • Objective weights alpha1, alpha2, alpha3 (Eq. 9) = Per environment: (1.0, 1.0, 10.0), (1.0, 1.0, 10.0), (1.0, 1.0, 10.0), (2.5, 2.5, 15.0); Table I
    Tuned per environment using the target agent's target-task performance on tuning seeds (Section V-A), the same success-rate and distance metrics reported in Fig. 3.
  • Transfer shaping weight beta (Eq. 10) = 0.1, 0.01, 0.02, 0.01 for the four environment pairs (Table I)
    Chosen per environment after tuning; the authors report that different PVED initializations change the optimal beta, so beta is fixed using one PVED initialization and applied to others (Section V-A).
  • Shared and individual latent dimensions = 10 for all subspaces
    Hand-chosen (Section III-F); no sensitivity analysis over latent dimension is provided.
  • PPO hyperparameters = Not reported
    The PPO configuration (learning rate, clipping, epochs, entropy coefficient) is not specified, leaving unstated tuning freedom that affects both PVED and baseline comparisons (Section III-F).
assumptions (4)
  • ad hoc to paper Assumption 1: phiS and phiT are injective mappings from (sS,aS), (sT,aT) into the shared subspaces cT, cS.
    Required to write rewards as R'(c) and to define inverse mappings f(c) in the proof of Theorem 1 (Section III-E). A continuous injective map from a higher-dimensional state-action space into the 10-dimensional latent space cannot exist for the tested agents (e.g., Humanoid has a 376-dimensional state), so the assumption is violated in the experimental regime.
  • domain assumption Paired correspondence: (sS,aS) and (sT,aT) can be aligned into corresponding control stages.
    PVED is trained on paired trajectories created by running both agents on the training task with matched conditions and subsampling the longer trajectory (Sections III-A, V-A). The meaningfulness of the shared subspace depends on this time-alignment being a valid correspondence across morphologies.
  • domain assumption Existence of shared morphology-invariant control factors.
    The method postulates that differently shaped agents performing related tasks share latent control stages (Section III-B). This premise is tested only indirectly through downstream transfer performance; no independent test of the postulate is given.
  • ad hoc to paper Reward similarity in the shared subspace: |R'T(c) - R'S(c)| <= m.
    Assumed in Theorem 1. Since m is defined through the learned mappings, it is never measured; if the subspace is poorly learned, m can be large and the bound becomes vacuous.
invented entities (2)
  • Shared latent subspace c (10-dimensional)
    purpose: Captures morphology-invariant control stages used for the consistency constraint (Eq. 7) and the transfer reward (Eq. 10).
    Postulated in Section III-B; its existence and dimension are assumed, and the only support is in-paper transfer gains plus selected t-SNE frames, not an external falsifiable handle.
  • Individual latent subspace y (10-dimensional)
    purpose: Absorbs agent-specific control so that the shared subspace is cleaner; ablations removing it perform worse (Section V-B).
    Introduced as the complement of the shared factors (Section III-B); supported only by in-paper ablations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity." pith.science (2026). https://pith.science/paper/7WB6D7GT

@misc{pith2026190805265,
  author       = {Pith},
  title        = {Pith review of: Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7WB6D7GT}},
  note         = {Machine review of arXiv:1908.05265}
}
read the original abstract

Transfer learning methods for reinforcement learning (RL) domains facilitate the acquisition of new skills using previously acquired knowledge. The vast majority of existing approaches assume that the agents have the same design, e.g. same shape and action spaces. In this paper we address the problem of transferring previously acquired skills amongst morphologically different agents (MDAs). For instance, assuming that a bipedal agent has been trained to move forward, could this skill be transferred on to a one-leg hopper so as to make its training process for the same task more sample efficient? We frame this problem as one of subspace learning whereby we aim to infer latent factors representing the control mechanism that is common between MDAs. We propose a novel paired variational encoder-decoder model, PVED, that disentangles the control of MDAs into shared and agent-specific factors. The shared factors are then leveraged for skill transfer using RL. Theoretically, we derive a theorem indicating how the performance of PVED depends on the shared factors and agent morphologies. Experimentally, PVED has been extensively validated on four MuJoCo environments. We demonstrate its performance compared to a state-of-the-art approach and several ablation cases, visualize and interpret the hidden factors, and identify avenues for future improvements.

Figures

Figures reproduced from arXiv: 1908.05265 by the authors.

Figure 1
Figure 1. Illustration of the proposed PVED model. Enc and Dec are abbreviations [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustrations of the agents and environments of training/target tasks used in our experiments. We show the source agent in the training task and the [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Performance of comparison algorithms on the target agent on the target tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Analysis of t-SNE embeddings on the button pushing tasks with 3-link and 4-link Reachers. (c): The timesteps are normalized to [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Analysis of t-SNE embeddings on the peg insertion tasks with the Jaco3 and Fetch robot. (c): The timesteps are normalized to [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Analysis of t-SNE embeddings on the curved plane climbing tasks with Walker2d and Hopper. (c): The timesteps are normalized to [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Analysis of t-SNE embeddings on the curved plane climbing tasks with Humanoid and Ant. (c): The timesteps are normalized to [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Statistics for the grayscale groups around the markers (4) and (2) in [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Embodiment Scaling Laws in Robot Locomotion

    cs.RO 2025-05 conditional novelty 6.0 of 10

    A policy trained on about one thousand simulated robot bodies generalizes progressively better to unseen bodies as the number of training bodies grows, and it transfers zero-shot to two real robots.

Reference graph

Works this paper leans on

64 extracted references · 59 canonical work pages · cited by 1 Pith paper

  1. [1]

    R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction , 2nd ed. MIT Press, 2018

    work page 2018

  2. [2]

    Human-level control through deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature, vol. 518, pp. 529–533, 2015

    work page 2015

  3. [3]

    Mastering the game of go with deep neural networks and tree search,

    D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. V . D. Driessche, J. Schrittwieser, I. Antonoglou, V . Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of go with deep neural networks and tree search,” Nature, vol. ...

    work page 2016

  4. [4]

    Continuous control with deep reinforcement learning,

    T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971 , 2015

    arXiv 2015

  5. [5]

    End-to-end training of deep visuomotor policies,

    S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” Journal of Machine Learning Research , vol. 17, pp. 1334–1373, 2016

    work page 2016

  6. [6]

    Transfer learning for reinforcement learning domains: A survey,

    M. E. Taylor and P. Stone, “Transfer learning for reinforcement learning domains: A survey,” Journal of Machine Learning Research , vol. 10, pp. 1633–1685, 2009

    work page 2009

  7. [7]

    Progressive neural networks,

    A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, “Progressive neural networks,” arXiv preprint arXiv:1606.04671 , 2016

    arXiv 2016

  8. [8]

    Learning to push by grasping: Using multiple tasks for effective learning,

    L. Pinto and A. Gupta, “Learning to push by grasping: Using multiple tasks for effective learning,” in IEEE International Conference on Robotics and Automation (ICRA) , 2017

    work page 2017

Show all 64 references
  1. [9]

    Successor features for transfer in reinforcement learning,

    A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. van Hasselt, and D. Silver, “Successor features for transfer in reinforcement learning,” in Neural Information Processing Systems (NeurIPS) , 2017

    2017

  2. [10]

    Zero-shot task generalization with multi-task deep reinforcement learning,

    J. Oh, S. Singh, H. Lee, and P. Kohli, “Zero-shot task generalization with multi-task deep reinforcement learning,” in International Conference on Machine Learning (ICML) , 2017

    2017

  3. [11]

    Synthesize policies for transfer and adaptation across tasks and environments,

    H. Hu, L. Chen, B. Gong, and F. Sha, “Synthesize policies for transfer and adaptation across tasks and environments,” in Neural Information Processing Systems (NeurIPS) , 2018

    2018

  4. [12]

    Mix & match - agent curricula for reinforcement learning,

    W. Czarnecki, S. Jayakumar, M. Jaderberg, L. Hasenclever, Y . W. Teh, N. Heess, S. Osindero, and R. Pascanu, “Mix & match - agent curricula for reinforcement learning,” in International Conference on Machine Learning (ICML), 2018

    2018

  5. [13]

    Transfer of samples in policy search via multiple importance sampling,

    A. Tirinzoni, M. Salvini, and M. Restelli, “Transfer of samples in policy search via multiple importance sampling,” in International Conference on Machine Learning (ICML) , 2019. 14

    2019

  6. [14]

    EPOpt: Learning robust neural network policies using model ensembles,

    A. Rajeswaran, S. Ghotra, B. Ravindran, and S. Levine, “EPOpt: Learning robust neural network policies using model ensembles,” in International Conference on Learning Representations (ICLR) , 2017

    2017

  7. [15]

    CAD2RL: Real single-image flight without a single real image,

    F. Sadeghi and S. Levine, “CAD2RL: Real single-image flight without a single real image,” in Robotics: Science and Systems , 2017

    2017

  8. [16]

    Sim-to- real transfer of robotic control with dynamics randomization,

    X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to- real transfer of robotic control with dynamics randomization,” in IEEE International Conference on Robotics and Automation (ICRA) , 2018

    2018

  9. [17]

    Policy transfer with strategy optimization,

    W. Yu, C. K. Liu, and G. Turk, “Policy transfer with strategy optimization,” in International Conference on Learning Representations (ICLR) , 2019

    2019

  10. [18]

    Alignment-based transfer learning for robot models,

    B. B ´ocsi, L. Csat ´o, and J. Peters, “Alignment-based transfer learning for robot models,” in International Joint Conference on Neural Networks (IJCNN), 2013

    2013

  11. [19]

    Unsupervised cross- domain transfer in policy gradient reinforcement learning via manifold alignment,

    H. B. Ammar, E. Eaton, P. Ruvolo, and M. E. Taylor, “Unsupervised cross- domain transfer in policy gradient reinforcement learning via manifold alignment,” in AAAI Conference on Artificial Intelligence (AAAI) , 2015

    2015

  12. [20]

    A preliminary study of transfer learning between unicycle robots,

    K. V . Raimalwala, B. A. Francis, and A. P. Schoellig, “A preliminary study of transfer learning between unicycle robots,” in AAAI Spring Symposium Series, 2016

    2016

  13. [21]

    Learning modular neural network policies for multi-task and multi-robot transfer,

    C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine, “Learning modular neural network policies for multi-task and multi-robot transfer,” in IEEE International Conference on Robotics and Automation (ICRA) , 2017

    2017

  14. [22]

    Learning invariant feature spaces to transfer skills with reinforcement learning,

    A. Gupta, C. Devin, Y . Liu, P. Abbeel, and S. Levine, “Learning invariant feature spaces to transfer skills with reinforcement learning,” in International Conference on Learning Representations (ICLR) , 2017

    2017

  15. [23]

    Hardware conditioned policies for multi-robot transfer learning,

    T. Chen, A. Murali, and A. Gupta, “Hardware conditioned policies for multi-robot transfer learning,” in Neural Information Processing Systems (NeurIPS), 2018

    2018

  16. [24]

    NerveNet: Learning structured policy with graph neural networks,

    T. Wang, R. Liao, J. Ba, and S. Fidler, “NerveNet: Learning structured policy with graph neural networks,” in International Conference on Learning Representations (ICLR) , 2018

    2018

  17. [26]

    Exploiting hierarchy for learning and transfer in KL-regularized RL,

    D. Tirumala, H. Noh, A. Galashov, L. Hasenclever, A. Ahuja, G. Wayne, R. Pascanu, Y . W. Teh, and N. Heess, “Exploiting hierarchy for learning and transfer in KL-regularized RL,” arXiv:1903.07438, 2019

    1903 arXiv

  18. [27]

    A survey of climbing robots: Locomotion and adhesion,

    B. Chu, K. J. C.-S. Han, and D. Hong, “A survey of climbing robots: Locomotion and adhesion,” International Journal of Precision Engineering and Manufacturing , vol. 11, pp. 633–647, 2010

    2010

  19. [28]

    Dual arm manipulation - a survey,

    C. Smith, Yiannis, Karayiannidis, L. Nalpantidis, X. Gratal, P. Qi, D. V . Dimarogonas, and D. Kragic, “Dual arm manipulation - a survey,” Robotics and Autonomous Systems , vol. 60, pp. 1340–1353, 2012

    2012

  20. [29]

    Auto-encoding variational bayes,

    D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in International Conference on Learning Representations (ICLR) , 2014

    2014

  21. [30]

    Stochastic backprop- agation and approximate inference in deep generative models,

    D. J. Rezende, S. Mohamed, and D. Wierstra, “Stochastic backprop- agation and approximate inference in deep generative models,” in International Conference on Machine Learning (ICML) , 2014

    2014

  22. [31]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,” arXiv preprint arXiv:1707.06347 , 2017

    2017 arXiv

  23. [32]

    Between mdps and semi-mdps: a framework for temporal abstraction in reinforcement learning,

    R. S. Sutton, D. Precup, and S. Singh, “Between mdps and semi-mdps: a framework for temporal abstraction in reinforcement learning,” Artificial Intelligence, vol. 112, pp. 181–211, 1999

    1999

  24. [33]

    Hierarchical deep reinforcement learning: Integrating temporal abstrac- tion and intrinsic motivation,

    T. D. Kulkarni, K. R. Narasimhan, A. Saeedi, and J. B.Tenenbaum, “Hierarchical deep reinforcement learning: Integrating temporal abstrac- tion and intrinsic motivation,” in Neural Information Processing Systems (NeurIPS), 2016

    2016

  25. [34]

    The option-critic architecture,

    P.-L. Bacon, J. Harb, and D. Precup, “The option-critic architecture,” in AAAI Conference on Artificial Intelligence (AAAI) , 2017

    2017

  26. [35]

    FeUdal networks for hierarchical rein- forcement learning,

    A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu, “FeUdal networks for hierarchical rein- forcement learning,” in International Conference on Machine Learning (ICML), 2017

    2017

  27. [36]

    Modular multitask reinforcement learning with policy sketches,

    J. Andreas, D. Klein, and S. Levine, “Modular multitask reinforcement learning with policy sketches,” in International Conference on Machine Learning (ICML), 2017

    2017

  28. [37]

    Data-efficient hierarchical reinforcement learning,

    O. Nachum, S. Gu, H. Lee, and S. Levine, “Data-efficient hierarchical reinforcement learning,” in Neural Information Processing Systems (NeurIPS), 2018

    2018

  29. [38]

    RL2: Fast reinforcement learning via slow reinforcement learning,

    Y . Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel, “RL2: Fast reinforcement learning via slow reinforcement learning,” in International Conference on Learning Representations (ICLR) , 2016

    2016

  30. [39]

    Model-agnostic meta-learning for fast adaptation of deep networks,

    C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in International Conference on Machine Learning (ICML), 2017

    2017

  31. [40]

    A simple neural attentive meta-learner,

    N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel, “A simple neural attentive meta-learner,” in International Conference on Learning Representations (ICLR), 2018

    2018

  32. [41]

    Bayesian model-agnostic meta-learning,

    J. Yoon, T. Kim, O. Dia, S. Kim, Y . Bengio, and S. Ahn, “Bayesian model-agnostic meta-learning,” in Neural Information Processing Systems (NeurIPS), 2018

    2018

  33. [42]

    Recasting gradient-based meta-learning as hierarchical bayes,

    E. Grant, C. Finn, S. Levine, T. Darrell, and T. Griffiths, “Recasting gradient-based meta-learning as hierarchical bayes,” in International Conference on Learning Representations (ICLR) , 2018

    2018

  34. [43]

    Learning to adapt in dynamic, real-world environments through meta-reinforcement learning,

    A. Nagabandi, I. Clavera, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn, “Learning to adapt in dynamic, real-world environments through meta-reinforcement learning,” in International Conference on Learning Representations (ICLR) , 2019

    2019

  35. [44]

    Embed to control: A locally linear latent dynamics model for control from raw images,

    M. Watter, J. T. Springenberg, J. Boedecker, and M. Riedmiller, “Embed to control: A locally linear latent dynamics model for control from raw images,” in Neural Information Processing Systems (NeurIPS) , 2015

    2015

  36. [45]

    Dis- entangling factors of variation in deep representations using adversarial training,

    M. Mathieu, J. Zhao, P. Sprechmann, A. Ramesh, and Y . LeCun, “Dis- entangling factors of variation in deep representations using adversarial training,” in Neural Information Processing Systems (NeurIPS) , 2016

    2016

  37. [46]

    Beta-vae: Learning basic visual concepts with a constrained variational framework,

    I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner, “Beta-vae: Learning basic visual concepts with a constrained variational framework,” in International Conference on Learning Representations (ICLR) , 2017

    2017

  38. [47]

    A hierarchical latent variable encoder-decoder model for generating dialogues,

    I. V . Serban, A. Sordoni, R. Lowe, L. Charlin, J. Pineau, A. Courville, and Y . Bengio, “A hierarchical latent variable encoder-decoder model for generating dialogues,” in AAAI Conference on Artificial Intelligence (AAAI), 2017

    2017

  39. [48]

    Disentan- gling factors of variation with cycle-consistent variational auto-encoders,

    A. H. Jha, S. Anand, M. Singh, and V . S. R. Veeravasarapu, “Disentan- gling factors of variation with cycle-consistent variational auto-encoders,” in European Conference on Computer Vision (ECCV) , 2018

    2018

  40. [49]

    Life-long disentangled representation learning with cross- domain latent homologies,

    A. Achille, T. Eccles, L. Matthey, C. P. Burgess, N. Watters, A. Lerchner, and I. Higgins, “Life-long disentangled representation learning with cross- domain latent homologies,” in Neural Information Processing Systems (NeurIPS), 2018

    2018

  41. [50]

    Learning disentangled joint continuous and discrete repre- sentations,

    E. Dupont, “Learning disentangled joint continuous and discrete repre- sentations,” in Neural Information Processing Systems (NeurIPS) , 2018

    2018

  42. [51]

    Visual reinforcement learning with imagined goals,

    A. Nair, V . Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine, “Visual reinforcement learning with imagined goals,” in Neural Information Processing Systems (NeurIPS) , 2018

    2018

  43. [52]

    Learning discrete and continuous factors of data via alternating disentanglement,

    Y . Jeong and H. O. Song, “Learning discrete and continuous factors of data via alternating disentanglement,” in International Conference on Machine Learning (ICML) , 2019

    2019

  44. [53]

    DARLA: Improving zero- shot transfer in reinforcement learning,

    I. Higgins, A. Pal, A. A. Rusu, L. Matthey, C. P. Burgess, A. Pritzel, M. Botvinick, C. Blundell, and A. Lerchner, “DARLA: Improving zero- shot transfer in reinforcement learning,” in International Conference on Machine Learning (ICML) , 2017

    2017

  45. [54]

    Learning an embedding space for transferable robot skills,

    K. Hausman, J. T. Springenberg, Z. Wang, N. Heess, and M. Ried- miller, “Learning an embedding space for transferable robot skills,” in International Conference on Learning Representations (ICLR) , 2018

    2018

  46. [55]

    Transfer learning from synthetic to real images using variational autoencoders for precise position detection,

    T. Inoue, S. Choudhury, G. D. Magistris, and S. Dasgupta, “Transfer learning from synthetic to real images using variational autoencoders for precise position detection,” in IEEE International Conference on Image Processing (ICIP), 2018

    2018

  47. [56]

    Efficient model-based deep reinforcement learning with variational state tabulation,

    D. Corneil, W. Gerstner, and J. Brea, “Efficient model-based deep reinforcement learning with variational state tabulation,” in International Conference on Machine Learning (ICML) , 2018

    2018

  48. [57]

    Recurrent world models facilitate policy evolution,

    D. Ha and J. Schmidhuber, “Recurrent world models facilitate policy evolution,” in Neural Information Processing Systems (NeurIPS) , 2018

    2018

  49. [58]

    Transfer of value functions via variational methods,

    A. Tirinzoni, R. R. Sanchez, and M. Restelli, “Transfer of value functions via variational methods,” in Neural Information Processing Systems (NeurIPS), 2018

    2018

  50. [59]

    Variational autoencoder for deep learning of images, labels and captions,

    Y . Pu, Z. Gan, R. Henao, X. Yuan, C. Li, A. Stevens, and L. Carin, “Variational autoencoder for deep learning of images, labels and captions,” in Neural Information Processing Systems (NeurIPS) , 2016

    2016

  51. [60]

    Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space,

    L. Wang, A. G. Schwing, and S. Lazebnik, “Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space,” in Neural Information Processing Systems (NeurIPS) , 2017

    2017

  52. [61]

    Cross-domain transfer in reinforcement learning using target apprentice,

    G. Joshi and G. Chowdhary, “Cross-domain transfer in reinforcement learning using target apprentice,” in International Conference on Robotics and Automation (ICRA) , 2018

    2018

  53. [62]

    Mujoco: A physics engine for model- based control,

    E. Todorov, T. Erez, and Y . Tassa, “Mujoco: A physics engine for model- based control,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2012

    2012

  54. [63]

    Openai gym,

    G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” 2016. 15

    2016

  55. [64]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations (ICLR) , 2015

    2015

  56. [65]

    Visualizing data using t-SNE,

    L. van der Maaten and G. Hinton, “Visualizing data using t-SNE,”Journal of Machine Learning Research , vol. 9, pp. 2579–2605, 2008

    2008

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.