REVIEW 3 major objections 5 minor 1 cited by
Learning Action-Transferable Policy with Action Embedding
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper proposes TRACE, a transfer-learning framework that lets a policy trained on one reinforcement-learning task be reused on a related task even when the two tasks have different state sets and different action sets, by learning…
desk verdict TRACE is a worthwhile transfer-RL paper with a genuinely new mechanism for same-domain action-embedding transfer, but its cross-domain claim rests on a thinner empirical base than the abstract suggests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the action-embedding matrix $W_{ae} \in \mathbb{R}^{|A| \times d}$, learned jointly with a stochastic transition model $f_{\theta_D}$ that predicts the next state from the current state, the action embedding, and a latent variable sampled from a variational encoder. The key objective is the prediction loss $\mathbb{E}[\|\tilde{s}_{t+1}-s_{t+1}\|_2^2 + \beta D_{KL}(\mathcal{N}(\mu_t,\sigma_t)\mid\mid \mathcal{N}(0,I))]$, which ties an action's representation to its observable effect on the environment. The nearest-neighbor rule $g(\hat{a}) = \arg\min_{a \in A} \|\hat{a}-e(a)\|_2$ then lets a policy output a proto-action in continuous embedding space and execute the discrete action with the most similar effect. Transferring the transition model's parameters, frozen in same-domain transfer and fine-tuned in cross-domain transfer, is what aligns the target action embeddings with the source ones.
What would settle it
Construct a source and target task with identical state spaces and rewards but with the target action set being a permutation of the source action effects, so that the target's 'damage' skill is actually a stun; if TRACE-PT still transfers quickly, the effect-based alignment story is wrong, and if it fails, the similarity premise is confirmed as load-bearing.
Extended reading notes
Core claim
The central claim is that a forward dynamics model can serve as the carrier of action semantics across domains. The paper argues that if the source and target transition functions are similar, then optimizing action embeddings so that $\tilde{s}_{t+1} \approx s_{t+1}$ makes semantically similar actions occupy nearby points in a common embedding space, even across different action sets. On the target task, initializing the policy and transition model with source parameters and re-learning only the target action embeddings and, in cross-domain cases, the state embedding aligns the two domains well enough that the nearest-neighbor action mapping selects the correct target actions. The reported consequence is improved sample efficiency: TRACE-PT, which transfers both the policy and the transition model, reaches higher return or win rate in fewer episodes than soft actor-critic from scratch, a basic transfer baseline, and a mutual-information-based transfer baseline across all tested tasks, including cases where the basic baseline shows negative transfer.
Load-bearing premise
The source and target tasks must share enough similarity in reward and transition structure, and their states must be projectable into a common embedding space; if those conditions fail, the learned action embeddings will not align across tasks and the nearest-neighbor policy will pick the wrong target actions.
Editorial extensions
If this is right
- In same-domain transfer, freezing the transferred transition model and relearning only the target action embeddings is enough to align action semantics and accelerate policy learning.
- In cross-domain transfer, jointly retraining a common state embedding with the transferred transition model lifts the alignment, allowing policies to transfer even when state dimensions and physical engines differ.
- The method yields action embeddings that organize by effect: in the n-step gridworld, actions sharing the same net displacement cluster together, and the cluster layout mirrors the grid's directional symmetry.
- The transfer gain comes mainly from the transferred policy combined with the action representation; transferring the transition model alone gives performance close to no-transfer, suggesting that action embeddings boost policy generalization.
- A basic transfer baseline that keeps the network and replaces only the input and output layers often produces negative transfer, while TRACE-PT accelerates learning in all tested settings.
Reading between the lines
- If action embeddings truly capture transition semantics, the same transition-model objective should produce transferable action representations for continuous or parametrized action spaces, which the paper itself lists as future work.
- The method implicitly assumes that a single transition model can predict both domains after a learned state embedding, so a natural diagnostic that the paper does not run is to measure the transferred transition model's prediction error on target-task data before committing to policy transfer.
- The nearest-neighbor policy mapping treats action choice as a lookup in effect space, so the same machinery could in principle select actions that were never in the target action set, as long as their embeddings are learned.
- A sharper boundary condition would be to test on a pair of tasks whose state spaces can be embedded into a common space but whose transition functions differ by a reordering of action effects; such a permutation should break alignment and produce negative transfer.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes TRACE, a transfer-learning framework for deep RL that learns action embeddings through a forward transition model and uses them together with a nearest-neighbor policy mapping to transfer across tasks with different action spaces and, in the cross-domain variant, different state spaces. Source-task training jointly optimizes a SAC policy over continuous proto-actions and a variational transition model that predicts next states from state and action embeddings. For same-domain transfer, the source transition model is frozen and target action embeddings are fitted to it; for cross-domain transfer, the transition model is fine-tuned while state and action embeddings are reinitialized. Experiments compare TRACE-PT with SAC, a basic parameter-transfer baseline (BT), and MIKT on gridworld navigation, discretized Mujoco/Roboschool pendulum tasks, and a commercial-game combat scenario, with ablations that transfer only the policy (TRACE-P) or only the transition model (TRACE-T), plus a study of action-embedding dimension.
Significance. If the claimed effects hold, the central idea---that action semantics can be captured by transition effects and used for action-space transfer---is a useful and generally applicable contribution to RL transfer learning. The paper deserves credit for a broad evaluation across three very different domains, for including explicit ablations of the two transferred components, for visualizing learned embeddings (Figures 3 and 6), and for transparently reporting in the appendix that freezing the transition model in cross-domain transfer leads to unstable training or negative transfer. These strengths make the empirical phenomenon credible at the level of overall sample-efficiency gains. However, the specific claim that cross-domain transfer works through action-embedding alignment is not directly supported: the construction provides no explicit alignment term between source and target action embeddings, the closest existing method (Zhang et al., 2021) is not benchmarked, and one of the three evaluation domains contains unresolved numeric inconsistencies. The significance of the paper is therefore conditional and needs to be strengthened with additional analysis and experiments.
major comments (3)
- [Section 4.3, Algorithm 2, Appendix D] In cross-domain transfer, the mechanism that aligns the target action embeddings with the source action embeddings is not guaranteed by the construction. The transition model parameters are fine-tuned and the state embedding is randomly reinitialized, so the target action embeddings are optimized against a moving reference rather than against the fixed source embedding structure. There is no loss term, constraint, or fixed reference that keeps e_T(a_T) aligned with e_S(a_S), and the transferred policy initially selects actions by nearest neighbor to randomly initialized vectors. The paper's own ablation (Section 5.4 and Figure 10) shows that TRACE-P, which transfers only the policy and reinitializes the transition model, is competitive with TRACE-PT, especially in Figure 5(b). This is consistent with the speedup coming substantially from policy initialization rather than from the proposed transition-model-based embedding alignment. Please provide direct evidence of cross-domain action-embedding alignment (e.g., quantitative correspondence between e_T and e_S, or a variant that enforces alignment through a frozen anchor or an auxiliary alignment loss), or explicitly restrict the claim about action-embedding transfer to the same-domain setting.
- [Section 2.1 and Section 5] The closest prior method, Zhang et al. (2021), learns state and action correspondence across domains using a cycle-consistency constraint on dynamics; this is the same mechanism family as TRACE, and it is cited in the related-work section but never compared in the experiments. Without this baseline, the claim that TRACE outperforms state-of-the-art transfer algorithms is not established. Add a comparison to Zhang et al. (2021) on at least the gridworld and pendulum tasks, or revise the claim to refer only to the baselines actually evaluated.
- [Section 5.3, Table 4, Appendix A.4] The combat-task domain description is internally inconsistent in the number of skills and actions: Section 5.3 says the action-space size is 10 for both classes, then states that the authors 'randomly sample 5 out of 15 skills'; Table 4 lists 14 skills for She Shou; and Appendix A.4 says the action space contains 'six unique skills' plus common operations. These numbers cannot all be reconciled from the text, which prevents reproduction of the commercial-game experiment and weakens the quantitative claims made for that domain. Clarify the exact skill inventory for each class, how the 10-element action space is composed, and how the 5-of-15 sampling step is used.
minor comments (5)
- [Algorithm 1, line 12; Section 4.1] The algorithm says to update the transition model and action embeddings 'over Equation. (A.1)', but Equation (A.1) does not appear in the text; the only transition-model loss is Equation (1). Correct the cross-reference.
- [Section 5.2 and Section 5.4] The text refers to 'Figure 10' when discussing the main cross-domain results, but the main-text figures are in Figure 5 and the appendix figures in Figure 10. Please renumber or correct the references so the reader is directed to the right figure.
- [Section 5 vs. Appendix B] The method is called TRACE-PT, TRACE-P, and TRACE-T in Section 5, but the appendix uses the name AE-SAC-PT (e.g., Section B.2). Unify the terminology.
- [Section 5.1] The sentence 'SAC performs better than TRACE in all tasks' should be qualified as 'TRACE without transfer', otherwise it appears to contradict the main comparison in which TRACE-PT outperforms SAC.
- [Appendix B.1] The reference in Section 5.2 to 'the learned action embeddings of the environments are shown in Figure 4 of Appendix' does not match the appendix numbering; the PCA plot appears as Figure 11 in Appendix B. Please fix the reference.
Circularity Check
No significant circularity: action embeddings are learned from transition data, transfer is evaluated on target-task returns, and no load-bearing premise reduces to a fitted constant or self-citation.
full rationale
The paper's derivation chain is self-contained. Action embeddings are produced by minimizing the transition-prediction loss in Eq. (1) / (A.1), and the same-domain transfer mechanism freezes the source transition model and fits target action embeddings to that same predictive model, so any resulting alignment is a designed property of the optimization rather than a renamed prediction. Cross-domain transfer fine-tunes the transition model and randomly reinitializes the state and action embeddings, which weakens the theoretical alignment guarantee; the paper itself reports in Appendix D that freezing the transition model leads to unstable training and negative transfer. That is an acknowledged limitation of the approach, not a circular step. The central empirical claims—accelerated policy learning and informative action embeddings—are evaluated through target-task return curves and qualitative/PCA analyses of the learned embeddings, not through quantities that are equal to the training objective by construction. References to prior work by overlapping authors, such as the PTF citation in Section 2.1, are contextual and not load-bearing for the main results. No equation in the paper reduces to its own input, and no fitted parameter is relabeled as a prediction. Therefore the paper exhibits no significant circularity.
Assumptions & free parameters
free parameters (3)
- action embedding dimension d =
2 (gridworld), 3 (Mujoco/Roboschool), 6 (combat)
- state embedding dimension =
5 (Mujoco/Roboschool), 25 (combat); none in gridworld
- KL weight beta =
1e-2 (combat only)
assumptions (4)
- domain assumption Action semantics are captured by state-transition effects: p(st+1|st, at) is approximately p(st+1|st, e(at)).
- domain assumption Source and target tasks have similar reward and transition functions under the learned embeddings.
- domain assumption A common state embedding exists, so a source-trained transition model is predictive in the target domain.
- domain assumption Nearest-neighbor in embedding space is a valid discretization for policy outputs.
Cite this review
Pith. "Pith review of Learning Action-Transferable Policy with Action Embedding." pith.science (2026). https://pith.science/paper/O5ZOWU64
@misc{pith2026190902291,
author = {Pith},
title = {Pith review of: Learning Action-Transferable Policy with Action Embedding},
year = {2026},
howpublished = {\url{https://pith.science/paper/O5ZOWU64}},
note = {Machine review of arXiv:1909.02291}
}
read the original abstract
Transfer learning (TL) is a promising way to improve the sample efficiency of reinforcement learning. However, how to efficiently transfer knowledge across tasks with different state-action spaces is investigated at an early stage. Most previous studies only addressed the inconsistency across different state spaces by learning a common feature space, without considering that similar actions in different action spaces of related tasks share similar semantics. In this paper, we propose a method to learning action embeddings by leveraging this idea, and a framework that learns both state embeddings and action embeddings to transfer policy across tasks with different state and action spaces. Our experimental results on various tasks show that the proposed method can not only learn informative action embeddings but accelerate policy learning.
Figures
Figures from the paper (7 more)
Forward citations
Cited by 1 Pith paper
-
Towards Embodiment Scaling Laws in Robot Locomotion
A policy trained on about one thousand simulated robot bodies generalizes progressively better to unseen bodies as the number of training bodies grows, and it transfers zero-shot to two real robots.
Reference graph
Works this paper leans on
-
[1]
[Ammar et al., 2015] Haitham Bou Ammar, Eric Eaton, Paul Ru- volo, and Matthew E Taylor. Unsupervised cross-domain trans- fer in policy gradient reinforcement learning via manifold align- ment. In Twenty-Ninth AAAI Conference on Artificial Intelli- gence,
work page 2015
-
[4]
[Chandak et al., 2019] Yash Chandak, Georgios Theocharous, James Kostas, Scott M. Jordan, and Philip S. Thomas. Learn- ing action representations for reinforcement learning. In ICML, pages 941–950,
work page 2019
-
[5]
Deep reinforcement learning in large discrete action spaces
[Dulac-Arnold et al., 2015] Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv:1512.07679,
arXiv 2015
-
[6]
Model-agnostic meta-learning for fast adaptation of deep net- works
[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep net- works. In ICML, pages 1126–1135,
work page 2017
-
[8]
Learning invariant feature spaces to transfer skills with reinforcement learning
[Gupta et al., 2017] Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. arXiv preprint arXiv:1703.02949,
arXiv 2017
-
[9]
Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor
[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290,
arXiv 2018
-
[15]
Uni- versal successor representations for transfer reinforcement learn- ing
[Ma et al., 2018] Chen Ma, Junfeng Wen, and Yoshua Bengio. Uni- versal successor representations for transfer reinforcement learn- ing. arXiv preprint arXiv:1804.03758,
arXiv 2018
-
[16]
Efficient estimation of word representations in vector space
[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781,
arXiv 2013
Show all 30 references
-
[17]
Human-level control through deep reinforcement learning
[Mnih et al., 2015] V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostro- vski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529,
2015
-
[18]
Neural network surgery with sets
[Raiman et al., 2019] Jonathan Raiman, Susan Zhang, and Christy Dennison. Neural network surgery with sets. arXiv preprint arXiv:1912.06719,
2019 arXiv
-
[19]
Mastering the game of go with deep neural net- works and tree search
[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural net- works and tree search. natu...
2016
-
[20]
Trans- fer learning for reinforcement learning domains: A survey
[Taylor and Stone, 2009] Matthew E Taylor and Peter Stone. Trans- fer learning for reinforcement learning domains: A survey. JMLR, 10(Jul):1633–1685,
2009
-
[23]
The natural language of actions
[Tennenholtz and Mannor, 2019] Guy Tennenholtz and Shie Man- nor. The natural language of actions. InInternational Conference on Machine Learning, pages 6196–6205,
2019
-
[24]
Mujoco: A physics engine for model-based control
[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In In- ternational Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE,
2012
-
[26]
Dynamics-aware embed- dings
[Whitney et al., 2020] William Whitney, Rajat Agarwal, Kyunghyun Cho, and Abhinav Gupta. Dynamics-aware embed- dings. In International Conference on Learning Representations,
2020
-
[27]
Mutual alignment transfer learning.arXiv preprint arXiv:1707.07907,
[Wulfmeier et al., 2017] Markus Wulfmeier, Ingmar Posner, and Pieter Abbeel. Mutual alignment transfer learning.arXiv preprint arXiv:1707.07907,
2017 arXiv
-
[28]
Efficient deep reinforcement learning through policy transfer
[Yang et al., 2020] Tianpei Yang, Jianye Hao, Zhaopeng Meng, Zongzhang Zhang, Yujing Hu, Yingfeng Chen, Changjie Fan, Weixun Wang, Zhaodong Wang, and Jiajie Peng. Efficient deep reinforcement learning through policy transfer. In Proceedings of the 19th International Conference ...
2020
-
[29]
Learning cross-domain correspon- dence for control with dynamics cycle-consistency
[Zhang et al., 2021] Qiang Zhang, Tete Xiao, Alexei A Efros, Ler- rel Pinto, and Xiaolong Wang. Learning cross-domain correspon- dence for control with dynamics cycle-consistency. ICLR,
2021
-
[30]
Here, we exhibit action embeddings learned from additional state embeddings in Mujoco and Roboschool tasks
B Extended Results B.1 Action Embedding In our paper, we have shown the learned embeddings of gridworld and combat tasks. Here, we exhibit action embeddings learned from additional state embeddings in Mujoco and Roboschool tasks. The Table 5: Parameter settings in combat tasks...
2000
-
[1997]
Gen- eralization to new actions in reinforcement learning
[Jain et al., 2020] Ayush Jain, Andrew Szot, and Joseph Lim. Gen- eralization to new actions in reinforcement learning. In Inter- national Conference on Machine Learning , pages 4661–4672. PMLR,
2020
-
[2007]
Distral: Robust multitask reinforcement learn- ing
[Teh et al., 2017] Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learn- ing. In Advances in Neural Information Processing Systems , pages 4496–4506,
2017
-
[2009]
Transfer learning via inter-task mappings for temporal difference learning
[Taylor et al., 2007] Matthew E Taylor, Peter Stone, and Yaxin Liu. Transfer learning via inter-task mappings for temporal difference learning. JMLR, 8(Sep):2125–2167,
2007
-
[2012]
Mutual information based knowledge transfer under state- action dimension mismatch
[Wan et al., 2020] Michael Wan, Tanmay Gangwani, and Jian Peng. Mutual information based knowledge transfer under state- action dimension mismatch. In Conference on Uncertainty in Artificial Intelligence, pages 1218–1227. PMLR,
2020
-
[2013]
End-to-end training of deep visuomotor poli- cies
[Levine et al., 2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor poli- cies. JMLR, 17(1):1334–1373,
2016
-
[2015]
Transfer in deep reinforce- ment learning using successor features and generalised policy im- provement
[Barreto et al., 2019] Andr´e Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Au- gustin ˇZ´ıdek, and Remi Munos. Transfer in deep reinforce- ment learning using successor features and generalised policy im- provement. arXiv preprint...
2019 arXiv
-
[2016]
Knowledge flow: Improve upon your teachers
[Liu et al., 2019] Iou Jen Liu, Jian Peng, and Alexander G Schwing. Knowledge flow: Improve upon your teachers. In 7th International Conference on Learning Representations, ICLR 2019,
2019
-
[2017]
Z-forcing: Training stochastic recurrent networks
[Goyal et al., 2017] Anirudh Goyal Alias Parth Goyal, Alessandro Sordoni, Marc-Alexandre C ˆot´e, Nan Rosemary Ke, and Yoshua Bengio. Z-forcing: Training stochastic recurrent networks. In Advances in neural information processing systems, pages 6713– 6723,
2017
-
[2018]
Long short-term memory
[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780,
1997
-
[2019]
Domain adaptation for reinforcement learning on the atari
[Carr et al., 2019] Thomas Carr, Maria Chli, and George V ogiatzis. Domain adaptation for reinforcement learning on the atari. In Proceedings of the 18th International Conference on Au- tonomous Agents and MultiAgent Systems , pages 1859–1861,
2019
-
[2020]
Auto-encoding variational bayes
[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114,
2013 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.