{"id":"26097836-f0dd-4b79-9d5e-779c14e7d898","arxiv_id":"1909.02291","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TRACE learns action embeddings from a transition model and transfers policies across reinforcement learning tasks with different state and action spaces, improving sample efficiency in experiments.","lead":"Researchers propose TRACE, a reinforcement learning method that learns a shared action embedding space from the effects actions have on the environment, then uses it to transfer a trained policy to new tasks with different action sets and state descriptions. The approach matters because sample-efficient transfer across differently configured tasks is a practical bottleneck in applying reinforcement learning to games and robotics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-domain transfer fine-tunes the transition model, so no mechanism ensures target action embeddings align with source embeddings; the competitive TRACE-P ablation suggests the gains may not come from action-embedding transfer.","rationale":"The reader's weakest assumption concerned the informal requirement that source and target MDPs share reward and transition similarities and a common state embedding space. My review finds a more specific gap in the cross-domain procedure: even when such similarities hold, the paper's design fine-tunes the transferred transition model rather than freezing it, so the fixed \"semantic translator\" that makes same-domain action-embedding alignment work is absent in cross-domain. With θD and the randomly initialized state embedding both adapting, there is no objective that keeps target action embeddings aligned to source action embeddings, and the nearest-neighbor policy mapping at initialization operates on random target embeddings. This is not an external disagreement with consensus but an internal tension between the claimed mechanism and the algorithm as stated. The ablation TRACE-P being competitive with TRACE-PT in cross-domain suggests the additional transition-model transfer may not be doing the alignment work, which directly tests the paper's explanatory claim. I do not conclude the empirical acceleration results are fabricated; the experiments are plausible and the same-domain results are well supported. The concern is that the cross-domain results may support a weaker claim than the one advertised. I also noted a potential pseudocode inconsistency: Algorithm 1 line 8 stores the proto-action â_t rather than the executed action a_t, which, if taken literally, prevents the transition-model update from using the action embedding matrix Wae; this is likely a typographical slip but should be corrected for reproducibility. Overall, the reader's CONDITIONAL verdict remains appropriate; the proposed check would determine whether the cross-domain mechanism is real or whether the paper should be reframed as a general fine-tuning method with an action-embedding policy head.","tokens_in":14205,"tokens_out":8772,"duration_ms":105107,"concrete_test":"Run the mP←rDP transfer setting from Figure 5(d), or a synthetic cross-domain MDP with a known ground-truth action correspondence. Train TRACE-PT and TRACE-P with the paper's hyperparameters, then: (1) compute, for each learned target action embedding, its nearest neighbor among source action embeddings in the shared R^3 space and compare the resulting mapping against the known physical force ordering, reporting top-1 matching accuracy; (2) compare mean return over the first 200 episodes for TRACE-PT versus TRACE-P. If matching accuracy is near chance, or if TRACE-PT does not clearly beat TRACE-P early in training, the cross-domain transfer gain is not attributable to action-embedding alignment, which would undercut the paper's central mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The same-domain transfer mechanism in §4.3 is coherent: freezing θD_S and optimizing target action embeddings against equation (A.1) forces each target action to occupy the source-embedding location whose transition effect it reproduces, as visualized in Figure 3(b). The central claim, however, includes cross-domain transfer with different state spaces, and there the paper explicitly abandons this mechanism. Section 4.3 states \"the transition model parameters are not frozen but finetuned,\" and Appendix D reports that freezing leads to unstable training or negative transfer. Once θD is fine-tuned, the target action embeddings are optimized against a moving target: the fine-tuned transition model and a randomly reinitialized state embedding fθse_T. There is no loss term, constraint, or fixed reference that keeps e_T(a_T) aligned with e_S(a_S). The \"common state embedding space\" is not aligned by any objective; fθse_T is trained only by the SAC policy loss and the transition prediction loss. The paper concedes there are no jump-starts in the curves because target action embeddings are randomly initialized, so the transferred policy's proto-actions initially select target actions by nearest neighbor to random vectors. Any subsequent alignment is emergent, not guaranteed by the construction. The paper's own cross-domain ablation (Section 5.4, Fig. 5(b)) shows that TRACE-P, which transfers only the policy and randomizes the transition model, is competitive with TRACE-PT. This is consistent with the possibility that the observed speedups come from generic parameter initialization or the action-embedding policy head, rather than from transferring action semantics through the transition model. Thus the cross-domain half of the central claim rests on an unverified assumption that the fine-tuned pipeline preserves the source action-embedding geometry.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TRACE, a transfer-learning framework for deep RL that learns action embeddings through a forward transition model and uses them together with a nearest-neighbor policy mapping to transfer across tasks with different action spaces and, in the cross-domain variant, different state spaces. Source-task training jointly optimizes a SAC policy over continuous proto-actions and a variational transition model that predicts next states from state and action embeddings. For same-domain transfer, the source transition model is frozen and target action embeddings are fitted to it; for cross-domain transfer, the transition model is fine-tuned while state and action embeddings are reinitialized. Experiments compare TRACE-PT with SAC, a basic parameter-transfer baseline (BT), and MIKT on gridworld navigation, discretized Mujoco/Roboschool pendulum tasks, and a commercial-game combat scenario, with ablations that transfer only the policy (TRACE-P) or only the transition model (TRACE-T), plus a study of action-embedding dimension.","tokens_in":14446,"tokens_out":6328,"duration_ms":68683,"significance":"If the claimed effects hold, the central idea---that action semantics can be captured by transition effects and used for action-space transfer---is a useful and generally applicable contribution to RL transfer learning. The paper deserves credit for a broad evaluation across three very different domains, for including explicit ablations of the two transferred components, for visualizing learned embeddings (Figures 3 and 6), and for transparently reporting in the appendix that freezing the transition model in cross-domain transfer leads to unstable training or negative transfer. These strengths make the empirical phenomenon credible at the level of overall sample-efficiency gains. However, the specific claim that cross-domain transfer works through action-embedding alignment is not directly supported: the construction provides no explicit alignment term between source and target action embeddings, the closest existing method (Zhang et al., 2021) is not benchmarked, and one of the three evaluation domains contains unresolved numeric inconsistencies. The significance of the paper is therefore conditional and needs to be strengthened with additional analysis and experiments.","major_comments":[{"comment":"In cross-domain transfer, the mechanism that aligns the target action embeddings with the source action embeddings is not guaranteed by the construction. The transition model parameters are fine-tuned and the state embedding is randomly reinitialized, so the target action embeddings are optimized against a moving reference rather than against the fixed source embedding structure. There is no loss term, constraint, or fixed reference that keeps e_T(a_T) aligned with e_S(a_S), and the transferred policy initially selects actions by nearest neighbor to randomly initialized vectors. The paper's own ablation (Section 5.4 and Figure 10) shows that TRACE-P, which transfers only the policy and reinitializes the transition model, is competitive with TRACE-PT, especially in Figure 5(b). This is consistent with the speedup coming substantially from policy initialization rather than from the proposed transition-model-based embedding alignment. Please provide direct evidence of cross-domain action-embedding alignment (e.g., quantitative correspondence between e_T and e_S, or a variant that enforces alignment through a frozen anchor or an auxiliary alignment loss), or explicitly restrict the claim about action-embedding transfer to the same-domain setting.","section":"Section 4.3, Algorithm 2, Appendix D"},{"comment":"The closest prior method, Zhang et al. (2021), learns state and action correspondence across domains using a cycle-consistency constraint on dynamics; this is the same mechanism family as TRACE, and it is cited in the related-work section but never compared in the experiments. Without this baseline, the claim that TRACE outperforms state-of-the-art transfer algorithms is not established. Add a comparison to Zhang et al. (2021) on at least the gridworld and pendulum tasks, or revise the claim to refer only to the baselines actually evaluated.","section":"Section 2.1 and Section 5"},{"comment":"The combat-task domain description is internally inconsistent in the number of skills and actions: Section 5.3 says the action-space size is 10 for both classes, then states that the authors 'randomly sample 5 out of 15 skills'; Table 4 lists 14 skills for She Shou; and Appendix A.4 says the action space contains 'six unique skills' plus common operations. These numbers cannot all be reconciled from the text, which prevents reproduction of the commercial-game experiment and weakens the quantitative claims made for that domain. Clarify the exact skill inventory for each class, how the 10-element action space is composed, and how the 5-of-15 sampling step is used.","section":"Section 5.3, Table 4, Appendix A.4"}],"minor_comments":[{"comment":"The algorithm says to update the transition model and action embeddings 'over Equation. (A.1)', but Equation (A.1) does not appear in the text; the only transition-model loss is Equation (1). Correct the cross-reference.","section":"Algorithm 1, line 12; Section 4.1"},{"comment":"The text refers to 'Figure 10' when discussing the main cross-domain results, but the main-text figures are in Figure 5 and the appendix figures in Figure 10. Please renumber or correct the references so the reader is directed to the right figure.","section":"Section 5.2 and Section 5.4"},{"comment":"The method is called TRACE-PT, TRACE-P, and TRACE-T in Section 5, but the appendix uses the name AE-SAC-PT (e.g., Section B.2). Unify the terminology.","section":"Section 5 vs. Appendix B"},{"comment":"The sentence 'SAC performs better than TRACE in all tasks' should be qualified as 'TRACE without transfer', otherwise it appears to contradict the main comparison in which TRACE-PT outperforms SAC.","section":"Section 5.1"},{"comment":"The reference in Section 5.2 to 'the learned action embeddings of the environments are shown in Figure 4 of Appendix' does not match the appendix numbering; the PCA plot appears as Figure 11 in Appendix B. Please fix the reference.","section":"Appendix B.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a 2021 revision of a 2019 preprint, and it cites Zhang et al. (2021) in related work. Since that cited method is nearly contemporary and directly addresses cross-domain state-action correspondence via dynamics consistency, omitting it from the experiments will be seen by readers as a serious gap, even if the authors believe their action-embedding framing is different. The combat-domain numeric inconsistencies also need to be resolved before the paper can be considered reproducible. If the authors can add the missing baseline and provide direct evidence of cross-domain embedding alignment, the paper could become a solid contribution; without those, the central novelty claim is not fully supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read Chen et al.'s TRACE. The core idea is new: learn action embeddings through a forward transition model and use them with a nearest-neighbor policy to transfer across tasks with different action spaces. That's a sensible departure from inverse-dynamics and sequence-based action representations, and the same-domain version is well worked out. When state spaces match, freezing the source transition model and optimizing target action embeddings against it gives a concrete alignment mechanism, and the gridworld experiments convincingly show that transfer is faster than from-scratch SAC and that the learned embeddings align semantically (the e(↑) ≈ 0.5*e(↑↑)+0.5*e(↑↓) observation is a nice touch).\n\nThe soft spots are where the claims outrun the evidence. For cross-domain transfer (different state spaces), the paper explicitly fine-tunes the transition model and randomly reinitializes the state embedding, so there is no constraint holding target action embeddings to the source geometry. Alignment is emergent at best, and the paper's own ablation shows TRACE-P (policy-only transfer, random transition model) is competitive with TRACE-PT in several cross-domain tasks. That means the speedup could come from generic parameter initialization or the action-embedding policy head, not from transferring action semantics through the transition model. The stress-test note lands here.\n\nAlso: the closest method, Zhang et al.'s dynamics cycle-consistency, is cited but never benchmarked. The combat task description has a skill-count inconsistency (10 actions vs. 15 skills sampled vs. 14 listed), and the commercial environment is unavailable. No code is released. These are fixable in revision but currently make it hard to pin down exactly what is being claimed.\n\nThe paper is honestly written and the appendices are thorough. The core mechanism is plausible and demonstrated in the same-domain setting. It's not a conceptual breakthrough, but it is a solid contribution to transfer RL.\n\nVerdict: deserves serious peer review. I'd send it to a competent referee with a request to push on the cross-domain ablations and the missing Zhang et al. baseline. It shouldn't be desk-rejected; it's a reasonable conditional accept with major revision.","headline":"TRACE is a worthwhile transfer-RL paper with a genuinely new mechanism for same-domain action-embedding transfer, but its cross-domain claim rests on a thinner empirical base than the abstract suggests.","tokens_in":15091,"tokens_out":3289,"would_cite":false,"duration_ms":32760,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes TRACE, a transfer-learning framework that lets a policy trained on one reinforcement-learning task be reused on a related task even when the two tasks have different state sets and different action sets, by learning…","keywords":["transfer learning","reinforcement learning","action embeddings","state-action space mismatch","transition model","sample efficiency","policy transfer","deep reinforcement learning"],"falsifier":"Construct a source and target task with identical state spaces and rewards but with the target action set being a permutation of the source action effects, so that the target's 'damage' skill is actually a stun; if TRACE-PT still transfers quickly, the effect-based alignment story is wrong, and if it fails, the similarity premise is confirmed as load-bearing.","tokens_in":13968,"feed_emoji":"🎮","tokens_out":5727,"duration_ms":54745,"temperature":0.7,"pith_summary":"The paper proposes TRACE, a transfer-learning framework for deep reinforcement learning that lets a policy trained on one task be reused on a related task even when the two tasks have different state spaces and different action spaces. Its central bet is that actions are best described by their effects: two skills in different action spaces that produce similar state transitions should sit close together in a shared action-embedding space. TRACE learns these embeddings from a forward transition model that predicts the next state from the current state and the action embedding, and the policy picks real actions by nearest neighbor in that space. On a target task, the source policy and transition model are transferred as initializations, and the reported experiments on gridworld, pendulum-control, and commercial-game combat tasks show faster learning than training from scratch or using a basic parameter-initialization transfer baseline.","feed_headline":"Action embeddings transfer RL skills across different action spaces","feed_subtitle":"A policy knows which target action to pick because similar actions land close in the learned embedding space.","key_machinery":"The load-bearing object is the action-embedding matrix $W_{ae} \\in \\mathbb{R}^{|A| \\times d}$, learned jointly with a stochastic transition model $f_{\\theta_D}$ that predicts the next state from the current state, the action embedding, and a latent variable sampled from a variational encoder. The key objective is the prediction loss $\\mathbb{E}[\\|\\tilde{s}_{t+1}-s_{t+1}\\|_2^2 + \\beta D_{KL}(\\mathcal{N}(\\mu_t,\\sigma_t)\\mid\\mid \\mathcal{N}(0,I))]$, which ties an action's representation to its observable effect on the environment. The nearest-neighbor rule $g(\\hat{a}) = \\arg\\min_{a \\in A} \\|\\hat{a}-e(a)\\|_2$ then lets a policy output a proto-action in continuous embedding space and execute the discrete action with the most similar effect. Transferring the transition model's parameters, frozen in same-domain transfer and fine-tuned in cross-domain transfer, is what aligns the target action embeddings with the source ones.","core_discovery":"The central claim is that a forward dynamics model can serve as the carrier of action semantics across domains. The paper argues that if the source and target transition functions are similar, then optimizing action embeddings so that $\\tilde{s}_{t+1} \\approx s_{t+1}$ makes semantically similar actions occupy nearby points in a common embedding space, even across different action sets. On the target task, initializing the policy and transition model with source parameters and re-learning only the target action embeddings and, in cross-domain cases, the state embedding aligns the two domains well enough that the nearest-neighbor action mapping selects the correct target actions. The reported consequence is improved sample efficiency: TRACE-PT, which transfers both the policy and the transition model, reaches higher return or win rate in fewer episodes than soft actor-critic from scratch, a basic transfer baseline, and a mutual-information-based transfer baseline across all tested tasks, including cases where the basic baseline shows negative transfer.","pith_inferences":["If action embeddings truly capture transition semantics, the same transition-model objective should produce transferable action representations for continuous or parametrized action spaces, which the paper itself lists as future work.","The method implicitly assumes that a single transition model can predict both domains after a learned state embedding, so a natural diagnostic that the paper does not run is to measure the transferred transition model's prediction error on target-task data before committing to policy transfer.","The nearest-neighbor policy mapping treats action choice as a lookup in effect space, so the same machinery could in principle select actions that were never in the target action set, as long as their embeddings are learned.","A sharper boundary condition would be to test on a pair of tasks whose state spaces can be embedded into a common space but whose transition functions differ by a reordering of action effects; such a permutation should break alignment and produce negative transfer."],"forward_implications":["In same-domain transfer, freezing the transferred transition model and relearning only the target action embeddings is enough to align action semantics and accelerate policy learning.","In cross-domain transfer, jointly retraining a common state embedding with the transferred transition model lifts the alignment, allowing policies to transfer even when state dimensions and physical engines differ.","The method yields action embeddings that organize by effect: in the n-step gridworld, actions sharing the same net displacement cluster together, and the cluster layout mirrors the grid's directional symmetry.","The transfer gain comes mainly from the transferred policy combined with the action representation; transferring the transition model alone gives performance close to no-transfer, suggesting that action embeddings boost policy generalization.","A basic transfer baseline that keeps the network and replaces only the input and output layers often produces negative transfer, while TRACE-PT accelerates learning in all tested settings."],"supporting_citations":[{"why":"Supplies the nearest-neighbor mapping from continuous proto-actions to discrete actions that TRACE uses to select real actions from the embedding space.","marker":"[Dulac-Arnold et al., 2015]"},{"why":"Provides the soft actor-critic (SAC) algorithm and loss that serve as the base RL training procedure for TRACE and all compared methods.","marker":"[Haarnoja et al., 2018]"},{"why":"Supplies the variational autoencoder-style latent variable and KL-divergence objective used in the stochastic transition model.","marker":"[Kingma and Welling, 2013]"},{"why":"Defines MIKT, the mutual-information-based transfer baseline under state-action dimension mismatch that TRACE must outperform.","marker":"[Wan et al., 2020]"},{"why":"Provides the MuJoCo physics engine and pendulum environments used as cross-domain transfer testbeds.","marker":"[Todorov et al., 2012]"},{"why":"Supplies the technique of using a stochastic latent variable in recurrent transition models, which TRACE adopts for stochastic environments.","marker":"[Goyal et al., 2017]"},{"why":"Contributes the idea of learning action representations from dynamics, which TRACE extends from inverse dynamics to a forward transition model.","marker":"[Chandak et al., 2019]"}],"fun_headline_variants":["Action embeddings align semantics across RL action spaces","Transfer RL policies by embedding actions via dynamics model","Dynamics model embeds action semantics for cross-domain RL transfer","Action embedding space makes cross-task RL transfer work","Similar actions cluster in embedding space for RL transfer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The source and target tasks must share enough similarity in reward and transition structure, and their states must be projectable into a common embedding space; if those conditions fail, the learned action embeddings will not align across tasks and the nearest-neighbor policy will pick the wrong target actions.","fun_headline_variants_meta":{"raw":{"variants":["Action embeddings align semantics across RL action spaces","Transfer RL policies by embedding actions via dynamics model","Dynamics model embeds action semantics for cross-domain RL transfer","Action embedding space makes cross-task RL transfer work","Similar actions cluster in embedding space for RL transfer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000246,"raw_usage":{"total_tokens":1485,"prompt_tokens":833,"completion_tokens":652,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":579}},"tokens_in":449,"tokens_out":652,"duration_ms":6780,"temperature":1.0,"reasoning_tokens":579,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:54:23.260445+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a source and target task with identical state spaces and rewards but with the target action set being a permutation of the source action effects, so that the target's 'damage' skill is actually a stun; if TRACE-PT still transfers quickly, the effect-based alignment story is wrong, and if it fails, the similarity premise is confirmed as load-bearing.","supporting_citations":[{"cited_title":"Mutual information based knowledge transfer under state- action dimension mismatch","cited_arxiv_id":null,"evidence_quote":"Defines MIKT, the mutual-information-based transfer baseline under state-action dimension mismatch that TRACE must outperform."},{"cited_title":"Mujoco: A physics engine for model-based control","cited_arxiv_id":null,"evidence_quote":"Provides the MuJoCo physics engine and pendulum environments used as cross-domain transfer testbeds."},{"cited_title":"Z-forcing: Training stochastic recurrent networks","cited_arxiv_id":null,"evidence_quote":"Supplies the technique of using a stochastic latent variable in recurrent transition models, which TRACE adopts for stochastic environments."},{"cited_title":"Jordan, and Philip S","cited_arxiv_id":null,"evidence_quote":"Contributes the idea of learning action representations from dynamics, which TRACE extends from inverse dynamics to a forward transition model."}],"review_version":1}