{"id":"b4093bb0-4b8b-4b64-aef2-57518a783ae2","arxiv_id":"2411.09891","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"DARAIL transfers a DARC-trained policy from a source domain to a target domain via importance-weighted imitation learning from observation, with a reward-augmented estimator.","lead":"This paper proposes DARAIL, a method that trains a policy in a source environment, then uses imitation learning to transfer its behavior to a target environment with different dynamics. It combines an existing reward-modification scheme (DARC) with observation-based imitation, aiming to outperform both DARC and other off-dynamics baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.1's proof silently sets I1=0, assuming target-optimal and DARC-source rewards coincide; Table 5 also contradicts the empirical claim.","rationale":"The reader's CONDITIONAL verdict is justified, but the weakest assumption is better stated as the proof's I1=0 step rather than merely 'DARC source error B is small.' I1=0 is stronger than small B: it requires exact equality of target-optimal expected reward and DARC-source expected reward, an equality that is asserted without derivation and is not implied by the reverse-KL objective. This is an internal inconsistency, not a disagreement with consensus. Separately, Table 5's Ant row is a concrete empirical contradiction of the paper's headline claim, and the paper does not reconcile it in the main text. These concerns reinforce the reader's conditional verdict rather than overturning it: the DARAIL construction may be salvageable with a corrected theorem, a narrowed empirical claim, and a rerun or explanation of the Ant 0.5-gravity result. I therefore recommend no change to the reader's CONDITIONAL verdict, with the condition made more specific: fix the I1 step in Appendix B and address the Table 5 contradiction before the paper can be accepted.","tokens_in":35421,"tokens_out":6946,"duration_ms":75616,"concrete_test":"Re-derive Theorem 4.1 from Appendix B without the 'I1=0' assertion: define π*_DARC explicitly as the optimizer of the DARC objective in Eq. (3.3) and compute I1 = E_{ptrg,π*}[Σ r] − E_{psrc,π*_DARC}[Σ r + H] under the broken-source Ant/pf=1.0 setting. If I1 cannot be shown to be zero or bounded by the stated B from the stated assumptions, the theorem needs an explicit additional I1 term. Then, using the released code, rerun the Ant 0.5-gravity experiment in Table 5 to confirm whether DARAIL's target reward remains below DARC Evaluation; if it does, the abstract-level empirical claim is false in that setting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing flaw is the zeroing of I1 in the proof of Theorem B.5/4.1. The proof decomposes the target regret into I1+I2+I3 and asserts: 'Based on the formulation, π*_DARC can generate optimal trajectories for the target domain in the source domain so that I1 = 0.' This equality says that the expected entropy-regularized reward of the DARC-optimal policy in the source equals the expected reward of the true target-optimal policy in the target. That is precisely the strong DARC guarantee the paper claims to avoid assuming, and it does not follow from the reverse-KL objective in Eq. (3.3). Reverse-KL matching of the source trajectory distribution to the target-optimal distribution does not imply equality of these expected rewards, especially when the target-optimal policy violates the no-exploit condition — the broken-source regime that motivates the paper. With I1 left nonzero, the bound in Theorem 4.1 is missing an unquantified term, so the claimed relaxation of DARC's assumption is unproven. The same fragility is visible empirically: Table 5 (0.5 gravity) reports DARAIL Ant reward 990±12 versus DARC Evaluation 2058±553, directly contradicting the abstract's statement that DARAIL outperforms DARC in target domains. The central claim therefore rests on an unproven equality in the theory and on a comparison that fails in a reported setting.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses off-dynamics reinforcement learning, where source and target domains share the same reward but have different transition dynamics and target reward is unavailable. The proposed method, DARAIL, first trains DARC in the source domain, collects its source-domain trajectories as expert demonstrations, and then runs generative adversarial imitation from observation with an importance-weighted discriminator and a \"reward augmented estimator\" (RAE) that combines the discriminator reward with the source reward. The paper claims a finite-sample target-domain error bound (Theorem 4.1) that relaxes DARC's no-exploit assumption, and reports experiments on HalfCheetah, Ant, Walker2d, and Reacher under broken-action, gravity, and density shifts, concluding that DARAIL outperforms both DARC and other baselines.","tokens_in":35746,"tokens_out":6237,"duration_ms":61226,"significance":"If the theoretical and empirical claims were fully established, DARAIL would be a practical and useful recipe for off-dynamics RL: it combines a reward-modification stage with an imitation stage, and the paper supplies a reproducible implementation, multiple-seed results with standard deviations, and several ablations (importance-weight clipping, discriminator update frequency, per-step versus cumulative importance weights, and matched target-rollout comparisons). However, the central theoretical claim is not proven as written, and the abstract's blanket empirical claim is contradicted by the paper's own appendix. The method may still be valuable in specific regimes, but the contributions need to be restated and the theory repaired before the claims can be accepted.","major_comments":[{"comment":"The proof sets the term I1 to zero with the assertion that \"Based on the formulation, π*_DARC can generate optimal trajectories for the target domain in the source domain so that I1 = 0.\" This equality does not follow from the reverse-KL objective in Eq. (3.3), which matches the distribution q(τ_src^{πDARC}) to p(τ_trg^{π*}); distribution matching does not imply equality of expected true rewards between the target-optimal policy in the target domain and some source policy in the source domain. The asserted equality is essentially the strong \"no-exploit\" condition that Section 4.1 claims to avoid, and it is especially implausible in the broken-source regime that motivates the paper. With I1 left nonzero, Theorem 4.1 contains an additional unquantified term, so the claimed relaxation of DARC's assumption is not established. The notation π*_DARC is also never defined.","section":"Appendix B, proof of Theorem B.5"},{"comment":"The abstract's statement that DARAIL outperforms DARC in target domains is contradicted by the paper's own appendix. In Table 5 (Ant, 0.5 gravity), DARAIL reports 990±12 while DARC Evaluation reports 2058±553. In Table 9 (Ant, 1.5 density), DARAIL reports 5193±463 while DARC Evaluation reports 5961±970. Table 6 further shows DARAIL essentially tied with DAIL and IS-ACL on 0.5-gravity Ant. The abstract, Section 3.2, and Section 5.2 should be revised to state the settings in which DARAIL outperforms DARC, and these tables should be presented in the main comparison rather than only in the appendix if they are part of the evaluation.","section":"Appendix Table 5 and Table 9"},{"comment":"The advertised error bound is a consistency decomposition rather than a formal comparison with DARC's target-domain performance. The first term is DARC's source-domain training error, the second term contains an unquantified approximation error inf_ζ d_D(τ̂_src^{πDARC}, τ̂_trg_ζ), and the I1 term is set to zero without justification. Consequently, the theorem does not imply that DARAIL's target reward exceeds DARC Evaluation. Moreover, Appendix E concedes that the method inherits the quality of DARC's source trajectories; for example, in Table 2 the Ant 1.5-gravity setting has DARC Training 2170±258 versus Optimal 5320±429, and DARAIL reaches only 3472±771. This dependence should be stated as a limitation of the theoretical guarantee, not only of the experimental setup.","section":"Theorem 4.1 and Section 4.1"}],"minor_comments":[{"comment":"The acronym is spelled inconsistently as both \"GAIfO\" and \"GAILfo\"; please use one form throughout.","section":"Section 2 and 3.2"},{"comment":"The caption contains a copy-paste error: it says \"(a) and (c) shows the performance of DARC under pf = 0.8, and (a) and (c) shows the performance of DARC under pf = 1.0,\" but panels (b) and (d) are the pf = 1.0 cases.","section":"Appendix C.3, Figure 5 caption"},{"comment":"The \"Optimal\" column reports single numbers without standard deviations, unlike all other tables; please clarify whether these are single-run estimates or provide the same statistics as elsewhere.","section":"Table 9"},{"comment":"The pseudocode contains formatting artifacts such as \"S ROLLOUT\" and undefined notation such as \"DπDARC src S ROLLOUT(πDARC, Msrc)\"; please clean up the pseudocode and define all symbols used.","section":"Algorithm 1"},{"comment":"\"Mujoco\" should be \"MuJoCo\" for consistency with the rest of the text.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The paper may be salvageable as an empirical off-dynamics RL method after correcting the abstract's overclaims and reporting all settings fairly. The theoretical contribution, however, is currently not usable: the I1=0 step in Theorem B.5 is the same kind of strong assumption the paper claims to remove, and the theorem as stated should not be cited as relaxing DARC's assumption until that term is either controlled by an explicit assumption or added to the bound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the actual contribution is a sensible combination: use DARC's source trajectories as expert data for importance-weighted GAIfO, with a doubly-robust-style reward estimator (RAE). In the broken-source setting it genuinely fixes DARC's target-domain degradation. Second, the theoretical bound in Theorem 4.1 is not the relaxation of DARC's assumption that the paper claims. The proof sets I1=0 with the hand-wavy assertion that π*_DARC generates optimal target trajectories in the source domain. That equality is essentially the strong DARC assumption the paper says it avoids; without it, the bound has an unquantified term. Also, Table 5 (Ant, 0.5 gravity) shows DARAIL at 990±12 versus DARC Evaluation at 2058±553, which directly contradicts the abstract's claim that DARAIL outperforms DARC in target domains.\n\nWhat is genuinely new is the RAE and the particular way of combining DARC with GAIfO. The empirical work in the main settings (broken source, 1.5 gravity) is decent: multiple seeds, error bars, released code, and careful ablations of discriminator update frequency and importance-weight clipping. I also appreciate the honesty in Appendix E about relying on DARC source quality.\n\nThe soft spots beyond the I1 step: (a) the error bound is mostly a decomposition with the approximation error left as inf_ζ d_D, so it gives no quantitative guarantee; (b) the blanket claims in the abstract and Section 5.2 are not supported by the appendix tables—the method fails on Ant under 0.5 gravity and also underperforms DARC Evaluation on several density settings; (c) the notation π*_DARC is never defined, which adds to the confusion.\n\nWho this is for: people working on off-dynamics RL and sim-to-real with limited target access. It deserves a serious referee. The idea is useful and the highlighted experiments are solid, but the paper needs a corrected proof and narrowed claims. I would send it out with a request for major revision.","headline":"The core idea is useful and the main experiments are solid, but the proof's key step is an unproven equality and one appendix table directly contradicts the headline claim.","tokens_in":36232,"tokens_out":3686,"would_cite":false,"duration_ms":34553,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reward-shaping RL fails on the target; imitation learning fixes it","keywords":["off-dynamics reinforcement learning","domain adaptation","imitation learning from observation","generative adversarial imitation learning","reward modification","doubly robust estimation","importance weighting"],"falsifier":"Run DARAIL on the broken-source Ant or Walker2d with freezing probability $p_f = 1.0$, a setting the paper reports where DARC cannot obtain near-optimal source reward; if DARAIL then does not beat DARC's target evaluation reward, the method's dependence on high-quality DARC source trajectories is confirmed.","tokens_in":35241,"feed_emoji":"🤖","tokens_out":5228,"duration_ms":45220,"temperature":0.7,"pith_summary":"This paper claims that existing off-dynamics RL methods such as DARC only match the source-domain behavior of a policy to the target-optimal trajectory distribution, and that directly deploying the resulting policy in the target domain suffers a systematic performance drop. DARAIL, the proposed method, adds imitation learning from observation: it uses DARC's source trajectories as expert demonstrations and trains a new policy, rolled out mostly in the source domain, to reproduce those state transitions in the target domain. To make this imitation stable, the paper introduces the reward augmented estimator (RAE), which blends the discriminator's learned reward with the true source reward through an importance weight, in the spirit of doubly robust estimation. The paper proves a finite-sample error bound for DARAIL and reports on four MuJoCo environments that it outperforms DARC and six baselines.","feed_headline":"Mimicking source trajectories boosts off-dynamics RL performance","feed_subtitle":"A hybrid method transfers policies across changed dynamics without target rewards, beating DARC and six baselines.","key_machinery":"The reward augmented estimator (RAE), defined as $RAE(s_t, s_{t+1}) = -\\log D_\\omega(s_t, s_{t+1}) + \\rho(s_t, s_{t+1})(r_{\\mathrm{src}}(s_t, a_t, s_{t+1}) + \\log D_\\omega(s_t, s_{t+1}))$, is the central object. It combines the discriminator signal from generative adversarial imitation learning from observation (GAIfO) with the true source reward, reweighted by the importance weight $\\rho(s_t, s_{t+1}) = p_{\\mathrm{trg}}(s_{t+1}|s_t,a_t)/p_{\\mathrm{src}}(s_{t+1}|s_t,a_t)$, so that the biased discriminator reward is corrected by the ground-truth reward, analogous to doubly robust estimation. The DARC-modified policy provides the expert demonstrations, and importance weighting lets the imitation objective be trained almost entirely from source rollouts, with only occasional target rollouts to estimate the classifiers.","core_discovery":"The central claim is that the off-dynamics RL failure mode of reward-modification methods can be cured by a second stage that imitates the modified-reward policy's trajectories: instead of deploying the DARC policy directly, DARAIL learns a new policy whose target-domain state-transition distribution matches the state-transition distribution DARC produced in the source domain. The theoretical result upper-bounds the target-domain suboptimality of DARAIL by the DARC source error plus an imitation-learning generalization term, thereby relaxing DARC's restrictive assumption that the target-optimal policy is good in the source domain. Empirically, DARAIL's target reward exceeds DARC's target reward on all four tested environments and generally exceeds the other baselines.","pith_inferences":["RAE is a template that can be read as a doubly robust correction for any biased reward estimator in imitation or offline RL, not just for the DARC discriminator; this connection is implied by the paper's analogy but not developed into a general algorithm.","A testable extension would be to replace DARC with another reward-modification method or with a distributionally robust policy in the first stage, to see whether the imitation stage compensates for different kinds of source suboptimality.","The bounded-importance-weight assumption suggests that extremely large dynamics shifts, where $\\rho$ becomes unbounded in practice, remain the hard regime; DARAIL's advantage over DARC should shrink or invert there."],"forward_implications":["A practical recipe for off-dynamics RL: first run a reward-modification method like DARC, then imitate its source trajectories instead of deploying it directly, requiring no target rewards.","The error bound relaxes the no-exploit assumption of DARC: the target-optimal policy no longer needs to be near-optimal in the source domain; only the importance weight needs to stay bounded.","Because RAE downweights the discriminator reward where the dynamics shift is large, DARAIL is less sensitive to the importance-weight clipping interval than importance-sampling baselines.","The method inherits the quality of the DARC source trajectories; when DARC achieves near-optimal source reward, DARAIL can match or exceed DARC's target evaluation reward across the tested environments."],"supporting_citations":[{"why":"supplies the DARC reward-modification baseline and the expert-trajectory generator that DARAIL imitates","marker":"[3]"},{"why":"supplies the generative adversarial imitation learning objective that DARAIL builds on","marker":"[7]"},{"why":"supplies the imitation-from-observation (GAIfO) variant using state pairs","marker":"[8]"},{"why":"supplies the doubly robust estimator idea that motivates the reward augmented estimator RAE","marker":"[12]"},{"why":"supplies the SAC policy optimizer used for the generator updates","marker":"[17]"},{"why":"supplies the GAIL generalization bound that the paper adapts into the imitation-learning error term of Theorem 4.1","marker":"[38]"}],"fun_headline_variants":["Imitation closes off-dynamics RL gap","Reward shaping + imitation beats pure reward shaping","Imitate edited trajectories for off-dynamics RL gains","Off-dynamics RL: add imitation to reward modifications"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method relies on DARC producing high-quality, near-optimal trajectories in the source domain; if the dynamics shift is so large that DARC itself fails, the imitation stage ends up copying suboptimal behavior and DARAIL can underperform DARC.","fun_headline_variants_meta":{"raw":{"variants":["Imitation closes off-dynamics RL gap","Reward shaping + imitation beats pure reward shaping","Imitate edited trajectories for off-dynamics RL gains","Off-dynamics RL: add imitation to reward modifications"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000555,"raw_usage":{"total_tokens":2622,"prompt_tokens":906,"completion_tokens":1716,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":1655}},"tokens_in":522,"tokens_out":1716,"duration_ms":14021,"temperature":1.0,"reasoning_tokens":1655,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:12:29.602995+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DARAIL on the broken-source Ant or Walker2d with freezing probability $p_f = 1.0$, a setting the paper reports where DARC cannot obtain near-optimal source reward; if DARAIL then does not beat DARC's target evaluation reward, the method's dependence on high-quality DARC source trajectories is confirmed.","supporting_citations":[{"cited_title":"vf+MOWCXlXkD7CB/Zj9tqm3hyT0=","cited_arxiv_id":null,"evidence_quote":"supplies the GAIL generalization bound that the paper adapts into the imitation-learning error term of Theorem 4.1"}],"review_version":1}