{"id":"02a2b6a8-1e13-40d7-935c-c8c5929700e0","arxiv_id":"2501.04870","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Re-weighted targeting enables provably beneficial transfer learning for deep Q-learning in non-stationary finite-horizon MDPs when reward differences are smoother than the Q-functions.","lead":"This paper develops a transfer learning method for offline reinforcement learning in non-stationary, finite-horizon environments, where the target task has few samples and source tasks have many. It introduces a re-weighted targeting procedure that constructs transferable pseudo-samples, with neural network approximation and error bounds, plus experiments on synthetic and sepsis data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The transfer advantage rests entirely on Assumption 3 (γ2>γ1), a smoothness gap that is asserted, not derived: δ*_agg contains transition-density weights, so even smooth reward differences can yield γ2≤γ1 and a transfer bound no better than target-only.","rationale":"The reader identified Assumption 3 (γ2>γ1) as the weakest link, and I agree that it is the single most load-bearing condition. The comparison in Remarks 4, 8, and 9 is literally an upper-bound comparison: if γ2≤γ1, the first term in Theorem 8 is at least the target-only rate, so the bound cannot demonstrate any transfer benefit regardless of how large nM is. This makes the assumption necessary for the central claim. My stress-test sharpens the reader's point: δ*_agg is not simply a reward difference but a transition-weighted aggregate, because \\barυ^(k)(s,a)=υ_k p^(k)(s,a)/P_agg(s,a) enters Eq. (21). In the non-transferable transition setting (Corollary 11), the smoothness of δ*_agg is upper-bounded by the smoothness of the transition densities unless the weights happen to be smoother. The paper never derives Assumption 3 from primitive assumptions or checks it empirically, so the advertised advantage is conditional on a gap that could fail exactly in the regime the method is designed for. That said, this is a genuine assumption about the data-generating process rather than an internal contradiction; the proof skeleton appears coherent conditional on it. The reader's CONDITIONAL verdict is therefore appropriate, and my read does not change it. I would add the derived-weight concern to the list of items the authors must address before acceptance.","tokens_in":39054,"tokens_out":16607,"duration_ms":165670,"concrete_test":"Construct K=1 with S=[0,1]^d, choose transition densities p^(0),p^(1) of known Hölder smoothness γ3<γ2, and a reward difference δ^(1) in H2 with known smoothness γ2. Compute δ*_agg = [υ_1 p^(1)/(υ_0 p^(0)+υ_1 p^(1))] δ^(1), then estimate its effective smoothness by fitting ReLU networks of increasing size to δ*_agg and measuring the decay of squared L2 error (or by estimating the exponent in the approximation error). If the estimated smoothness is ≈γ3 rather than γ2, Assumption 3 is violated in a natural non-transferable-transition setting, and the Theorem 8 improvement over target-only q-learning cannot be expected to hold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 8 (Eq. 26) bounds the transfer estimator by (J log n0/n0)^{2γ2/(2γ2+1)} for the debiasing term, while target-only Q-learning has rate (J log n0/n0)^{2γ1/(2γ1+1)} (Remark 4). Because f(γ)=2γ/(2γ+1) is increasing, γ2≤γ1 makes the first term at least as large as the target-only rate; adding the positive aggregation and transition-ratio terms then gives an upper bound strictly worse than target-only. Thus every advertised improvement in Remarks 4, 8, and 9 requires Assumption 3's γ2>γ1. The assumption is not a primitive condition on reward functions: δ*_agg(s,a)=Σ_k \\barυ^(k)(s,a)δ^(k)(s,a) with \\barυ^(k)=υ_k p^(k)/P_agg (Eq. 21 and Section 2.5). Even if each reward difference δ^(k) is smooth, the mixing weight contains transition densities; when transitions are non-transferable, the aggregate can inherit the roughness of p^(k), so γ2>γ1 may fail even when individual reward differences are smooth. The paper provides no derivation of Assumption 3 from more basic smoothness assumptions and no empirical verification of γ2>γ1 in the experiments, so the central claim is conditional on an unverified and potentially fragile premise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies transfer learning for offline, non-stationary finite-horizon MDPs with deep ReLU network function approximation. The authors propose a re-weighted targeting procedure that reweights source pseudo-responses by an estimated transition density ratio and then uses backward inductive Q-learning with a two-step estimator: a pooled pilot estimator of an aggregated Q* function and a target debiasing step that estimates the reward difference. They provide non-asymptotic L2 error bounds (Theorem 8, Corollaries 11–12) in terms of the target sample size n0, the total pooled sample size nM, function-class smoothness parameters γ1,γ2,γ3,γ4, and transition-ratio estimation error, and they evaluate the method on a synthetic two-stage MDP and a MIMIC-III sepsis management calibration. The main theoretical claim is that, when the aggregated reward-difference function is smoother than the target Q* function (γ2>γ1) and the source pool is large, the transfer estimator improves on target-only Q-learning.","tokens_in":39312,"tokens_out":10666,"duration_ms":96185,"significance":"If the stated assumptions hold, the paper offers a coherent framework and, to my knowledge, a novel non-asymptotic analysis of transfer deep Q-learning for non-stationary finite-horizon MDPs. The proof strategy—separating pilot aggregation error from debiasing error, handling temporal dependence without a completeness assumption, and bounding neural-network transition-density-ratio estimators—contains useful ideas that may be reusable. The empirical demonstration on a calibrated sepsis environment is a strength. However, the advertised transfer advantage is entirely driven by the unverified smoothness gap γ2>γ1 in Assumption 3, and the manuscript contains internal inconsistencies in sign conventions and in the definition of the pooled pilot estimator. The central derivation appears sound once the assumptions are taken at face value, but the paper is not acceptable in its current form.","major_comments":[{"comment":"The advertised advantage of transfer is entirely driven by the inequality γ2>γ1. Because f(γ)=2γ/(2γ+1) is increasing in γ, if γ2≤γ1 the debiasing term (J log n0/n0)^{2γ2/(2γ2+1)} in Eq. (26) is no smaller than the target-only rate (J log n0/n0)^{2γ1/(2γ1+1)} used in Remark 4, and the positive aggregation and transition-ratio terms make the bound strictly worse. The paper does not derive γ2>γ1 from primitive conditions: δ*_agg in Eq. (21) is Σ_k \\barυ^(k) δ^(k) with mixing weights \\barυ^(k)=υ_k p^(k)/P_agg, so even smooth reward differences can produce a rough aggregate when the transition densities are non-transferable. The experiments in Section 5 provide no check of γ2>γ1 or of the smoothness of δ*_agg. I am not treating Assumption 3 as circular; rather, it is a legitimate but unverified premise that carries the central claim. The manuscript should either derive γ2>γ1 from more basic conditions or provide an empirical diagnostic.","section":"Section 3.1, Assumption 3; Section 4.1, Eq. (26)"},{"comment":"The sign conventions are internally inconsistent. With δ^(k)_t = r^(0)_t - r^(k)_t as in Eq. (8), Eqs. (14)-(15) give Σ_k \\barυ^(k)_t E[Y^(rwt-k)_t | s,a] = Q*^(0)_t - Σ_k \\barυ^(k)_t δ^(k)_t, whereas Eq. (19) writes this as Q*^(0)_t + Σ_k \\barυ^(k)_t δ^(k)_t. Eq. (21) then defines δ*_agg = r*_agg - r^(0) and identifies it with Σ_k \\barυ^(k) δ^(k), which has the opposite sign relative to (8). The ambiguity propagates to the debiasing step (24) and Lemma 16, where the estimator bδ_t is supposed to estimate Q*^(0)_t - Q*^agg_t. Please fix the sign convention and state explicitly whether δ*_agg in Assumption 3 is r*_agg - r^(0) or its negative, so that the approximation argument in Lemma 16 matches the estimand.","section":"Section 2.5, Eqs. (19)-(21)"},{"comment":"The pilot pooled estimator is defined over k∈[K] (source tasks only) in Algorithm 1, but over k∈{0}∪[K] in Eq. (24). The proof of Lemma 13 in Appendix B includes target samples, since it sets y^(rwt-0)=y^(0) and treats the task label as random over all tasks. The pilot rate (J log nM/nM)^{2γ1/(2γ1+1)} in Theorem 8 requires a pooled sample of size nM = n0 + Σ_k n_k. As written, Algorithm 1 computes a different object from the one analyzed in Theorem 8. Please align the algorithm with the analysis and with the definition of P^agg_t in Section 4, which includes k=0.","section":"Algorithm 1, Eq. (22); Section 3, Eq. (24)"},{"comment":"The comparison rate for standard Q-learning is stated as (J log n0/n0)^{2γ3/(2γ3+1)}, but Remarks 4 and 9 and Corollary 12 use (J log n0/n0)^{2γ1/(2γ1+1)} for target-only backward inductive Q-learning. The inequality displayed in Remark 8 therefore does not establish the claimed advantage unless γ3≥γ1 is imposed. The text says this condition 'can be verified through equation (6)', but no proof or empirical verification is given. Please state the correct target-only rate and make the extra condition explicit.","section":"Section 4.3, Remark 8"}],"minor_comments":[{"comment":"The theorem statement lists Assumptions 4(i), 5, 6, and 7, but the rate uses γ1 and γ2 from Assumption 3; please include Assumption 3 in the statement.","section":"Theorem 8"},{"comment":"The text mentions an RKHS/kernel instantiation in Appendix E, but the appended Appendix E is the MIMIC-III calibration and the RKHS material appears as 'Appendix ??'; the missing appendix should be supplied or the claim removed.","section":"Section 2.6; supplementary index"},{"comment":"There is an unresolved citation 'as introduced by ?' for the Deep & Cross Network blocks; please add the reference.","section":"Section 5.2"},{"comment":"Assumption 4(iii) is stated for 'one fixed source k', while Algorithm 4 and Corollary 12 use multiple sources; please clarify how the multi-source case is handled.","section":"Assumption 4; Algorithm 4"},{"comment":"The definition of \\barυ^(k)_t(s,a) uses denominators P_t(s,a) that are not defined; please define them and reconcile this with the definition of P^agg_t in Section 4, which includes the target task.","section":"Section 2.5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript leans substantially on the authors' own previously posted work (Chen, Li & Jordan 2022; Chen, Song & Jordan 2022) for the transfer-Q-learning framework and for the MIMIC-III calibration; independent grounding for the density-ratio transfer step would strengthen confidence. The empirical section does not provide code or data, so the reported figures are not independently reproducible. The advertised RKHS/kernel instantiation is missing from the supplement; this should be fixed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version. The paper delivers something real: non-asymptotic error bounds for a re-weighted targeting transfer Q-learning algorithm in non-stationary finite-horizon MDPs, with ReLU network approximation and estimated transition density ratios. Lemmas 13 and 16 decompose the error cleanly, and the rates in Theorem 8 and the corollaries follow under the stated assumptions. The construction of 'transferable' samples via the density-ratio reweighting is a natural and useful idea, and the proof handles temporal dependence without a completeness assumption, which is worth noting.\n\nThe soft spots are real but fixable. First, Assumption 3 (γ2>γ1) is the engine of every advertised advantage, and the paper never derives it from more primitive conditions or checks it empirically. Your stress-test note is right that the aggregated difference δ*_agg is a transition-density-weighted average, so a smooth δ does not automatically give γ2>γ1 when transitions are non-transferable. I do not think this is fatal, because the assumption is explicit and standard in transfer learning, but the paper should say plainly that the theoretical gain is conditional on this gap and discuss when it holds.\n\nSecond, there are internal inconsistencies: Eq. (19) and Eq. (21) carry a sign conflict relative to the definition of δ in Eq. (8), and Algorithm 1's pooled sum omits target samples while Section 3's version includes them. These look like typos, but they undermine trust as written. Third, the abstract claims naive pooling fails in MDPs, but the experiments only compare transfer vs no-transfer; there is no naive pooling baseline and no error bars. That is a substantive gap in the empirical evidence. Fourth, 'pioneers' overstates it, since the paper's own citations include earlier transfer Q-learning for non-stationary MDPs under linear assumptions.\n\nNet: the central argument holds up under the stated assumptions, the contributions are new, and the proof skeleton deserves referee time. I would send it to review with a request for major revision: fix the sign and display issues, add the naive pooling comparison, and either verify the γ-gap in the experiments or at least give a primitive condition that implies it. This is a paper I would cite once cleaned up.","headline":"Useful extension of transferred Q-learning to neural networks and density ratio estimation, with coherent but conditional error bounds; the advertised gain hinges on an unverified smoothness gap and the paper needs cleanup before it can be trusted.","tokens_in":39894,"tokens_out":5818,"would_cite":true,"duration_ms":54222,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G08","68T05","90C40"],"pacs":[],"model":"deepseek-v4-flash","headline":"When the reward gap between source and target tasks is smoother than the target's own Q-function, transferring source trajectories through a re-weighted targeting procedure provably improves offline Q-learning bounds.","keywords":["transfer learning","offline reinforcement learning","non-stationary Markov decision processes","backward inductive Q-learning","deep ReLU neural networks","density ratio estimation","hierarchical composition model"],"falsifier":"Run the Section 5.2 two-stage experiment with a source task whose reward gap is deliberately chosen to be rough (so that $\\gamma_2\\le\\gamma_1$) and compare transfer versus target-only regret; Theorem 8 predicts the transfer bound's leading term worsens relative to the target-only rate, so the regret advantage should shrink or reverse.","tokens_in":38760,"feed_emoji":"🎯","tokens_out":12152,"duration_ms":103195,"temperature":0.7,"pith_summary":"This paper works out when and how data from similar but different sequential decision problems can help learn the optimal Q-function of a target task in offline, non-stationary finite-horizon Markov decision processes. It shows that naively pooling source and target trajectories, a strategy that works for regression transfer, fails here because the source and target optimal Q-functions differ in a way that reward differences alone cannot correct. The proposed remedy is a re-weighted targeting procedure: re-weight source future rewards by the transition density ratio and re-target them with the target Q-function, creating pseudo-samples whose only remaining bias is the source-target reward gap. Using deep ReLU networks as function classes, the main theorem bounds the target estimation error by three additive terms—reward-difference estimation, reward aggregation, and transition-ratio estimation—and shows the transfer advantage is genuine when the reward gap is smoother than the target Q-function and source samples are abundant. Experiments on a synthetic two-stage MDP and a calibrated sepsis management environment support the theory.","feed_headline":"Transfer learning speeds up offline RL when reward gaps are smooth","feed_subtitle":"A re-weighted targeting procedure turns source trajectories into training data for a non-stationary target MDP, with provable error bounds.","key_machinery":"The load-bearing constructions are the re-weighted targeting (RWT) pseudo-response and the aggregated reward and Q-functions. RWT replaces the source Q-function in the pseudo-response with the target Q-function and multiplies the future term by the transition density ratio $\\omega^{(k)}_t(s'|s,a) = p^{(0)}_t(s'|s,a)/p^{(k)}_t(s'|s,a)$, transporting source samples to the target distribution; the identity $\\mathbb{E}^{(k)}[Y^{(rwt-k)}_{t,i}|s,a] = Q^{*(0)}_t(s,a) - \\delta^{(k)}_t(s,a)$ turns transfer into a supervised regression problem with a known bias term. The debiasing step then estimates $\\delta^{*\\,agg}_t = r^{*\\,agg}_t - r^{(0)}_t$ from target samples only. The rates are controlled by the hierarchical composition model's dimension-adjusted smoothness $\\gamma^*(H) = \\min_{(\\beta,t)\\in\\mathcal P}\\beta/t$, with $\\gamma_1$ for Q-functions and $\\gamma_2$ for reward gaps; Assumption 3 requires $\\gamma_2>\\gamma_1$, and the density-ratio estimators add terms involving $\\gamma_3$ and $\\gamma_4$.","core_discovery":"The central claim is that transfer learning for offline non-stationary Q-learning succeeds through a two-step debiasing scheme. At each backward stage $t$, source pseudo-responses are built as $\\widehat y^{(rwt-k)}_{t,i} = r^{(k)}_{t,i} + \\gamma\\,\\widehat\\omega^{(k)}_{t,i}\\max_a \\widehat Q^{(0)}_{t+1}$; because the source next-state distribution is re-weighted by the density ratio $\\omega^{(k)}_t = p^{(0)}_t/p^{(k)}_t$, the conditional mean of these responses equals $Q^{*(0)}_t(s,a) - \\delta^{(k)}_t(s,a)$, so the only remaining mismatch is the reward gap $\\delta$. Pooling these responses gives a low-variance but biased pilot estimator of an aggregated Q-function; a second regression on target samples fits the reward-difference function $\\delta^{*\\,agg}_t$ and corrects the bias. The main theorem (Theorem 8) states that, with probability at least $1-7Te^{-u}$, the squared $\\mathrm{L}^2$ error under the target distribution is bounded by $(T-t)\\max\\{\\kappa,1\\}^{T-t}$ times a sum of the nonparametric rate for estimating $\\delta^{*\\,agg}_t$ with $n_0$ samples, the rate for estimating the aggregated Q-function with $n_M$ samples, the transition-ratio estimation error, and a tail term. When the reward gap is smoother than the target Q-function ($\\gamma_2>\\gamma_1$) and $n_M\\gg n_0$, this bound improves on target-only backward inductive Q-learning.","pith_inferences":["The paper leaves implicit that source tasks could be selected or weighted adaptively by estimating whether $\\gamma_2>\\gamma_1$ holds for each task pair; the theory predicts that sources with rough reward gaps should be downweighted or dropped.","The debias-then-pool scheme is likely to carry over to policy evaluation ($Q^\\pi$) and to supervised domain-shift problems, since the same density-ratio reweighting machinery appears there; a concrete follow-up would instantiate the same estimators and check whether the $n_0/n_M$ rate splitting survives.","The horizon factor $(T-t)\\max\\{\\kappa,1\\}^{T-t}$ in the bound suggests the practical transfer advantage is largest for short-horizon decisions such as adaptive treatment regimes, although the paper does not make this comparative claim.","Because the calibrated sepsis experiment uses sex as the source/target split, a natural robustness check is to repeat the evaluation on other subgroup splits where the reward-gap smoothness assumption is less likely to hold."],"forward_implications":["Under total transition similarity ($\\omega=1$), the transfer bound splits into a reward-difference rate in $n_0$ plus an aggregation rate in $n_M$; when $n_0\\lesssim n_M$ and $\\gamma_2>\\gamma_1$, this improves on the target-only rate $(J\\log n_0/n_0)^{2\\gamma_1/(2\\gamma_1+1)}$.","When transition densities must be estimated, the transition-ratio error enters the Q-function bound; transfer still pays off with nontransferable transitions when $n_M\\gg n_0$ and $\\gamma_3\\ge\\gamma_1$, and with transferable transitions when $\\gamma_4\\ge\\gamma_1$ as well.","The analysis removes the function-class completeness assumption used in earlier deep Q-learning work and gives non-asymptotic bounds for deep-neural-network transition density and density-ratio estimators, so the tools apply to domain shift beyond value iteration.","The same RWT sample construction is instantiated with kernel function classes and fits an online Explore-Then-Commit extension, giving a template for transfer RL beyond the offline batch setting."],"supporting_citations":[{"why":"supplies the backward inductive Q-learning framework that the transfer algorithm extends","marker":"Murphy 2005"},{"why":"provides the Q-learning formulation and pseudo-response construction used at each stage","marker":"Clifton & Laber 2020"},{"why":"the deep Q-learning error-propagation analysis whose function-class completeness assumption is removed here","marker":"Fan et al. 2020"},{"why":"defines the hierarchical composition model and the smoothness parameter gamma underlying the rates","marker":"Kohler & Langer 2021"},{"why":"gives the convex M-estimation approach adapted for transition density ratio estimation","marker":"Nguyen et al. 2010"},{"why":"supplies the ReLU network approximation and empirical-process bounds used in every error term","marker":"Fan & Gu 2023"},{"why":"the covariate-shift assumption (bounded likelihood ratio eta) is borrowed for comparing target and aggregate distributions","marker":"Ma et al. 2023"},{"why":"the earlier transferred Q-learning setting (linear Q*, identical transitions) that this work generalizes to nonlinear non-stationary MDPs","marker":"Chen, Li & Jordan 2022"}],"fun_headline_variants":["Re-weighted targeting makes transfer deep Q-learning work offline","Smooth reward gaps enable provable transfer in offline RL","Debiasing source responses powers offline transfer Q-learning","Two-step reweighting debiases source data for offline RL","Transfer Q-learning with provable bounds when reward gaps are smooth"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole transfer advantage rests on Assumption 3: the source-target reward gap must be a smoother, lower-complexity function than the target Q-function itself; if $\\gamma_2\\le\\gamma_1$, the leading error term from transfer can be slower than learning from target data alone.","fun_headline_variants_meta":{"raw":{"variants":["Re-weighted targeting makes transfer deep Q-learning work offline","Smooth reward gaps enable provable transfer in offline RL","Debiasing source responses powers offline transfer Q-learning","Two-step reweighting debiases source data for offline RL","Transfer Q-learning with provable bounds when reward gaps are smooth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000693,"raw_usage":{"total_tokens":3226,"prompt_tokens":1127,"completion_tokens":2099,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":743,"completion_tokens_details":{"reasoning_tokens":2017}},"tokens_in":743,"tokens_out":2099,"duration_ms":16404,"temperature":1.0,"reasoning_tokens":2017,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:23:44.193826+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Section 5.2 two-stage experiment with a source task whose reward gap is deliberately chosen to be rough (so that $\\gamma_2\\le\\gamma_1$) and compare transfer versus target-only regret; Theorem 8 predicts the transfer bound's leading term worsens relative to the target-only rate, so the regret advantage should shrink or reverse.","supporting_citations":[],"review_version":1}