{"id":"ced8b41f-0897-46c7-ae08-2c38cf5b6796","arxiv_id":"2608.03108","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"CSDG modifies the offline Bellman backup by scaling a smoothed convex-hull-neighborhood correction against an in-sample expectile target and reports strong D4RL aggregate performance.","lead":"This paper proposes CSDG, an offline reinforcement learning update that adds a small, adjustable local correction to an in-sample value target so nearby out-of-distribution actions can help without destabilizing bootstrapping. It reports the highest aggregate scores among the listed methods on the D4RL Gym-MuJoCo and AntMaze benchmarks, along with idealized operator bounds and a code link.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theory covers only the fixed-kernel operator; Algorithm 1 trains a time-varying, Monte-Carlo, learned-expectile operator, so Theorems 2-3 do not govern the reported results.","rationale":"The reader's weakest_assumption already identifies this gap; my reading confirms and sharpens it. The strongest claim (Eq. 7 makes the generalized contribution explicit while retaining an in-sample target) is an operator-level statement, but the evidence offered for it is Algorithm 1 plus Table 2. The proof of Theorem 4 and the opening of Appendix D explicitly assume a fixed pi' and fixed perturbation kernels, while Algorithm 1 updates pi_phi' every delayed step and the actor objective depends on Q_theta, making the target action center Q-dependent. Thus the contraction argument cannot be invoked for the trained update. The paper is transparent about the V_psi approximation but does not quantify it, and there is no theorem linking the residuals or the changing actor to the fixed-point bounds. This is a theory/algorithm mismatch rather than an internal contradiction: it does not invalidate the empirical D4RL numbers, but it means the theoretical contribution and the empirical claim are not connected by the presented analysis. The reader's CONDITIONAL verdict is therefore appropriate, and my stress test does not change it. I recommend UNCHANGED.","tokens_in":21627,"tokens_out":9222,"duration_ms":86574,"concrete_test":"Retrain CSDG on six representative tasks (halfcheetah-medium, hopper-medium, walker2d-medium, hopper-medium-expert, antmaze-medium-play, antmaze-large-diverse) with every component unchanged except that the target action center in Eq. (19) is a fixed behavior-cloning policy pi_BC pretrained on the dataset, replacing the adaptively updated pi_phi'. Use the same lambda, mu, noise scales, seeds, and final-ten-evaluation averaging. If the aggregates stay within roughly 5% of Table 2, the fixed-kernel idealization is behaviorally adequate and the theory-algorithm gap is benign for these tasks; if they drop more than that, the reported gains depend on the time-varying actor that Theorem 4 explicitly excludes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix D begins by stating that the target action selector and the two perturbation kernels remain fixed throughout the analysis. This is exactly the condition under which Theorem 4 proves S_mu and V^Q_tau are non-expansive and T_CSDG is a gamma-contraction; the proof evaluates both Q1 and Q2 at the same candidates generated by the fixed kernels. Algorithm 1 instead constructs targets (Eq. 19) as clip(pi_phi'(s') + eta_j), where pi_phi' is updated by Polyak averaging. Since the actor loss (Eq. 17) depends on Q_theta and V_psi, the perturbation center is a function of the learned critic, so the smoothing kernel is neither fixed nor independent of Q. The object updated by Eqs. (18)-(21) is therefore not the operator T_CSDG of Definition 3/Eq. (7), and there is no guarantee that the trained critic is near a fixed point of T_CSDG. The paper acknowledges the V_psi residual in Sec. 3.4 but gives no bound for it, no bound for the single-sample Monte Carlo target in Eq. (20), and no bound for the drift of pi_phi'. Consequently, Theorem 2's iterate/fixed-point bounds and Theorem 3's non-degradation criterion do not apply to the algorithm whose D4RL scores appear in Table 2. The central claim, that CSDG controls local correction propagation under geometric bounds, is established only for an idealized operator, while the empirical headline rests on a separate, unproven approximation. This mismatch is the load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CSDG, a Bellman backup that adds a convex-hull-neighborhood smoothed local correction to an in-sample expectile target, with a scalar λ gating how much of the correction enters each recursive update. The theoretical part defines an idealized operator (Definition 3, Eq. (7)), proves an exact one-step decomposition, iterate and fixed-point bounds under fixed perturbation kernels, and a conditional induced-policy performance comparison. The practical part (Section 3.4, Algorithm 1) instantiates the idea with two-scale bounded noise, an expectile value network, twin critics, and a delayed actor, and reports D4RL aggregates of 1199.4 on Gym-MuJoCo and 445.8 on AntMaze.","tokens_in":21961,"tokens_out":9837,"duration_ms":92369,"significance":"If the theory governed the implemented algorithm, the operator-level decomposition would be a conceptually clean way to separate an in-sample reference path from a locally generalized correction, and the reported broad empirical gains would be valuable. The proofs in Appendix D are careful and the operator identities in Theorems 1 and 4 are correct under the stated idealized assumptions; the empirical study is broad, with ablations, learning curves, and code. However, the main theoretical guarantees are proven only for a fixed-kernel operator, while the reported algorithm uses a time-varying target actor, single-sample Monte Carlo targets, and a learned expectile surrogate. This mismatch means the central control claim currently applies to an idealized object, not to the algorithm whose scores are the headline evidence. The concern raised in the stress test is therefore valid and is the primary reason the paper needs substantial revision.","major_comments":[{"comment":"The theoretical operator T_CSDG in Definition 3 (Eq. (7)) and the contraction, iterate, and fixed-point results in Theorems 2–4 require a fixed deterministic target action selector π' and fixed perturbation kernels ν_In and ν_OOD; Appendix D states this condition explicitly at its opening. Algorithm 1 instead updates the target actor π_φ' by Polyak averaging (Step 8), and the actor loss in Eq. (17) depends on Q_θ and V_ψ, so the perturbation center changes during training. Moreover, Eq. (19) forms the smoothed target from one Monte Carlo pair per kernel, and Eq. (20) replaces the exact expectile V^{Q_θ'}_τ with the learned V_ψ. The paper acknowledges the V_ψ residual in Section 3.4 but provides no bound on it, no bound on the Monte Carlo variance, and no bound on the drift of π_φ'. Consequently, the object updated by Eqs. (18)–(21) is not shown to be near a fixed point of T_CSDG, and Theorems 2 and 3 do not govern the D4RL results in Table 2. Please either supply approximation-error and drift bounds that cover the practical update, or explicitly restrict the theoretical claims to the idealized operator and describe the practical target as a heuristic approximation.","section":"§3.4, Algorithm 1, and Appendix D"},{"comment":"The fixed-point bound ||Q*_CSDG − Q*_In||_∞ ≤ γλe*/(1−γ) has e* = ||S_μQ*_CSDG − V^{Q*_CSDG}_τ||_∞ on its right-hand side, which is exactly the branch discrepancy that the correction is supposed to control. The theorem does not provide an upper bound on e* in terms of independent quantities such as Lipschitz constants of Q, the perturbation radii δ_In and δ_OOD, dataset coverage, or λ; the companion residual bound for a general κμ likewise involves e(Q̄) without control. Thus Eq. (12) is a valid a posteriori identity, but by itself it does not establish that CSDG controls local correction propagation. An a priori bound on e*, or a precise statement of the conditions under which e* is provably small, is needed to support the paper's central claim.","section":"§3.3, Eq. (12)"},{"comment":"The non-degradation criterion λg_μ ≥ (1−λ)ξ is exactly the nonnegativity condition for the expression λg_μ − (1−λ)ξ appearing in the preceding line, and the quantities g_μ and ξ are not bounded or connected to the discrepancy e* or to the perturbation radii. As stated, the criterion cannot be checked from the data or from the analysis, and it does not provide an actionable rule for selecting λ, such as the paper's fixed choice λ=0.25. Please either provide concrete conditions under which g_μ and ξ can be bounded, or rephrase this result explicitly as a consistency statement rather than a design criterion.","section":"§3.3, Eq. (14)"}],"minor_comments":[{"comment":"The weight w(s,a) uses a clipping constant w_max that is never defined or listed in Table 5; please add its value or a reference to the TD3BC-style setting.","section":"Eq. (15)"},{"comment":"The column header 'CQL+C4' is ambiguous: it is unclear whether this is a single method combining CQL and C4 or a typo; please clarify and ensure the relevant reference is cited.","section":"Table 2"},{"comment":"The text says CSDG consistently improves both tested in-sample backbones across all D4RL v2 locomotion tasks, but Table 3 reports only the medium, medium-replay, and medium-expert subsets; please qualify the claim accordingly.","section":"§4.2, Table 3"},{"comment":"The behavior-cloning coefficient ν is listed as '0.1 or 10 for Gym locomotion'; please state explicitly which datasets receive which value.","section":"Table 5"},{"comment":"The aggregate scores in Table 2 are reported without standard deviations or significance tests, and some tasks are decided by small margins; please report per-seed aggregates or state that baseline aggregates are taken from the original papers.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a machine learning journal, and I see no citation-pattern red flags. The central issue is that the theoretical framework and the implemented algorithm are different objects, and the paper currently does not quantify the gap. If the authors can provide approximation-error bounds or clearly restrict the theoretical claims to the idealized operator while repositioning the empirical contribution, the paper could become acceptable. I do not see grounds for rejection, but the revision needs to be substantive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"That paper is worth a serious look, but the caveat is that the proof and the algorithm are not about the same object. CSDG's target construction—an expectile in-sample value plus a lambda-scaled, two-scale CHN-smoothed correction—is new as far as I can tell, and the operator identities in Theorems 1 and 4 check out under the stated fixed-kernel assumptions. The D4RL aggregates (1199.4 Gym, 445.8 AntMaze) are strong, with sensible ablations and stability curves, and the authors are upfront that the implementation uses task-tuned noise scales and doesn't track the anchoring geometry.\n\nThe soft spot is the theory–practice gap, and the stress-test note is right about it. Appendix D explicitly assumes a fixed target action selector and fixed perturbation kernels. Algorithm 1 uses a target actor updated by Polyak averaging, so the smoothing center moves with the learned critic, and the single-sample Monte Carlo target is never the S-mu operator in the proofs. That means Theorems 2 and 3 don't govern the trained critic. The paper acknowledges the V_psi residual but gives no bound, and the fixed-point bound in Eq. (12) involves e*, which is the correction the proof is supposed to control. So the theoretical contribution is real but narrower than the abstract suggests.\n\nI wouldn't call this fatal. The same pattern appears in many offline RL papers, and the empirical work is credible: standard errors, multiple seeds, fair baselines. The authors should be asked to either extend the analysis to time-varying target actors or clearly state that the theory applies only to the idealized operator. A committed reviewer could get useful work out of that. The code URL lacks a commit hash, and the hyperparameter table shows domain-specific values, but reproduction is not out of reach. Overall: serious referee, yes; accept without changes, no.","headline":"The CSDG mechanism is genuinely new and the D4RL results are strong, but the theory covers only the fixed-kernel operator while the algorithm trains a time-varying one—a real gap worth closing in revision.","tokens_in":22534,"tokens_out":2472,"would_cite":true,"duration_ms":22759,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes CSDG, which rewrites the offline Bellman backup as an in-sample expectile target plus a scaled local correction and reports top D4RL aggregates of 1199.4 and 445.8.","keywords":["offline reinforcement learning","out-of-distribution actions","local generalization","convex hull neighborhood","expectile regression","Bellman backup","bootstrapping error","D4RL benchmark"],"falsifier":"Run Algorithm 1 with $\\lambda=1$ and a fast-updating target actor, recording the empirical branch discrepancy $\\|S_\\mu Q - V_\\tau^Q\\|_\\infty$ and the distance from the learned critic to the in-sample critic's fixed point over iterations; if the distance grows faster than $\\gamma\\lambda\\sum_{t=0}^{k-1}\\gamma^{k-1-t}e_t$ or exceeds $\\gamma\\lambda e^*/(1-\\gamma)$, the fixed-kernel contraction premise is violated and the advertised bounds do not govern the trained critic.","tokens_in":21436,"feed_emoji":"🤖","tokens_out":10322,"duration_ms":79438,"temperature":0.7,"pith_summary":"Offline reinforcement learning usually forces a choice between staying inside the logged data and risking overestimation from out-of-distribution actions. This paper proposes a third option: keep an in-sample expectile value as the base of every Bellman backup, and add a separately scaled correction built from actions perturbed just outside the data's local geometry. The correction is the difference between a smoothed value over small- and large-radius perturbations and the in-sample expectile, and a coefficient $\\lambda$ controls how much of it propagates at each update while the discount remains $\\gamma$. The paper derives an exact one-step identity, iterate and fixed-point bounds in terms of the branch discrepancy, and a conditional non-degradation criterion for the induced policies. It reports the top aggregate D4RL scores among the compared methods, 1199.4 on Gym–MuJoCo and 445.8 on AntMaze, and shows consistent gains when the correction is added to IQL and XQL.","feed_headline":"One dial feeds safe out-of-distribution value into offline RL","feed_subtitle":"CSDG rewrites the Bellman target as in-sample value plus a gated local correction, topping D4RL at 1199 and 446.","key_machinery":"The load-bearing object is the CHN smooth dual-generalization operator. State-conditional CHN (convex hull and its neighborhood) is the geometric reference: for a state $s$, it is the set of actions within radius $\\delta$ of the convex hull of the $k$ nearest dataset actions. The two-scale smoother averages values $S_\\mu Q(s) = \\mu\\mathbb{E}[Q(s,a_{\\mathrm{In}})] + (1-\\mu)\\mathbb{E}[Q(s,a_{\\mathrm{OOD}})]$ over in-sample-oriented and OOD-oriented candidates sampled at radii $\\delta_{\\mathrm{In}} < \\delta_{\\mathrm{OOD}}$. The expectile functional $V_\\tau^Q(s)$ supplies the in-sample reference, and the difference $B^\\tau_{\\mathrm{CHN}} Q(s) = S_\\mu Q(s) - V_\\tau^Q(s)$ is the local correction gated by $\\lambda$ inside each backup. The argument works because $S_\\mu$ and the expectile map are non-expansive, making both $T_{\\mathrm{CSDG}}$ and the in-sample operator $\\gamma$-contractions; the nearest-anchor decomposition then splits the branch discrepancy into a geometric term and a reference term.","core_discovery":"The central claim is that the Bellman backup for offline RL can be decomposed as $T_{\\mathrm{CSDG}} Q(s,a) = R(s,a) + \\gamma \\mathbb{E}_{s'\\sim P}[V_\\tau^Q(s') + \\lambda(S_\\mu Q(s') - V_\\tau^Q(s'))]$, where $V_\\tau^Q$ is the in-sample expectile value and $S_\\mu Q$ is a smoothed value obtained by averaging over small-radius (in-sample-oriented) and larger-radius (OOD-oriented) perturbations of a fixed target action. The difference $S_\\mu Q - V_\\tau^Q$ is the 'local correction,' and $\\lambda$ scales how much of that generalization enters each recursive update while the discount stays $\\gamma$. The paper proves an exact one-step identity, a time-varying iterate bound, and a fixed-point bound $\\|Q^*_{\\mathrm{CSDG}} - Q^*_{\\mathrm{In}}\\|_\\infty \\le \\gamma\\lambda e^*/(1-\\gamma)$, where $e^*$ is the branch discrepancy at the fixed point, and shows that the fixed points are value functions of explicit induced policies. It then translates these into a practical algorithm using asymmetric bounded noise and expectile regression, without exact support classification or a pessimistic OOD penalty.","pith_inferences":["The dual-target decomposition is likely portable: any in-sample learner that exposes an expectile or softmax value could accept the same correction term, and an adaptive $\\lambda$ scheduled from the measured branch discrepancy would tighten the iterate bound during training—neither of which the paper tests.","Because the fixed-point bound depends only on the branch discrepancy $e^*$, monitoring $\\|S_\\mu Q - V_\\tau^Q\\|_\\infty$ during training could serve as an early-warning diagnostic for value instability; the paper notes it does not track anchoring geometry during training.","The proof's fixed-kernel assumption suggests the algorithm's stability is coupled to slow target-actor updates; a variant that adapts the smoothing kernels online would need a time-varying contraction argument beyond the one in the appendix."],"forward_implications":["A single scalar $\\lambda$ becomes an interpretable dial for how much OOD value enters each Bellman update, with the per-step contribution exactly $\\gamma\\lambda$ times the expected smoothed-versus-in-sample gap.","Because the operator is a $\\gamma$-contraction under fixed kernels, CSDG inherits the stable fixed-point behavior of in-sample learning rather than the overestimation typical of OOD maximization.","The induced-policy result gives a testable non-degradation criterion: if $\\lambda g_\\mu \\ge (1-\\lambda)\\xi$, the CSDG fixed point does not underperform the in-sample reference.","The practical recipe needs no behavior model, no exact support classification, and no additional pessimistic penalty, so it can be grafted onto in-sample methods; the paper shows it improves both IQL and XQL on every evaluated locomotion task."],"supporting_citations":[{"why":"Defines the state-conditional CHN geometry and smooth OOD generalization that CSDG adopts for local actions.","marker":"Yao et al., 2025"},{"why":"DMG shows how a generalized value can be blended with an in-sample value during bootstrapping; CSDG turns this blend into an explicit correction.","marker":"Mao et al., 2024"},{"why":"IQL supplies the expectile in-sample value functional used as the reference path and as one backbone.","marker":"Kostrikov et al., 2021"},{"why":"XQL is the second in-sample backbone that CSDG extends with the two-scale smoothed target.","marker":"Garg et al., 2023"},{"why":"D4RL provides the 21 benchmark datasets and evaluation protocol behind the reported 1199.4 and 445.8 aggregates.","marker":"Fu et al., 2020"},{"why":"CQL represents the pessimistic-penalty baseline that CSDG contrasts with by adding no OOD penalty.","marker":"Kumar et al., 2020"}],"fun_headline_variants":["Offline RL: in-sample value plus a safe gated OOD correction","CSDG: one dial controls OOD influence in offline RL backups","Decompose Bellman target to add controlled OOD generalization","Explicit OOD correction passes a mixture gate in offline RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bounds hold for a fixed deterministic target action selector and fixed perturbation kernels, whereas the implemented algorithm updates its target actor during training and uses one Monte Carlo sample per kernel, with a learned expectile network in place of the exact functional.","fun_headline_variants_meta":{"raw":{"variants":["Offline RL: in-sample value plus a safe gated OOD correction","CSDG: one dial controls OOD influence in offline RL backups","Decompose Bellman target to add controlled OOD generalization","Explicit OOD correction passes a mixture gate in offline RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1734,"prompt_tokens":1079,"completion_tokens":655,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":695,"completion_tokens_details":{"reasoning_tokens":580}},"tokens_in":695,"tokens_out":655,"duration_ms":5860,"temperature":1.0,"reasoning_tokens":580,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:51:58.282866+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 1 with $\\lambda=1$ and a fast-updating target actor, recording the empirical branch discrepancy $\\|S_\\mu Q - V_\\tau^Q\\|_\\infty$ and the distance from the learned critic to the in-sample critic's fixed point over iterations; if the distance grows faster than $\\gamma\\lambda\\sum_{t=0}^{k-1}\\gamma^{k-1-t}e_t$ or exceeds $\\gamma\\lambda e^*/(1-\\gamma)$, the fixed-kernel contraction premise is violated and the advertised bounds do not govern the trained critic.","supporting_citations":[{"cited_title":"Conservative q-learning for offline reinforcementlearning.Advancesinneuralinformationprocessingsystems,33:1179–1191,2020","cited_arxiv_id":null,"evidence_quote":"CQL represents the pessimistic-penalty baseline that CSDG contrasts with by adding no OOD penalty."}],"review_version":2}