{"id":"b3e5a6cb-731e-49a6-a31a-92ca6a88ba6e","arxiv_id":"2506.08463","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"R2CSL provably reaches the in-distribution optimal stitched policy by conditioning on the maximum return-to-go per state, improving on standard RCSL without dynamic programming.","lead":"Reinforced RCSL modifies return-conditioned supervised learning by conditioning the policy on the best return achievable at each state in the offline dataset, rather than a fixed target return. The paper proves this stitched policy is at least as good as any consistent RCSL policy and demonstrates modest gains on D4RL benchmarks.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.1 is proven only for deterministic transitions; in stochastic MDPs conditioning on the in-distribution max RTG does not guarantee that return, so the central domination claim lacks support outside deterministic environments.","rationale":"The reader's weakest-assumption analysis correctly identifies the load-bearing gap. The central claim of the paper is that R2CSL provably dominates all consistent RCSL policies; that claim is Theorem 4.1. Its proof in Appendix A.1 uses a singleton successor state, which is legitimate only under deterministic transition dynamics. In stochastic MDPs, the in-distribution optimal RTG f* is an observed maximum along a particular trajectory branch, not an expected return achievable by conditioning on f*. After the chosen action, the environment may branch to a state where the maximum RTG is lower, so pi*_beta can deliver less than f*. This breaks the comparison against consistent RCSL policies. The paper's own Section 9 admits that RCSL-style objective policies can be arbitrarily suboptimal in stochastic environments, which supports the concern. Section 5.2 and Theorem 5.5 do not address the gap: they only guarantee finite-sample convergence to pi*_beta, not that pi*_beta is a good policy or dominates pi_f. The deterministic tabular analysis (Theorems 5.3 and 6.2) is coherent and is a genuine contribution; the issue is the scope of the headline claim, not the internal logic of the deterministic argument. The appropriate remedy is either to restrict Theorem 4.1 to deterministic MDPs or to supply a stochastic proof with additional assumptions. Since the reader already returned CONDITIONAL, no verdict change is warranted; the condition should be the proof fix or scope restriction for Theorem 4.1.","tokens_in":26109,"tokens_out":11341,"duration_ms":144443,"concrete_test":"Analytical check: re-derive the proof of Theorem 4.1 (Appendix A.1) in a minimal stochastic MDP with H=2, deterministic rewards, and a transition distribution P(s'|s,a) with two reachable successors for the action selected by pi*_beta, whose observed RTGs differ. Replace the singleton successor s_2(s,a*_1) in A.1 by the distribution P(·|s,a*_1) and verify whether the induction still yields J(pi*_beta) >= E_{s~rho}[f*(s,1)]. The check lands if the inequality requires the condition P(s'|s,a) > 0 to imply f*(s',h+1) = f*(s,h) - r_h, which is exactly determinism (or an equality-over-support assumption not stated in Theorem 4.1). A numeric witness: behavior beta uniform over actions L and S, P(High|s0,L)=0.1, P(Medium|s0,S)=1, with terminal rewards 100, 0, and 60; compare J(pi*_beta) against J(pi_f) for a consistent f that selects S.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 4.1 is stated without a determinism assumption, but its proof in Appendix A.1 is explicitly deterministic: it reasons about 'the state transitioned from s by taking a*_1' as a single state and asserts that the chosen action yields return f*(s,1) - r. In a stochastic MDP, after selecting the action that realizes the in-distribution maximum RTG f*(s,h), the next state is sampled from P(·|s,a); the realized state s' need not be the branch on which f*(s,h) was observed, and f*(s',h+1) can be strictly smaller than f*(s,h) - r. Hence V^{pi*_beta}(s) can be strictly below f*(s,1), and the chain J(pi*_beta) >= E[f(s,1)] >= J(pi_f) breaks at the first inequality. Theorem 5.5 only bounds J(pi*_beta) - J(hat_pi*_D) in terms of estimation error; it does not establish that pi*_beta dominates consistent RCSL policies in stochastic environments. The paper itself concedes in Section 9 that such RCSL-style objective policies 'can be arbitrarily suboptimal' in stochastic environments. Therefore the central claim that R2CSL provably improves over every consistent RCSL policy is currently supported only in deterministic environments; the stochastic analysis in Section 5.2 and the D4RL experiments do not repair the proof gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Reinforced RCSL (R2CSL), an offline RL method that conditions a return-conditioned supervised learning policy on an \"in-distribution optimal RTG\" f*(s,h) defined as the maximum return-to-go achievable under the behavior policy's feasible trajectory set. The central theoretical claim (Theorem 4.1) is that the resulting policy pi*_beta dominates every consistent RCSL policy pi_f, providing a provable stitching guarantee without dynamic programming. The paper gives finite-sample bounds for a tabular deterministic setting (Theorem 5.3) and a general stochastic setting with function approximation (Theorem 5.5), analyzes expectile versus quantile regression for estimating f* (Section 6), and proposes a multi-step relabeling scheme claimed to recover the optimal in-distribution policy (Theorem 8.1). Experiments on PointMaze, D4RL Gym, and AntMaze report improvements over RvS and DT baselines.","tokens_in":26380,"tokens_out":3476,"duration_ms":40828,"significance":"If the main claim is correct in the stated generality, this would be the first provable stitching guarantee for RCSL that avoids dynamic programming, a significant conceptual contribution to offline RL theory. The paper also provides a useful comparison of expectile and quantile estimators for the conditioning function and shows that quantile regression can recover the in-distribution maximum RTG in a deterministic tabular setting. The empirical study is reasonably broad, and the theoretical results, while building on existing RCSL analysis, address a real gap. However, the central domination claim is currently proven only under deterministic transitions, and the stochastic analysis does not establish that R2CSL dominates consistent RCSL policies in stochastic environments.","major_comments":[{"comment":"Theorem 4.1 is stated for general MDPs, but the proof in Appendix A.1 is explicitly deterministic. The proof reasons about 'the state transitioned from s by taking a*_1' as a single state, which is only valid when P(s'|s,a) is deterministic. In a stochastic MDP, after selecting the action realizing f*(s,h), the next state s' is random, and f*(s',h+1) can be strictly smaller than f*(s,h)-r. Consequently the chain J(pi*_beta) >= E[f(s,1)] >= J(pi_f) breaks at the first inequality. The theorem as stated is therefore unsupported; the paper should either restrict Theorem 4.1 to deterministic environments or provide a genuinely stochastic proof with appropriate assumptions.","section":"Theorem 4.1 and Appendix A.1"},{"comment":"The stochastic analysis does not repair the gap in Theorem 4.1. Theorem 5.5 only bounds J(pi*_beta) - J(hat_pi*_D), the error relative to the in-distribution optimal stitched policy pi*_beta. It does not show that pi*_beta dominates all consistent RCSL policies in stochastic environments; indeed the paper itself concedes in Section 9 that such objective policies 'can be arbitrarily suboptimal' and cites prior work to this effect. Thus the central contribution, 'provably improve RCSL,' is established only for deterministic transitions. The authors should clearly state this limitation in the abstract and introduction, or extend the analysis.","section":"Section 5.2 and Theorem 5.5"},{"comment":"The proof of Theorem 8.1 is informal and narrative rather than a rigorous induction. It asserts that after H-1 relabeling passes one has ~g^{H-1}_h = Q*_beta_h(s_h,a_h), but the argument is a sequence of case discussions ('Q1', 'Q2', 'case (i)', 'case (ii)') without a formal inductive hypothesis or a precise definition of what dataset distribution the relabeled trajectories induce. Since Theorem 8.1 supports the claim of closing the gap with dynamic programming, it needs a complete proof or an explicit label as a proof sketch, with the formal details supplied in an appendix.","section":"Appendix A.6, Theorem 8.1"},{"comment":"The proof of Theorem 6.2 in Appendix A.5 shows that the number of samples at f*(s,h) is large and concludes that the alpha-quantile equals f*(s,h). This relies on the deterministic, no-tie assumption in Assumption 6.1, which is stated, but the proof does not explicitly handle the case where multiple actions or trajectories attain the conditioning value, nor does it connect the empirical quantile from the L1 loss to the population quantile under the assumed coverage. A precise statement of why the finite-sample quantile estimate coincides with f* with high probability would strengthen the result; currently the proof is incomplete at the step 'then setting alpha > 1 - c~/2, the alpha-quantile is exactly f*(s,h)'.","section":"Section 6, Theorem 6.2 proof"}],"minor_comments":[{"comment":"In the sentence 'We set α > 1−˜c/2 in the Lα2 loss with quantile regression', the quantile loss is L1, not Lα2; the notation should be corrected.","section":"Section 6, before Theorem 6.2"},{"comment":"Theorems are referenced inconsistently: 'Assume Theorems 5.1 and 5.2 hold' should read 'Assumptions 5.1 and 5.2'; likewise 'satisfy Theorem 5.4' should be 'Assumption 5.4'. This makes the logical dependencies harder to follow.","section":"Sections 5.1 and 5.2"},{"comment":"The proof of Theorem 4.1 says 'By Corollary 2 of [1], we have J(pi^{RCSL}_f) = E[f(s,1)]' and later 'it is trivial that the trajectory induced by pi* is better', but the deterministic argument is not written in formal notation; a concrete induction on h would significantly improve clarity.","section":"Appendix A.1"},{"comment":"The text states that with 1% Type I trajectories α=0.90 fails, but the caption of Figure 4 mentions α=0.85 for the failure case; please align the caption with the main text.","section":"Section 7.1, Figure 4 caption"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about Theorem 4.1 is valid and is the main technical gap. The paper explicitly acknowledges in Section 9 that RCSL-style objective policies can be arbitrarily suboptimal in stochastic environments, which directly contradicts the unqualified phrasing of the central claim in the abstract and introduction. I would suggest the editor ask the authors to either (a) restrict all domination claims to deterministic environments and clearly revise the abstract, or (b) provide a stochastic version of Theorem 4.1 with additional assumptions. The informal proof of Theorem 8.1 also needs to be formalized before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on 2506.08463. The paper is worth your time for the deterministic-environment analysis: it formalizes the in-distribution optimal return-to-go and shows that conditioning on it, R2CSL provably dominates standard RCSL in deterministic tabular MDPs. Theorem 5.3's sample-complexity bound is coherent, and the expectile-versus-quantile regression comparison in Theorem 6.2 is careful. The multi-step relabeling idea in Section 8 is a nice conceptual bridge to dynamic programming, even though its proof is more narrative than formal.\n\nThe main problem is Theorem 4.1. It is stated for general MDPs, but the proof in Appendix A.1 explicitly assumes deterministic transitions. In a stochastic MDP, the action that maximizes the in-distribution RTG at (s,h) need not lead to the state that realizes that RTG—the next state is random and f* at that next state can be lower. So the chain J(pi*_beta) >= E[f(s,1)] has no basis, and the claimed domination over all consistent RCSL policies is unsupported. The paper's own Section 9 concedes that RCSL-style objective policies can be arbitrarily suboptimal in stochastic environments, which underscores why this gap matters. Theorem 5.5 does not repair it; that bound only shows convergence to pi*_beta, and pi*_beta carries no guaranteed advantage over RCSL in stochastic settings.\n\nThat said, this is a proof gap rather than a sign of careless thinking. The deterministic results are solid and new, the related work is engaged honestly, and the paper is transparent about the stochastic limitation. The experiments are modest, high-variance, and come without code, but they are consistent with the theory.\n\nI would send this to peer review. A serious referee should ask the authors to either restrict Theorem 4.1 to deterministic environments or supply a correct stochastic proof. With that fix, this becomes a useful contribution to the theory of RCSL; without it, the central overclaim should not stand.","headline":"Genuine deterministic-environment results for RCSL stitching, but the central theorem's stochastic claim is unsupported by its own proof; a fixable overreach that still deserves peer review.","tokens_in":26930,"tokens_out":2306,"would_cite":true,"duration_ms":28697,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","68Q32"],"pacs":[],"model":"deepseek-v4-flash","headline":"Provable stitching for return-conditioned reinforcement learning, without dynamic programming.","keywords":["return-conditioned supervised learning","stitching","offline reinforcement learning","in-distribution optimal return-to-go","quantile regression","expectile regression","sample complexity","dynamic programming"],"falsifier":"Construct a two-stage MDP with a single start state $s_1$, reward $r(s_1,a)=0$, and two actions: action $L$ leads with probability 0.5 to a state with max return-to-go 10 and with probability 0.5 to a state with max return-to-go 0; action $R$ leads deterministically to a state with max return-to-go 9. Then $f^\\star(s_1,1)=10$ via action $L$, so the policy $\\pi^\\star_\\beta$ conditions on 10 and its expected return is $0.5\\cdot 10 + 0.5\\cdot 0 = 5$, which is below $J(\\pi_f)=9$ for the consistent $f$ that picks action $R$. This directly contradicts Theorem 4.1's inequality in a stochastic environment.","tokens_in":25870,"feed_emoji":"🧵","tokens_out":5580,"duration_ms":61993,"temperature":0.7,"pith_summary":"This paper proposes Reinforced RCSL (R2CSL), a modification of return-conditioned supervised learning (RCSL) that replaces the test-time conditioning return with the largest return-to-go that actually appears in the offline dataset at the current state. The central claim is that the resulting “in-distribution optimal stitched policy” has value at least as high as any RCSL policy whose conditioning function satisfies the classical consistency condition, giving the first provable stitching guarantee for an RCSL-style method that does not use dynamic programming. If correct, it would mean that a purely supervised method can assemble better-than-dataset trajectories from suboptimal segments, at the same sample-complexity order as ordinary RCSL. The paper also shows that iterating a return-relabeling step $H-1$ times recovers the optimal in-distribution policy in deterministic environments, closing a theoretical gap with dynamic programming methods.","feed_headline":"Conditioning on the best in-data return provably stitches","feed_subtitle":"R2CSL shows return-conditioned RL can beat the dataset ceiling at the same sample cost, no dynamic programming needed.","key_machinery":"The load-bearing object is the in-distribution optimal return-to-go, $f^\\star(s,h) := \\arg\\max_{f \\in \\mathcal{F}_\\beta(s,h)} f(s,h)$, i.e., the largest return-to-go that appears in any trajectory of the behavior policy passing through state $s$ at stage $h$. The in-distribution optimal stitched policy is $\\pi^\\star_\\beta(a|s,h) := P_\\beta(a|s,h,f^\\star(s,h))$: the behavior policy's action distribution conditioned on that maximal achievable return. It carries the argument because conditioning on $f^\\star$ at each step lets the policy select the best future from whichever trajectory offers it, without ever querying an out-of-distribution return. A second mechanism is the relabeling rule $\\tilde{g}^k_h = \\max\\{ r_h + f^\\star_{k-1}(s_{h+1},h+1),\\, r_h + \\tilde{g}^k_{h+1} \\}$, which propagates best future returns backward; after $H-1$ passes it yields $Q^{\\star,\\beta}_h(s_h,a_h)$ in deterministic environments, showing the gap to dynamic programming closes.","core_discovery":"The paper's core discovery is that the ceiling RCSL inherits from the dataset comes from the consistency condition on the conditioning function $f$, not from supervised learning itself. By defining $f^\\star(s,h)$ as the maximum return-to-go over all conditioning functions that are feasible under the behavior policy—that is, the best future return that occurs in any dataset trajectory through $(s,h)$—and conditioning the learned policy on $f^\\star$, the induced policy $\\pi^\\star_\\beta$ is proved to satisfy $J(\\pi^\\star_\\beta) \\ge J(\\pi_f)$ for every consistent $f$ (Theorem 4.1). This is the first proof that an RCSL-style method can stitch: combine pieces of different trajectories to exceed the best trajectory in the data. Finite-sample versions show R2CSL converges to $\\pi^\\star_\\beta$ at rate $N^{-1/4}$ with the same coverage assumptions as ordinary RCSL (Theorem 5.3), and with quantile regression the in-distribution optimal RTG is recovered exactly once the dataset covers it (Theorem 6.2).","pith_inferences":["The domination theorem is proven under deterministic transitions; in stochastic environments the conditioning function $f^\\star(s,h)$ can exceed the return any policy can guarantee, so the practical gains in stochastic benchmarks likely rest on the policy's ability to hedge rather than on Theorem 4.1 itself.","The relabeling rule (8.1) suggests a natural stochastic extension: replace the max with an expectile or quantile of the next-state value and test whether the $H-1$-pass optimality persists with a risk-sensitive objective.","Since the only role of $f^\\star$ is to pick the strongest feasible conditioning signal, the same construction could be applied to goal-conditioned supervised learning, where the “return” is a goal-achievement score rather than cumulative reward."],"forward_implications":["R2CSL policies can exceed the best trajectory in the dataset by stitching together high-return segments from different trajectories, breaking the ceiling that ordinary RCSL inherits.","The improvement comes at the same sample-complexity order as classical RCSL: the $N^{-1/4}$ rate and the coverage constants $c^\\star_\\beta$ and $\\tilde{c}$ play the same roles as the corresponding terms in the prior RCSL analysis.","Quantile regression, not expectile regression, is the correct estimator for the optimal RTG: with enough data it recovers $f^\\star$ exactly, while expectile regression can learn out-of-distribution returns and fail.","The multi-step relabeling variant achieves the in-distribution optimal policy after $H-1$ passes, so RCSL-style methods can match dynamic-programming-based methods' guarantee in deterministic settings.","The framework is flexible enough to be layered onto RvS, DT, and QT-style architectures, improving each in the reported benchmarks."],"supporting_citations":[{"why":"Supplies the consistency and in-distribution conditions for RCSL conditioning functions, the non-stitching theorem, and the sample-complexity bound that R2CSL improves on.","marker":"[1]"},{"why":"Defines Decision Transformer, the RCSL architecture used to instantiate the policy class in the DT-R2CSL variant.","marker":"[2]"},{"why":"Defines RvS and the target-RTG inference procedure that R2CSL replaces with the in-distribution optimal RTG.","marker":"[3]"},{"why":"Provides expectile regression, one of the two conditioning-function estimators analyzed; the paper shows it can produce out-of-distribution RTGs.","marker":"[22]"},{"why":"Provides quantile regression, the estimator proved to recover the in-distribution optimal RTG exactly.","marker":"[14]"},{"why":"Provides a dynamic-programming baseline (CQL) whose optimality guarantee the multi-step R2CSL result is compared against.","marker":"[17]"},{"why":"Provides the QT hybrid baseline in experiments and the comparison point for R2CSL variants that incorporate dynamic programming components.","marker":"[10]"}],"fun_headline_variants":["Reinforced RCSL proves return conditioning can stitch","In-distribution optimal return breaks RCSL's dataset ceiling","Provable stitching for return-conditioned learning, no dynamic programming","R2CSL: Conditioning on best in-data future return beats standard RCSL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"For the main domination theorem, the proof assumes deterministic transitions, so that $f^\\star(s,h)$ is actually achievable by the policy that conditions on it; if the environment branches stochastically, the policy can fail to realize the max return-to-go it was told to aim for.","fun_headline_variants_meta":{"raw":{"variants":["Reinforced RCSL proves return conditioning can stitch","In-distribution optimal return breaks RCSL's dataset ceiling","Provable stitching for return-conditioned learning, no dynamic programming","R2CSL: Conditioning on best in-data future return beats standard RCSL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000283,"raw_usage":{"total_tokens":1683,"prompt_tokens":971,"completion_tokens":712,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":637}},"tokens_in":587,"tokens_out":712,"duration_ms":8708,"temperature":1.0,"reasoning_tokens":637,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:11:55.891282+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a two-stage MDP with a single start state $s_1$, reward $r(s_1,a)=0$, and two actions: action $L$ leads with probability 0.5 to a state with max return-to-go 10 and with probability 0.5 to a state with max return-to-go 0; action $R$ leads deterministically to a state with max return-to-go 9. Then $f^\\star(s_1,1)=10$ via action $L$, so the policy $\\pi^\\star_\\beta$ conditions on 10 and its expected return is $0.5\\cdot 10 + 0.5\\cdot 0 = 5$, which is below $J(\\pi_f)=9$ for the consistent $f$ that picks action $R$. This directly contradicts Theorem 4.1's inequality in a stochastic environment.","supporting_citations":[{"cited_title":"When does return-conditioned supervised learning work for offline reinforcement learning? Advances in Neural Information Processing Systems, 35:1542–1553, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the consistency and in-distribution conditions for RCSL conditioning functions, the non-stitching theorem, and the sample-complexity bound that R2CSL improves on."},{"cited_title":"Decision transformer: Reinforcement learning 1For two distributionsPandQ,P≪QmeansPis absolutely continuous w.r.t","cited_arxiv_id":null,"evidence_quote":"Defines Decision Transformer, the RCSL architecture used to instantiate the policy class in the DT-R2CSL variant."},{"cited_title":"Rvs: What is essential for offline rl via supervised learning? InInternational Conference on Learning Representations,","cited_arxiv_id":null,"evidence_quote":"Defines RvS and the target-RTG inference procedure that R2CSL replaces with the in-distribution optimal RTG."},{"cited_title":"Asymmetric least squares estimation and testing","cited_arxiv_id":null,"evidence_quote":"Provides expectile regression, one of the two conditioning-function estimators analyzed; the paper shows it can produce out-of-distribution RTGs."},{"cited_title":"Quantile regression.Journal of economic perspectives, 15(4):143–156, 2001","cited_arxiv_id":null,"evidence_quote":"Provides quantile regression, the estimator proved to recover the in-distribution optimal RTG exactly."},{"cited_title":"Conservative q-learning for offline reinforcement learning.Advances in Neural Information Processing Systems, 33: 1179–1191, 2020","cited_arxiv_id":null,"evidence_quote":"Provides a dynamic-programming baseline (CQL) whose optimality guarantee the multi-step R2CSL result is compared against."},{"cited_title":"Q-value regularized transformer for offline reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Provides the QT hybrid baseline in experiments and the comparison point for R2CSL variants that incorporate dynamic programming components."}],"review_version":1}