{"id":"0d9b0f22-cc5b-4e60-9d87-4e4d72464bcf","arxiv_id":"2412.07762","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Warm-start RL (WSRL) fine-tunes offline-pretrained RL agents online with no offline data retention, using 5,000 warm-up rollouts from the frozen pre-trained policy followed by standard high-UTD SAC.","lead":"This paper shows that reinforcement learning agents pre-trained on offline data can be fine-tuned online without keeping the offline dataset, if the online run is seeded with a short warm-up of rollouts from the pre-trained policy. The method, WSRL, prevents the sudden Q-value collapse that otherwise destroys the pre-trained initialization, and it matches or beats approaches that do retain offline data on several benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The fixed K=5,000 warmup length is the load-bearing assumption: WSRL's 'recalibration' is not theoretically guaranteed, and the paper's own K-sweep (Fig. 19) and Appendix O show the method is sensitive to K and to pretraining quality, so the universal no-retention claim is conditional.","rationale":"After reading the paper in full, I find the empirical case for WSRL on the tested D4RL domains and the real robot credible: warmup ablations (Figure 10, Appendix D), Q/policy initialization ablations (Figures 12-13), and the pre-training-algorithm agnosticism (Appendix H) are consistent with the stated mechanism. The central claim is nevertheless narrower than the abstract suggests. The success of WSRL depends on the warmup phase providing enough on-policy data to keep early SAC updates from amplifying pessimistic TD targets, and there is no formal argument or diagnostic for choosing K. The paper's own K-sweep shows environment-dependent sensitivity, and Appendix O concedes failure when the offline initialization is poor. These are exactly the conditions that make the 'no-retention is always sufficient' claim conditional. This matches the reader's weakest assumption, so I agree with the CONDITIONAL verdict rather than proposing a stronger one.","tokens_in":21362,"tokens_out":9799,"duration_ms":109955,"concrete_test":"Run WSRL on a held-out set of sparse-reward tasks (e.g., Adroit-pen-binary-v0 and a stochastic variant of Kitchen-partial) with K in {0, 1,000, 5,000, 20,000}, 10 seeds each, and report final success and steps-to-threshold. If no single K is within, say, 5% of the best per-domain K, the fixed-K=5,000 recipe is not domain-agnostic. Also instrument the first 10,000 post-warmup critic updates: if the minimum TD target falls below the pre-trained Q-values by a pre-specified margin even with K=5,000, the 'recalibration prevents downward spiral' mechanism is not supported and the method's success must be explained differently.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the warmup phase of K=5,000 rollouts from the frozen pre-trained policy being sufficient to 'recalibrate' the offline Q-function to the online distribution (Section 4, Algorithm 1). Algorithm 1 performs no gradient updates during warmup; the Q-function is frozen for the first K steps, so the recalibration is actually carried out by the first SAC updates after step K. The paper gives no guarantee that K=5,000 warmup transitions prevent the pessimistic TD-target spiral (Section 3.1, Figures 4 and 5) for a given offline initialization. The support is empirical: Figures 7, 11, and 8. The paper's own ablations bound this assumption: Appendix D Figure 19 shows that K=1,000 can be unstable on Door-binary while K=20,000 hurts Kitchen-mixed, so K=5,000 is a selected middle value, and Appendix O shows WSRL fails when pre-training is poor (Adroit-binary-v0). Thus the universal version of the abstract's claim is not established; it is conditional on K being adequate for the domain and on the pre-trained policy/Q-function being informative.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies online RL fine-tuning of an offline pre-trained policy and Q-function without retaining the offline dataset. It first analyzes why existing offline-to-online methods fail in this \"no-retention\" setting, attributing the failure to a Q-value underestimation spiral caused by distribution shift and pessimistic TD targets at the onset of fine-tuning. It then proposes WSRL, which initializes the online replay buffer with K=5,000 rollouts from the frozen pre-trained policy before switching to standard high-UTD SAC fine-tuning. The paper reports that WSRL outperforms prior methods on D4RL Antmaze, Kitchen, Adroit, and MuJoCo tasks, and shows a real-robot peg insertion result. Appendices provide ablations on warmup length, warmup data source, policy/Q initialization, offline algorithm choice, UTD ratio, and robustness to poor pre-training.","tokens_in":21669,"tokens_out":3883,"duration_ms":39738,"significance":"If the central claim holds, the paper makes a useful conceptual and practical contribution: it identifies Q-value recalibration as the key failure mode in no-retention fine-tuning and shows that a simple warmup procedure can avoid it without retaining offline data. The paper is strong on empirical breadth and transparency: code is released, ablations are extensive, the KL-divergence analysis attempts to separate unlearning from forgetting, and the real-robot result gives out-of-distribution credibility. The main weakness is that the universal claim in the abstract is stronger than the evidence: the method depends on a fixed and domain-sensitive warmup length K and on the pre-trained policy/Q-function being reasonably informative, and the main figures do not report error bars or seed counts. These issues make the paper suitable for major revision rather than acceptance in its current form.","major_comments":[{"comment":"The claim that WSRL works \"irrespective of whether [previous methods] retain offline data or not\" is not supported by the paper's own ablations. Appendix D Figure 19 shows that K=1,000 can be unstable on Door-binary and K=20,000 hurts Kitchen-mixed, so K=5,000 is a selected middle value rather than a robust default. Since Algorithm 1 takes K as a fixed input and no criterion is provided for choosing K on a new domain, the universal version of the central claim should be revised to a conditional statement about settings where the warmup length is adequate.","section":"Abstract, Section 5.2, Appendix D (Figure 19)"},{"comment":"The mechanism of \"recalibration\" is underspecified. In Algorithm 1, no gradient updates occur during the warmup phase; the Q-function is frozen for the first K steps, so any recalibration is actually carried out by the first SAC updates after step K. The paper does not provide an argument, formal or diagnostic, that K transitions from the frozen policy will prevent the pessimistic TD-target spiral described in Section 3.1 (Figures 4 and 5). Figure 11 supports the claim on one environment, but a criterion for detecting or guaranteeing successful recalibration would be needed to substantiate the mechanism.","section":"Algorithm 1, Section 4, Section 3.1"},{"comment":"Appendix O shows that WSRL completely fails on Adroit-binary-v0 and performs poorly on Kitchen-complete-v0 when CalQL pre-training is unsuccessful. The paper acknowledges this, but the main text and abstract do not carry the resulting scope restriction. As written, the claim that WSRL can fine-tune without retaining offline data is conditional on the pre-trained policy and Q-function being informative; this boundary condition should be stated explicitly in the main text.","section":"Appendix O, Figure 30, Section 5.2"},{"comment":"No error bars, standard deviations, or seed counts are reported in the main experimental figures. Fine-tuning comparisons in this setting are known to be high-variance, and the paper's central claims of \"faster\" and \"higher asymptotic performance\" rest on curve comparisons. The paper should report means and standard errors over at least 3-5 seeds, or explicitly state if the curves are single runs, and justify that the reported differences are robust.","section":"Section 5, Figures 7, 9, 10, 12, 13"}],"minor_comments":[{"comment":"There is a typo: \"Recolate-binary-v0\" should be \"Relocate-binary-v0\".","section":"Appendix O"},{"comment":"There are typos: \"perfrom\" should be \"perform\" and \"paris\" should be \"pairs\".","section":"Appendix C and Appendix G"},{"comment":"The label \"Doffline\" appears to be a rendering artifact of \"Doffline\" or \"Offline\"; please fix.","section":"Figure 2"},{"comment":"References [3] and [4] are duplicate entries for the same paper (Ball et al., arXiv:2302.02948); they should be merged.","section":"References"},{"comment":"The note that some ablation experiments use a Kitchen version with maximum episode length 1000 while main comparisons use 280 is important for reproducibility; please state this in the main text or in a clearly visible setup paragraph so readers do not misinterpret cross-figure differences.","section":"Appendix I (Kitchen episode length)"},{"comment":"The real-robot comparison reports success counts over 20 initial poses but no indication of variance across repeats or training runs; a sentence on the number of runs and variability would strengthen the claim.","section":"Table 1 (real robot)"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within the scope of the journal and the novelty is acceptable: WSRL is simple but the problem framing and the empirical analysis of the no-retention regime are valuable. The main reason for major revision is the gap between the strong universal claim in the abstract and the conditional evidence in the paper's own ablations, together with the absence of error bars in the central comparisons. I would not reject the paper, because the core empirical finding is plausible and the method is clearly described; a revised version with a scoped claim and statistical reporting would be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid empirical paper that deserves a serious referee. The new thing is the no-retention fine-tuning setting itself, plus the finding that a short warmup with the frozen pre-trained policy is enough to avoid the Q-value \"downward spiral\" that otherwise destroys pre-trained initializations when you discard the offline data. The analysis in Section 3 is genuinely useful: they show the divergence is a distribution-shift problem, not just a pessimism problem, and that CalQL still fails without data retention. The main results across D4RL domains, the ablations (warmup length, policy vs Q init, online algorithm, UTD, dataset warmup), and the real-robot peg insertion are convincing evidence that the mechanism is real and the method works when the pre-trained policy is informative. Code is released.\n\nSoft spots, in proportion. First, the abstract overclaims: \"irrespective of whether they retain offline data or not\" is too strong. Their own Appendix O shows WSRL fails when pre-training fails (Adroit-binary-v0), and Appendix D Figure 19 shows K=5,000 is a sweet spot: 1k can be unstable on Door, 20k hurts on Kitchen. So the method is conditional on having a decent pre-trained Q/policy and on K being adequate. That is fine for an empirical methods paper, but the framing should say so. Second, no error bars or seed counts in the main figures; for a paper whose central claim is empirical superiority, that is a real reporting gap. Third, the \"recalibration\" story is mechanistic and plausible, but there is no theoretical guarantee; the warmup phase collects data, but the actual recalibration happens after the first SAC updates post-K. This is a minor exposition point, and the paper is honest that the support is empirical.\n\nOverall, the central claim holds up for what it is: a simple, well-validated recipe for no-retention fine-tuning, not a universal law. I would send it to review and ask for error bars, a tempered abstract, and a discussion of when K and pre-training quality matter. A serious referee will get value from this paper.","headline":"A solid, well-validated empirical paper that makes a real case for no-retention fine-tuning with a short warmup, but the universal claim is overstated and the method is conditional on decent pre-training and a tuned K.","tokens_in":22150,"tokens_out":2212,"would_cite":true,"duration_ms":21199,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that online RL fine-tuning can discard the offline pre-training dataset entirely, provided the replay buffer is first seeded with a few thousand rollouts from the frozen pre-trained policy.","keywords":["offline-to-online RL","no-retention fine-tuning","warm-start RL","Q-value divergence","catastrophic forgetting","high update-to-data ratio","sample-efficient reinforcement learning","real-robot RL"],"falsifier":"Take one of the paper's own tasks, discard the offline dataset after pre-training, and run WSRL while monitoring the TD error on the discarded data as in the paper's Figure 3; if that error grows large and the policy's success rate collapses without recovering, the warm-up claim is wrong for that case. More sharply, find any environment with healthy pre-training where 5,000 warm-up steps are insufficient to keep Q-values from diverging, and the fixed-warmup recipe is falsified.","tokens_in":21179,"feed_emoji":"🤖","tokens_out":6770,"duration_ms":58686,"temperature":0.7,"pith_summary":"Reinforcement-learning systems are usually pre-trained on static offline data and then fine-tuned online, but practical recipes keep the offline data in the replay buffer throughout fine-tuning. This paper argues that retaining offline data is unnecessary: it is only needed at the very start of fine-tuning to stop the value function from diverging when online rollouts shift the state-action distribution. The paper's method, Warm-start RL (WSRL), replaces that data with a brief warm-up phase in which the frozen pre-trained policy collects about 5,000 online transitions before standard online RL begins. With this warm-up, the offline Q-function recalibrates to the online distribution and the pessimistic 'downward spiral' of value targets is avoided. The authors report that WSRL fine-tunes faster and reaches higher asymptotic performance than methods that keep the offline data, on simulated benchmarks and a real robot peg-insertion task.","feed_headline":"5,000 warmup steps replace the offline dataset","feed_subtitle":"A short warm-up with the frozen pre-trained policy prevents Q-value collapse, beating methods that keep offline data.","key_machinery":"The load-bearing mechanism is the warm-up phase: K=5,000 transitions collected by the frozen pre-trained policy at the start of fine-tuning, seeded into the replay buffer. These transitions stand in for the offline dataset, but are drawn from the distribution the online policy actually visits, so they bridge the distribution shift and give the Q-function a set of state-action pairs whose targets are not pessimistically suppressed. The second component is the choice of a standard, high-update-to-data online RL algorithm after warm-up, so that pessimism or behavioral constraints do not slow asymptotic learning. The paper also relies on a value-function ensemble and layer normalization to stabilize the high-update-to-data regime.","core_discovery":"The central claim is that the sole useful role of offline data during fine-tuning is to prevent a sudden divergence of the Q-function at the onset of online training. The paper identifies a mechanism: when online rollouts are used without offline data, Bellman backups query the pre-trained Q-function on out-of-distribution state-action pairs; because pessimistic pre-training suppressed those values, the targets are too low, propagating underestimation and eventually destroying the pre-trained initialization. WSRL avoids this by collecting K=5,000 warm-up rollouts from the frozen pre-trained policy, which places near-on-policy data in the replay buffer and 'recalibrates' the Q-function before the online algorithm updates. After warm-up, WSRL runs a standard online actor-critic (Soft Actor-Critic with a high update-to-data ratio, an ensemble of Q-functions, and layer normalization) with no pessimism or behavioral constraints. The paper's experiments indicate that this both prevents forgetting and speeds up learning, outperforming methods that retain offline data.","pith_inferences":["Inference: This suggests a broader design rule for offline-to-online transitions: a short on-policy calibration phase, not necessarily fixed at 5,000 steps, may be the missing ingredient whenever the pre-trained value function was learned under pessimism or behavioral constraints.","Inference: The method inherits the quality of pre-training, as Appendix O shows when pre-training fails on a hard task, warm-up cannot create a useful signal; a natural extension would be to detect low-confidence pre-training and fall back to pure online RL.","Inference: The same recalibration idea might apply to continual RL or domain adaptation, where a policy is fine-tuned to a new task distribution without access to past experience, and warm-up rollouts from the old policy could play the role of the offline data.","Inference: Because the paper's result is empirical, a theoretical account of how the required warm-up length scales with distribution shift would take the claim beyond the fixed K=5,000 recipe."],"forward_implications":["Offline datasets can be discarded after pre-training, so large-scale RL fine-tuning no longer needs to store or batch through historical data.","Doing so is not merely tolerated: WSRL reports faster fine-tuning and higher final performance than algorithms that keep offline data across the benchmark tasks.","The value function recovers from an initial performance dip without losing the pre-trained initialization, as measured by KL divergence to the pre-trained policy and Q-function.","The recipe works with different offline pre-training algorithms (CQL, IQL, and CalQL), so WSRL is agnostic to the choice of offline initialization.","After warm-up, fine-tuning with a standard online RL algorithm is substantially better than continuing with a pessimistic offline RL algorithm, so the online algorithm choice is a deliberate part of the method."],"supporting_citations":[{"why":"Supplies the calibrated pre-training initialization used by default and the earlier observation that Q-value scale calibration matters for fine-tuning.","marker":"[37]"},{"why":"Provides the pessimistic offline RL algorithm (CQL) used both as a baseline and as a pre-training choice in the paper's experiments.","marker":"[29]"},{"why":"Provides the behavior-constrained offline RL algorithm (IQL) used as a baseline and as an alternative pre-training initialization.","marker":"[28]"},{"why":"Supplies the high-update-to-data online RL recipe with Q-ensemble and layer normalization that WSRL adopts, and a strong baseline that retains offline data.","marker":"[3]"},{"why":"Supplies the standard online actor-critic algorithm (Soft Actor-Critic) that WSRL runs after the warm-up phase.","marker":"[18]"},{"why":"Provides the closest baseline, a method that also uses a pre-trained policy to guide fine-tuning but discards the value function.","marker":"[55]"},{"why":"Provides the benchmark suite and task definitions used in the simulated experiments.","marker":"[12]"},{"why":"Provides the randomized ensembled Q-learning technique that WSRL uses to stabilize high-update-to-data training.","marker":"[6]"}],"fun_headline_variants":["Warm-up rollouts replace offline data in RL fine-tuning","No offline data needed: just 5,000 warm-up steps","Warm-start RL: fine-tune without offline data","5,000 warm-up steps beat keeping offline data","Warm-up phase lets you discard offline data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on the empirical assumption that 5,000 rollouts collected by the pre-trained policy are enough to stabilize the learned value function before ordinary online training begins; if the pre-training itself is poor, those rollouts do not carry useful information and the method fails.","fun_headline_variants_meta":{"raw":{"variants":["Warm-up rollouts replace offline data in RL fine-tuning","No offline data needed: just 5,000 warm-up steps","Warm-start RL: fine-tune without offline data","5,000 warm-up steps beat keeping offline data","Warm-up phase lets you discard offline data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000776,"raw_usage":{"total_tokens":3495,"prompt_tokens":1069,"completion_tokens":2426,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":685,"completion_tokens_details":{"reasoning_tokens":2345}},"tokens_in":685,"tokens_out":2426,"duration_ms":15563,"temperature":1.0,"reasoning_tokens":2345,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:30:37.592393+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the paper's own tasks, discard the offline dataset after pre-training, and run WSRL while monitoring the TD error on the discarded data as in the paper's Figure 3; if that error grows large and the policy's success rate collapses without recovering, the warm-up claim is wrong for that case. More sharply, find any environment with healthy pre-training where 5,000 warm-up steps are insufficient to keep Q-values from diverging, and the fixed-warmup recipe is falsified.","supporting_citations":[{"cited_title":"Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning.Advances in Neural Information Processing Systems, 36, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the calibrated pre-training initialization used by default and the earlier observation that Q-value scale calibration matters for fine-tuning."},{"cited_title":"Jump-startreinforcementlearning","cited_arxiv_id":null,"evidence_quote":"Provides the closest baseline, a method that also uses a pre-trained policy to guide fine-tuning but discards the value function."}],"review_version":1}