REVIEW 4 major objections 6 minor 12 references
Online Pre-Training for Offline-to-Online Reinforcement Learning
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Adding a second value function, pre-trained in a new Online Pre-Training phase, fixes the slow online fine-tuning of offline pre-trained agents and lifts average D4RL scores by about 30% on TD3 and SPOT.
desk verdict A clean, well-ablated plug-in for offline-to-online RL; the main caveats are a per-environment κ schedule and D4RL-only evidence for the meta-adaptation mechanism. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the second value function $Q_{\text{on-pt}}$ together with the Online Pre-Training phase that trains it. The phase collects $N_\tau$ online transitions with the frozen offline policy, then optimizes the meta-adaptation objective $\mathcal{L}_{\text{pretrain}}(\psi) = \mathcal{L}_{\text{off}}(\psi) + \mathcal{L}_{\text{on}}(\psi - \alpha \nabla \mathcal{L}_{\text{off}}(\psi))$, in which the online Bellman error (the temporal-difference error used to train value functions) is evaluated after one gradient step on the offline Bellman error, so that the critic is selected for fast adaptation to the online data distribution rather than for reproducing the offline critic. During fine-tuning the policy loss blends the two critics through the coefficient $\kappa$, which is linearly annealed from 0.1 toward 0.9, or fixed at 1 for random datasets, so that supervision shifts from the offline-trained critic to the online-adaptive one; balanced replay completes the machinery by ensuring the new critic is updated predominantly on recently collected samples.
What would settle it
Take the walker2d-random-v2 task, where OPT's gain is largest, and replace the meta-adaptation objective of Eq. (3) with the ordinary sum $\mathcal{L}_{\text{off}}(\psi)+\mathcal{L}_{\text{on}}(\psi)$ while keeping all hyperparameters and the Online Pre-Training phase identical; if this variant matches OPT's final score, the claim that the one-step gradient adaptation is what makes the new critic adaptive is falsified, since the paper's own ablation in Table 9 shows the same variant collapsing on Antmaze-large.
Extended reading notes
Core claim
On its own terms, the paper establishes that the slow performance improvement in offline-to-online RL stems from an offline-trained value function that misestimates values under the distribution shift of online data, and that introducing a second value function dedicated to the online regime resolves the problem. The offline critic $Q_{\text{off-pt}}$ is retained because it is reliable on the offline dataset, while the online critic $Q_{\text{on-pt}}$ is initialized from scratch and trained in Online Pre-Training so that it can adapt quickly to the evolving policy. The two are combined in the policy objective $-( (1-\kappa)Q_{\text{off-pt}} + \kappa Q_{\text{on-pt}} )$ with $\kappa$ scheduled from low to high over fine-tuning, and balanced replay ensures the new critic sees plenty of online samples. Empirically the paper shows this raises average normalized scores by roughly 30% on the TD3 and SPOT backbones across three D4RL domains, that the added critic reduces value-estimation bias early in training relative to a near-optimal reference, and that the module transfers to IQL and RLPD.
Load-bearing premise
The load-bearing premise is that the one-step meta-adaptation objective of Eq. (3), applied to a freshly initialized value function, makes $Q_{\text{on-pt}}$ track the online data distribution well enough; the paper supports this only with experiments, so under larger distribution shifts than those in D4RL, the advantage could vanish and OPT would reduce to balanced replay with an extra critic.
Editorial extensions
If this is right
- On TD3 and SPOT backbones, OPT raises the average normalized D4RL score by about 30% within 300k online steps, with large single-task jumps such as walker2d-random from 0.1 to 88.0 and door-cloned from 1.6 to 51.1.
- The added critic reduces value-estimation bias early in fine-tuning: bias measurements against a near-optimal reference value function show $Q_{\text{on-pt}}$ adapting faster than $Q_{\text{off-pt}}$.
- OPT transfers across backbone structures: IQL plus OPT improves by about 25% on average, and RLPD plus OPT beats both vanilla RLPD and its offline-to-online variant in total score.
- The ablations show that random initialization of the new critic, or pre-training it on online samples alone, both underperform the meta-adaptation pre-training, so the Online Pre-Training phase itself is what makes the second critic useful early on.
- The $\kappa$ schedule matters mainly as a gradual transition: fixed $\kappa = 0$, $0.5$, or $1$ all underperform the annealed schedule on Antmaze-large, while the exact endpoint makes little difference.
Reading between the lines
- Not tested in the paper: the same fresh-critic recipe should transfer to other distribution-shift transitions such as sim-to-real or changing reward functions, where the offline-to-online gap is larger than D4RL's; comparing OPT against random re-initialization with identical replay and $\kappa$ would reveal whether the mechanism survives a larger shift.
- The paper hand-schedules $\kappa$ per dataset quality; a data-driven $\kappa$, for instance based on the relative TD errors of the two critics, is a natural untested extension that could remove the per-domain tuning.
- Because the adaptation mechanism does not require the offline phase to converge, an untested consequence is that OPT should also accelerate purely online RL that starts from a small warm-start dataset, not just offline-to-online pipelines.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Online Pre-Training for Offline-to-Online RL (OPT), a three-phase method for value-based offline-to-online reinforcement learning. After standard offline pre-training of a policy and a value function Q_off-pt, OPT introduces a newly initialized value function Q_on-pt and trains it during an "Online Pre-Training" phase on a balanced mix of the offline dataset and N_tau online samples collected by the frozen offline policy. The training objective (Eq. 3) is a one-step meta-adaptation loss inspired by OEMA, combining an offline TD term and an online TD term at the adapted parameters. During online fine-tuning, the policy is updated against a convex combination of Q_off-pt and Q_on-pt with a scheduling coefficient kappa, and both value functions are updated with TD learning while balanced replay is used. The paper validates OPT on D4RL MuJoCo, Antmaze, and Adroit using TD3, SPOT, IQL, and RLPD as backbones, reporting an average 30% improvement over the backbones and competitive or superior results against prior offline-to-online methods.
Significance. The paper makes a clean empirical contribution. The controlled backbone comparisons (TD3 vs TD3+OPT at the same UTD ratio, SPOT vs SPOT+OPT, IQL vs IQL+OPT) are well designed and show consistent gains, and the ablations in Section 5 and Appendix C isolate the role of the new value function, the meta-adaptation objective, kappa scheduling, N_tau, and the replay buffer. The authors provide an open-source implementation, report IQM with confidence intervals, and test three structurally different backbone families, which supports the plug-in-module claim. However, the headline superiority over prior baselines and the general adaptability claim are not yet fully established: the UTD-ratio confound in the MuJoCo comparison and the per-environment tuning of kappa limit the strength of the conclusions. If the remaining issues are addressed, OPT would be a useful and simple module for offline-to-online RL.
major comments (4)
- [§4.1, Table 1] The MuJoCo headline comparison is not documented fairly. The paper states that TD3+OPT and TD3 are run with a UTD ratio of 5, but it does not state the UTD ratio used for Off2On, OEMA, PEX, ACA, FamO2O, and Cal-QL in Table 1. If those baselines are run at their default UTD ratio (often 1), the reported total-score advantage (939.1 vs 860.9 for Off2On) conflates the proposed mechanism with a higher number of gradient updates per environment step. The authors should report the UTD ratio for every baseline and, ideally, rerun the strongest baselines with UTD 5 so that the comparison isolates the OPT contribution.
- [§3.1, Eq. (3)] The central adaptability claim rests on an assumption that is only tested on D4RL. Eq. (3) uses a single inner gradient step on L_off to position Q_on-pt for a TD update on L_on, and the paper concludes that Q_on-pt "better adapt[s] to the changing dynamics of the online setting." All evidence for this is empirical on D4RL, where the offline buffers are broad and the policy shift is moderate; even the random datasets contain wide coverage. Because the online pre-training buffer is collected by the frozen policy pi_off, it is not evident that the one-step meta-adaptation will help when the online distribution is far from pi_off. I would like to see a targeted distribution-shift experiment, for example fine-tuning after changing the reward function or dynamics, or using a narrow-coverage offline dataset paired with a target behavior far from pi_off, with a comparison of Q_on-pt, random initialization, and Q_off-pt. Without such a test, the general adaptability claim is not established beyond the benchmark family.
- [§5.2 and Appendix H, Table 15] The method as evaluated requires an environment-specific kappa schedule. Table 15 sets kappa_init=1 for all MuJoCo random datasets, 0.3 for medium, 0.1 for medium-replay, Antmaze, and Adroit, and uses different T_decay values (100k, 150k, 200k, 250k). The paper explains this choice post hoc by dataset quality in §5.2. The ablations in Table 7 show robustness to small changes in the endpoint, but they do not test a single fixed schedule across all environments. For the claimed "general plug-in module," the authors should either provide an automatic rule for setting kappa or show that one reasonable schedule performs comparably across all D4RL domains.
- [Appendix D, Figures 9–10] The bias measurement that supports the mechanism is under-specified. The "optimal" reference is a TD3 agent trained "with sufficient steps," but no training length or achieved return is reported, and the bias is measured on only 10 fixed state-action pairs. Since Figures 9–10 are the only direct evidence that OPT reduces value-estimation bias, the reference should be fully specified (steps, seeds, performance), the number and selection of evaluation state-action pairs should be reported, and confidence intervals across seeds should be provided.
minor comments (6)
- [§3.1, Eq. (3)] The notation for the meta-adaptation step is ambiguous: specify whether the target network Q_bar_psi in the second term is evaluated at the original parameters psi or at the adapted parameters psi - alpha * grad L_off(psi), and whether gradients are stopped at the inner update.
- [§5.5] The text refers to "Figure 12" for the N_tau sweep, but Figure 12 in Appendix J is the wall-clock-time plot; the N_tau results appear in Figure 8 and Table 10. Please correct the cross-reference.
- [Appendix C.3, Table 9] Table 9 has a mislabeled caption: the table is an initialization-method ablation, not an N_tau ablation.
- [§5.3] The phrase "where contains few successful demonstrations" is ungrammatical; it should be "which contains" or "where the dataset contains."
- [Abstract and Appendix E] The abstract's "average 30% improvement" is not precisely defined; Appendix E reports per-domain gains that vary from 16.6% to 54.4%, so please state the aggregation rule, including which backbones and environments are included and whether per-task scores are averaged before or after computing the percentage.
- [Figure 1] The "from scratch" baseline is described as using a replay buffer initialized with the offline dataset, which is unusual terminology; please clarify in the caption what is meant by "from scratch" in this comparison.
Circularity Check
No significant circularity: OPT's claims are empirical and benchmark-validated, with no derivation that reduces to its own inputs.
full rationale
The paper's central claims are empirical: that a newly introduced value function Qon-pt, trained during an Online Pre-Training phase with the meta-adaptation objective of Eq. (3), improves offline-to-online fine-tuning. This is not a derivation from first principles that could reduce to its inputs by construction. Eq. (3) is an explicit design choice, described as 'inspired by the OEMA (Guo et al. 2023)', an external citation, and its benefit is supported by ablations (Tables 8–9, Figures 5, 7, 9–10) and comparisons against external baselines on D4RL, rather than by assuming the conclusion. The policy objective in Eq. (4) combines Qoff-pt and Qon-pt, but the adaptability of Qon-pt is measured, not posited as a theorem. The per-environment κ schedule in Table 15 is hyperparameter selection after seeing data, which may raise reproducibility concerns but is not circularity: the final scores are benchmark results, not quantities fitted to themselves. The only author-overlapping citation, Kim et al. (2024), appears in a list of prior work on extrapolation error and is not load-bearing for any claimed result. The paper also reports negative ablations (e.g., random initialization and Bon-only pretraining underperform), which independently indicate that its specific objective matters. Concerns about D4RL distribution-shift coverage are external-validity criticisms, not evidence that the paper's outputs are equivalent to its inputs.
Assumptions & free parameters
free parameters (5)
- κ schedule =
random: 1→1; medium: 0.3→0.9; medium-replay: 0.1→0.9; Antmaze: 0.1→0.9; Adroit: 0.1→0.9 (Table 15)
- Nτ (Online Pre-Training samples) =
25,000
- Npretrain (Online Pre-Training steps) =
50,000
- UTD ratio =
5
- α (meta-adaptation inner learning rate) =
not reported in main text (from OEMA)
assumptions (3)
- standard math Markov decision process with discounted return is an appropriate model for the D4RL tasks.
- domain assumption Offline pretrained value functions are inaccurate on out-of-distribution actions, and this inaccuracy is the cause of slow online fine-tuning.
- ad hoc to paper A single inner gradient step in the meta-adaptation objective (Eq. 3) is sufficient to make the new value function adapt to online samples.
Cite this review
Pith. "Pith review of Online Pre-Training for Offline-to-Online Reinforcement Learning." pith.science (2026). https://pith.science/paper/ZRJ25QH4
@misc{pith2026250708387,
author = {Pith},
title = {Pith review of: Online Pre-Training for Offline-to-Online Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZRJ25QH4}},
note = {Machine review of arXiv:2507.08387}
}
read the original abstract
Offline-to-online reinforcement learning (RL) aims to integrate the complementary strengths of offline and online RL by pre-training an agent offline and subsequently fine-tuning it through online interactions. However, recent studies reveal that offline pre-trained agents often underperform during online fine-tuning due to inaccurate value estimation caused by distribution shift, with random initialization proving more effective in certain cases. In this work, we propose a novel method, Online Pre-Training for Offline-to-Online RL (OPT), explicitly designed to address the issue of inaccurate value estimation in offline pre-trained agents. OPT introduces a new learning phase, Online Pre-Training, which allows the training of a new value function tailored specifically for effective online fine-tuning. Implementation of OPT on TD3 and SPOT demonstrates an average 30% improvement in performance across a wide range of D4RL environments, including MuJoCo, Antmaze, and Adroit.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[5]
Of- fline reinforcement learning with fisher divergence critic regularization
Kostrikov, I., Fergus, R., Tompson, J., and Nachum, O. Of- fline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774–5783. PMLR, 2021a. Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021b. Kumar, ...
-
[7]
Offline rein- forcement learning: Tutorial, review, and perspectives on open problems
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643,
arXiv 2005
-
[10]
Behavior regu- larized offline reinforcement learning
Wu, Y ., Tucker, G., and Nachum, O. Behavior regu- larized offline reinforcement learning. arXiv preprint arXiv:1911.11361,
arXiv 1911
-
[11]
Policy expansion for bridging offline-to-online reinforcement learning
Zhang, H., Xu, W., and Yu, H. Policy expansion for bridging offline-to-online reinforcement learning. arXiv preprint arXiv:2302.00935,
-
[12]
and SO2 (Zhang et al. 2024). BOORL employs a Bayesian approach to address the trade-off between optimism and pessimism in offline-to-online reinforcement learning, providing a comprehensive framework for tackling this fundamental challenge. Similarly, SO2 incorporates advanced techniques into Q-function training to mitigate Q-value estimation errors, offe...
work page 2024
-
[2017]
A., Krishna- murthy, A., and Sun, W
Song, Y ., Zhou, Y ., Sekhari, A., Bagnell, J. A., Krishna- murthy, A., and Sun, W. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint arXiv:2210.06718,
-
[2018]
Off-policy deep reinforcement learning without exploration
Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In Interna- tional conference on machine learning, pp. 2052–2062. PMLR,
-
[2020]
Proto: Iterative policy regularized offline-to-online reinforcement learning
Li, J., Hu, X., Xu, H., Liu, J., Zhan, X., and Zhang, Y .- Q. Proto: Iterative policy regularized offline-to-online reinforcement learning. arXiv preprint arXiv:2305.15669,
Show all 12 references
-
[2021]
Ba, J. L. Layer normalization. arXiv preprint arXiv:1607.06450,
-
[2022]
Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization
Lei, K., He, Z., Lu, C., Hu, K., Gao, Y ., and Xu, H. Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization. arXiv preprint arXiv:2311.03351,
-
[2023]
Bayesian design princi- ples for offline-to-online reinforcement learning
Hu, H., Yang, Y ., Ye, J., Wu, C., Mai, Z., Hu, Y ., Lv, T., Fan, C., Zhao, Q., and Zhang, C. Bayesian design princi- ples for offline-to-online reinforcement learning. arXiv preprint arXiv:2405.20984,
-
[2024]
D4rl: Datasets for deep data-driven reinforcement learning
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219,
2004 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.