REVIEW 4 major objections 5 minor 56 references
Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A general offline-to-online RL framework that repairs both the evaluation and improvement mismatches before fine-tuning, so any frozen offline policy can be lifted into SAC, TD3, or PPO with stable early gains and top D4RL scores.
desk verdict Solid O2O recipe with overclaimed generality; theory needs tightening, but the empirical package earns a real review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the behavior-regularized MDP formulation of offline RL, which frames every offline method as maximizing return minus a penalty $f(\pi(a|s)/\mu(a|s))$ on divergence from the behavior policy; the paper uses this identity to attribute both mismatches to one cause. Three named mechanisms carry the argument. Policy re-evaluation rests on Assumption 4.1 (single-policy concentrability, $\max_{s,a} \frac{d^{\pi_\theta}(s,a)}{d^\mu(s,a)} \le C$) and a fitted Q-evaluation error bound that justifies replacing the pessimistic offline critic with an optimistic reconstruction from the offline policy alone. Value alignment uses the energy-policy identity $Q(s,a) = V(s) + \alpha \log \pi(a|s)$ for stochastic policies, a Gaussian model of normalized Q-values around the deterministic action for TD3, and an entropy-shifted log-probability auxiliary advantage for PPO, all anchored to the offline policy's most likely action. Constrained fine-tuning casts the online objective as a constrained MDP and solves it with a Lagrange multiplier that anneals to zero, giving a convergence guarantee to the unconstrained optimum while stabilizing early updates against out-of-distribution states and actions.
What would settle it
The cleanest check is to compare the reconstructed critic against empirical returns: roll out the offline policy in the environment, compute Monte Carlo returns for dataset-covered states, and test whether the re-evaluated and aligned Q-values track them, since Corollary 4.2 promises the fitted-Q-evaluation error stays bounded under concentrability. A second, sharper test is to violate Assumption 4.1 directly, by training a policy on a narrow dataset and fine-tuning it in an environment whose state-action support shifts, and watching for the early fine-tuning collapse the method claims to prevent.
Extended reading notes
Core claim
The central claim is that a single algorithmic pipeline can absorb any offline-trained policy into SAC, TD3, or PPO and fine-tune it without the early performance collapse that plagues direct transfer. The paper treats offline RL through the behavior-regularized MDP view, in which every offline method adds a divergence penalty between the learned policy and the behavior policy; this extra coupling is what creates the two mismatches when the policy is handed to an online rule that couples only to Q-values. Its fix has three stages. Policy re-evaluation re-trains the critic from scratch against the frozen offline policy using an optimistic off-policy evaluation rule, justified by a single-policy concentrability assumption and a fitted-Q-evaluation error bound, so the reconstructed Q-values are optimistic rather than pessimistic or exploding. Value alignment then calibrates the critic so the offline policy's most likely action anchors the Q-scale and overestimated actions are pulled down, for SAC via the maximum-entropy identity $Q(s,a) = V(s) + \alpha \log \pi(a|s)$, for TD3 by modeling normalized Q-values around the deterministic action as a Gaussian, and for PPO by adding an auxiliary advantage built from the offline policy's log-probabilities. Constrained fine-tuning solves a constrained MDP with a Lagrange multiplier that keeps the policy inside a trust region around the best policy observed online and provably decays to zero. Empirically, the paper reports total MuJoCo normalized scores of 1230.41 for O2SAC, 1171.66 for O2TD3, and 1126.48 for O2PPO, each above the six compared baselines, and reports O2PPO reaching 184.3 total on the two AntMaze umaze tasks, the best among the methods shown.
Load-bearing premise
The load-bearing premise is that the offline policy is genuinely reliable and that the dataset covers the actions it takes. If the pre-trained policy is weak or ventures where the dataset has no support, the optimistic re-evaluation inherits extrapolation error, the alignment anchors to bad actions, and constrained fine-tuning simply pins the policy to a poor reference.
Editorial extensions
If this is right
- Any offline RL method, such as CQL, IQL, TD3+BC, or even a decision-transformer-style policy class, can be connected to SAC, TD3, or PPO without designing a fine-tuning scheme per offline method.
- The early online phase no longer exhibits the pessimism-to-optimism Q-value jump: in the reported D4RL runs, re-evaluation plus alignment keep fine-tuning stable from the first evaluation epoch.
- The reported MuJoCo totals place the new instantiations ahead of the comparison set: O2SAC at 1230.41, O2TD3 at 1171.66, and O2PPO at 1126.48, versus 1204.5 for the strongest baseline Off2On.
- On-policy fine-tuning becomes usable in the O2O setting: O2PPO, initialized from IQL, reaches 184.3 total on the two AntMaze umaze tasks, ahead of Cal-QL's 168.2.
- Because the constraint acts as a soft reward-shaping term, the framework composes with sample-efficient online techniques such as higher update-to-data ratios, which the paper shows accelerates learning further.
Reading between the lines
- The value-alignment stage trusts that the offline policy's most likely action is locally optimal; a natural stress test, not run in the paper, is to feed the pipeline a deliberately corrupted offline policy and measure whether the anchored critic drags fine-tuning down.
- Because alignment uses only the offline policy's action distribution, the reconstruction phase is effectively a critic transplant that could in principle attach to online algorithms beyond SAC, TD3, and PPO, whenever the offline policy comes in stochastic or deterministic form.
- The paper itself notes in Appendix A that O2PPO can improve more slowly than direct IQL-to-PPO fine-tuning on some tasks, such as walker2d-medium-replay-v2, because the auxiliary advantage constrains updates even when the critic is already accurate; that is the price of guaranteed stability.
- The convergence corollary, which says the constraint multiplier decays to zero, suggests the method buys early stability with temporary conservatism; a testable implication is that the fixed-interval reference update, used when online evaluation is forbidden, should degrade on narrow datasets, a risk the paper observes on hopper-medium-v2.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an offline-to-online (O2O) reinforcement learning framework (OCR-CFT) that combines three components: policy re-evaluation (optimistically re-fitting the critic with an off-policy evaluation method), value alignment (calibrating the critic so that action probabilities and Q-values are consistent with a reliable offline policy), and constrained fine-tuning (adding a trust-region-style constraint during online updates). The framework is instantiated for SAC, TD3, and PPO (O2SAC, O2TD3, O2PPO), and evaluated on D4RL MuJoCo locomotion and AntMaze tasks, reporting stable performance improvements over several baselines. The paper also includes a theoretical analysis (an FQE error bound, properties of value alignment, and a convergence claim for the constrained fine-tuning procedure), an ablation study, and a discussion of transferability to heterogeneous offline policies such as decision transformers.
Significance. If the empirical results hold, the paper makes a useful contribution to O2O RL: it demonstrates a combination of optimistic re-evaluation, critic-policy alignment, and constrained fine-tuning that stabilizes online fine-tuning from several offline methods, and it releases code and ablates each component. The paper also extends O2O methods to on-policy fine-tuning via PPO, which is comparatively underexplored. However, the central claim of 'any offline method' is broader than what the experiments and the method itself support: the method presumes a reliable offline policy, and the experimental section excludes settings where the offline policy is known to be poor (e.g., TD3+BC on AntMaze medium/large, as acknowledged in Appendix C.2). The theoretical results are not fully supported: the convergence guarantee of Corollary 4.5 is circular, and the Gaussian Q-model for O2TD3 is introduced with an unsupported derivation. These issues limit the generality claims made in the abstract and introduction.
major comments (4)
- [§4.3, Corollary 4.5, Appendix F] The proof of Corollary 4.5 is circular: it assumes that 'πref = π* when the algorithm converges' and then uses this to conclude that the constraint is inactive and λ* = 0, but the corollary is precisely the claim that the algorithm converges to the optimal policy of the unconstrained MDP. The proof also invokes Theorem 2 of RCPO without verifying its assumptions (e.g., regularity of the constraint function, convergence of the Lagrange multipliers), and the phrase 'appropriate learning rates' is not made precise. Consequently, the statement that Eq. (20) converges almost surely to a fixed point (θ*, μ*, λ*) with λ* = 0 is not established.
- [§4.2, Eq. (15), Appendix G.2] The Gaussian calibration for O2TD3 rests on an unsupported step. In Appendix G.2, the paper writes 'Q(s,ȧ) = Q(s,ȧ)/√(2πΣ), which means Σ = 1/(2π)', which is not mathematically meaningful as written (the peak value of a Gaussian is not equal to the total probability mass). The assumption that normalized Q-values around the policy action follow a Gaussian distribution is introduced without empirical validation, and the derivation from Eqs. (40)-(43) is not a rigorous argument. Since Eq. (15) is the core mechanism for aligning the TD3 critic, the O2TD3 contribution is left on a heuristic footing rather than a theoretically justified one.
- [§1 and §4.2, Appendix C.2] The claim of 'general O2O learning from any offline method' is not supported by the paper's own assumptions and experiments. Section 4.2 states 'Given that the well-trained offline policy is reliable' and anchors value alignment to the modal actions of the offline policy, and Appendix C.2 explicitly excludes TD3+BC on AntMaze medium/large because that policy is poor and fine-tuning from it 'helps little'. Thus the framework is general only among offline policies that are already reliable; the abstract and introduction should state this qualification instead of implying unrestricted applicability to any offline method.
- [§4.2, Proposition 4.3] Proposition 4.3 is essentially a restatement of the min-operator construction: because Eq. (12) replaces overestimated Q-values with a clipped anchor, the resulting state value is sandwiched between the FQE value and the anchor value by definition. The proposition does not establish that the aligned critic is 'optimistic' in any meaningful sense or that it yields reliable policy improvement; it is a consistency check of the design rather than a substantive theoretical guarantee.
minor comments (5)
- [Abstract] The abstract contains a duplicated word: 'an optimistic and and aligned critic' should be 'an optimistic and aligned critic'.
- [Appendix A] The phrase 'as showned in Table 2' should be 'as shown in Table 2'.
- [§5.1 and Table 2] The paper claims 'optimal performance in most cases', but in Table 2 Off2On achieves a higher total MuJoCo score (1204.5) than O2TD3 (1171.66), and O2SAC (1230.41) is only slightly above it; the claim should be more precise about which methods are compared and under what interaction budgets.
- [§5 and Appendix C.2] The number of interaction steps for O2PPO is inconsistent across the paper: Section 5 states 250,000 steps, while Appendix C.2 says 'here we run O2PPO with 250,000 environments steps ... while in Section 5.1, we run O2PPO with 200,000 environments steps'. This should be reconciled, as it affects the fairness of comparisons.
- [Appendix F, proof of Proposition 4.4] In Eq. (35), the factor α in the auxiliary advantage appears to be dropped in the final expression, and the constant C is not defined; the derivation should be corrected for clarity.
Circularity Check
One load-bearing theoretical step is circular: Corollary 4.5's proof assumes πref = π* to conclude convergence to π*, while πref is only the best historical policy. The empirical D4RL comparisons and the FQE bound remain independent external content.
-
other
[Appendix F, Proof of Corollary 4.5]
"As the constraint is Eπ[f (π(at|st), πref(at|st))] < αand πref is the best one among old policies during online evaluations, πref = π⋆ when the algorithm converges, so the constraint term tends to −α. According to the update function Eq. (20) of λ, λ tends to be reduced, which is in contradiction to the Assumption. Therefore, λ⋆ = 0 when the algorithm converges."
Corollary 4.5 promises that Eq. (20) converges to the unconstrained MDP optimum (θ*, Q*) with λ*=0. The proof's only argument for λ*=0 is that πref, the best historical policy, equals π* at convergence. But π* is precisely the unconstrained optimal policy whose attainment is to be proved; the RCPO theorem cited from [41] guarantees only a fixed point of the constrained problem. Assuming the algorithm has already reached π* to prove it reaches π* is begging the question, so this theoretical guarantee is circular rather than an independent derivation.
full rationale
The circularity is confined to the proof of the constrained fine-tuning guarantee. The policy re-evaluation bound (Corollary 4.2) is imported from external FQE bounds [22, 29] and is not circular. Value alignment (Eq. 12) is explicitly a calibration that anchors Q-values to the offline policy's log-probabilities; Proposition 4.3 restates the min-operator design, but since the paper presents value alignment as a design choice rather than as an empirical prediction, we do not count that as a major circular step. The core empirical claim is tested against external D4RL baselines (AWAC, IQL, PEX, Off2On, Cal-QL, ACA, PROTO), so those results have independent content and do not reduce to a fit. There is no load-bearing self-citation chain: the cited external theorems are from other groups, not from the present authors. The paper's own Appendix C.2 narrows the 'any offline method' claim by excluding poor TD3+BC initializations on AntMaze; that is an admitted scope limitation, not a definitional circle. Overall, the substantial circular step is Corollary 4.5, which warrants a moderate score.
Assumptions & free parameters
free parameters (5)
- SAC temperature α =
0.2 (medium-quality MuJoCo), 0.5 (expert-quality and AntMaze varying)
- TD3 alignment constant k =
1
- Constraint threshold τ schedules =
O2SAC: 0.125→2.0 (medium/replay), 0.005→0.125 (medium-expert/expert); O2TD3: 0.0025→0.01 and 0.000025→0.000625
- O2PPO β decay schedule =
1→0 over 250k steps (medium/replay/AntMaze); 1→0 over 500k steps (medium-expert/expert)
- Lagrange update weight ω =
ω = |0.7 − I(f > τ)|
assumptions (5)
- domain assumption Single-policy concentrability: max_{s,a} d^{πθ}(s,a)/d^{µ}(s,a) ≤ C (Assumption 4.1).
- domain assumption The well-trained offline policy is reliable and its high-probability actions are good actions.
- ad hoc to paper Normalized Q-values around the TD3 policy action follow a Gaussian, Q(s,a)/Q(s,ȧ) ∼ N(ȧ,Σ), with Σ fixed to 1/(2π).
- standard math FQE error bound (Theorem 4.2 in Le et al. 2019 / Theorem 4.9 in Mao et al. 2023).
- standard math RCPO convergence (Theorem 2 in Tessler et al. 2018).
Cite this review
Pith. "Pith review of Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL." pith.science (2026). https://pith.science/paper/7OJY7CFB
@misc{pith2026241218855,
author = {Pith},
title = {Pith review of: Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL},
year = {2026},
howpublished = {\url{https://pith.science/paper/7OJY7CFB}},
note = {Machine review of arXiv:2412.18855}
}
read the original abstract
Offline-to-online (O2O) reinforcement learning (RL) provides an effective means of leveraging an offline pre-trained policy as initialization to improve performance rapidly with limited online interactions. Recent studies often design fine-tuning strategies for a specific offline RL method and cannot perform general O2O learning from any offline method. To deal with this problem, we disclose that there are evaluation and improvement mismatches between the offline dataset and the online environment, which hinders the direct application of pre-trained policies to online fine-tuning. In this paper, we propose to handle these two mismatches simultaneously, which aims to achieve general O2O learning from any offline method to any online method. Before online fine-tuning, we re-evaluate the pessimistic critic trained on the offline dataset in an optimistic way and then calibrate the misaligned critic with the reliable offline actor to avoid erroneous update. After obtaining an optimistic and and aligned critic, we perform constrained fine-tuning to combat distribution shift during online learning. We show empirically that the proposed method can achieve stable and efficient performance improvement on multiple simulated tasks when compared to the state-of-the-art methods.
Figures
Figures from the paper (11 more)
Reference graph
Works this paper leans on
-
[1]
Constrained policy optimization
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22–31. PMLR, 2017
2017
-
[2]
Reinforcement learning: Theory and algorithms
Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32:96, 2019
work page 2019
-
[3]
Reincarnating reinforcement learning: Reusing prior computation to accelerate progress
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Belle- mare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. Advances in Neural Information Processing Systems, 35:28955–28971, 2022
work page 2022
-
[4]
Efficient online reinforcement learning with offline data
Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. arXiv preprint arXiv:2302.02948, 2023
arXiv 2023
-
[5]
Improving td3-bc: Relaxed policy constraint for offline learning and stable online fine-tuning
Alex Beeson and Giovanni Montana. Improving td3-bc: Relaxed policy constraint for offline learning and stable online fine-tuning. arXiv preprint arXiv:2211.11802, 2022
arXiv 2022
-
[6]
Decision transformer: Reinforcement learning via sequence modeling
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021
2021
-
[7]
Safe exploration in continuous action spaces
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757, 2018
arXiv 2018
-
[8]
Uncertainty-aware model-based offline reinforcement learning for automated driving
Christopher Diehl, Timo Sebastian Sievernich, Martin Krüger, Frank Hoffmann, and Torsten Bertram. Uncertainty-aware model-based offline reinforcement learning for automated driving. IEEE Robotics and Automation Letters, 8(2):1167–1174, 2023. 10
work page 2023
Show all 56 references
-
[9]
D4rl: Datasets for deep data-driven reinforcement learning
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020
2004 arXiv
-
[10]
A minimalist approach to offline reinforcement learning
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132–20145, 2021
2021
-
[11]
Addressing function approximation error in actor-critic methods
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587–1596. PMLR, 2018
2018
-
[12]
Off-policy deep reinforcement learning without exploration
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052–2062. PMLR, 2019
2019
-
[13]
Extreme q-learning: Maxent rl without entropy
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.02328, 2023
2023 arXiv
-
[14]
Open and real-world human-ai coordination by heterogeneous training with communication
Cong GUAN, Ke XUE, Chunpeng FAN, Feng CHEN, Lichao ZHANG, Lei YUAN, Chao QIAN, and Yang YU. Open and real-world human-ai coordination by heterogeneous training with communication. Frontiers of Computer Science, 19(4):194314, 2025
2025
-
[15]
A simple unified uncertainty-guided framework for offline-to-online reinforcement learning
Siyuan Guo, Yanchao Sun, Jifeng Hu, Sili Huang, Hechang Chen, Haiyin Piao, Lichao Sun, and Yi Chang. A simple unified uncertainty-guided framework for offline-to-online reinforcement learning. arXiv preprint arXiv:2306.07541, 2023
2023
-
[16]
Reinforcement learning with deep energy-based policies
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning , pages 1352–1361. PMLR, 2017
2017
-
[17]
Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. PMLR, 2018
2018
-
[18]
Idql: Implicit q-learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573, 2023
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573, 2023
2023 arXiv
-
[19]
Uncertainty-driven pessimistic q-ensemble for offline-to- online reinforcement learning
Ingook Jang and Seonghyun Kim. Uncertainty-driven pessimistic q-ensemble for offline-to- online reinforcement learning. In 3rd Offline RL Workshop: Offline RL as a”Launchpad” , 2022
2022
-
[20]
Offline reinforcement learning with implicit q-learning
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2021
2021
-
[21]
Conservative q-learning for offline reinforcement learning
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179– 1191, 2020
2020
-
[22]
Batch policy learning under constraints
Hoang Le, Cameron V oloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning, pages 3703–3712. PMLR, 2019
2019
-
[23]
Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 1702–1712. PMLR, 2022
2022
-
[24]
Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization
Kun Lei, Zhengmao He, Chenhao Lu, Kaizhe Hu, Yang Gao, and Huazhe Xu. Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization. arXiv preprint arXiv:2311.03351, 2023
2023 arXiv
-
[25]
Proto: Iterative policy regularized offline-to-online reinforcement learning
Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, and Ya-Qin Zhang. Proto: Iterative policy regularized offline-to-online reinforcement learning. arXiv preprint arXiv:2305.15669, 2023. 11
2023 arXiv
-
[26]
Finetuning from offline reinforcement learning: Challenges, trade-offs and practical solutions
Yicheng Luo, Jackie Kay, Edward Grefenstette, and Marc Peter Deisenroth. Finetuning from offline reinforcement learning: Challenges, trade-offs and practical solutions. arXiv preprint arXiv:2303.17396, 2023
2023 arXiv
-
[27]
Mildly conservative q-learning for offline reinforcement learning
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711–1724, 2022
2022
-
[28]
Moore: Model-based offline-to- online reinforcement learning
Yihuan Mao, Chao Wang, Bin Wang, and Chongjie Zhang. Moore: Model-based offline-to- online reinforcement learning. arXiv preprint arXiv:2201.10070, 2022
2022 arXiv
-
[29]
Supported trust region optimization for offline reinforcement learning
Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported trust region optimization for offline reinforcement learning. In International Conference on Machine Learning, pages 23829–23851. PMLR, 2023
2023
-
[30]
Fine-tuning offline policies with optimistic action selection
Max Sobol Mark, Ali Ghadirzadeh, Xi Chen, and Chelsea Finn. Fine-tuning offline policies with optimistic action selection. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022
2022
-
[31]
Planning to go out-of-distribution in offline-to-online reinforcement learning
Trevor McInroe, Stefano V Albrecht, and Amos Storkey. Planning to go out-of-distribution in offline-to-online reinforcement learning. arXiv preprint arXiv:2310.05723, 2023
2023 arXiv
-
[32]
Awac: Accelerating online reinforcement learning with offline datasets
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020
2006 arXiv
-
[33]
Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. arXiv preprint arXiv:2303.05479, 2023
2023 arXiv
-
[34]
Bridging offline reinforcement learning and imitation learning: A tale of pessimism
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34:11702–11716, 2021
2021
-
[35]
Trust region policy optimization
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889–1897. PMLR, 2015
2015
-
[36]
High- dimensional continuous control using generalized advantage estimation
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015
2015 arXiv
-
[37]
Proximal policy optimization algorithms
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017
2017 arXiv
-
[38]
Sutton and AndrewG
RichardS. Sutton and AndrewG. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, page 285–286, Jan 2005
2005
-
[39]
Lever- aging factored action spaces for efficient offline reinforcement learning in healthcare
Shengpu Tang, Maggie Makar, Michael Sjoding, Finale Doshi-Velez, and Jenna Wiens. Lever- aging factored action spaces for efficient offline reinforcement learning in healthcare. Advances in Neural Information Processing Systems, 35:34272–34286, 2022
2022
-
[40]
CORL: Research-oriented deep offline reinforcement learning library
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offline RL Workshop: Offline RL as a ”Launchpad”, 2022
2022
-
[41]
Reward constrained policy optimization
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074, 2018
2018 arXiv
-
[42]
Jump-start reinforcement learning
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. In International Conference on Machine Learning, pages 34556–34583. PMLR, 2023. 12
2023
-
[43]
Train once, get a family: State-adaptive balances for offline-to- online reinforcement learning
Shenzhi Wang, Qisen Yang, Jiawei Gao, Matthieu Gaetan Lin, HAO CHEN, Liwei Wu, Ning Jia, Shiji Song, and Gao Huang. Train once, get a family: State-adaptive balances for offline-to- online reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Sy...
2023
-
[44]
Supported policy optimization for offline reinforcement learning
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:31278–31291, 2022
2022
-
[45]
Policy finetuning: Bridging sample-efficient offline and online reinforcement learning
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34:27395–27407, 2021
2021
-
[46]
A policy-guided imitation approach for offline reinforcement learning
Haoran Xu, Li Jiang, Li Jianxiong, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:4085– 4098, 2022
2022
-
[47]
Offline rl with no ood actions: In-sample learning via implicit value regularization
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303.15810, 2023
2023 arXiv
-
[48]
Actor-critic alignment for offline-to-online reinforcement learning
Zishun Yu and Xinhua Zhang. Actor-critic alignment for offline-to-online reinforcement learning. In International Conference on Machine Learning, pages 40452–40474. PMLR, 2023
2023
-
[49]
Understanding, predicting and better resolving q-value divergence in offline-rl
Yang Yue, Rui Lu, Bingyi Kang, Shiji Song, and Gao Huang. Understanding, predicting and better resolving q-value divergence in offline-rl. arXiv preprint arXiv:2310.04411, 2023
2023 arXiv
-
[50]
Policy expansion for bridging offline-to-online reinforcement learning
Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforcement learning. arXiv preprint arXiv:2302.00935, 2023
2023 arXiv
-
[51]
Ensemble-based offline-to-online reinforcement learning: From pessimistic learning to optimistic exploration
Kai Zhao, Yi Ma, Jinyi Liu, Yan Zheng, and Zhaopeng Meng. Ensemble-based offline-to-online reinforcement learning: From pessimistic learning to optimistic exploration. arXiv preprint arXiv:2306.06871, 2023
2023 arXiv
-
[52]
Adaptive behav- ior cloning regularization for stable offline-to-online reinforcement learning
Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behav- ior cloning regularization for stable offline-to-online reinforcement learning. arXiv preprint arXiv:2210.13846, 2022
2022 arXiv
-
[53]
Online decision transformer
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In international conference on machine learning, pages 27042–27059. PMLR, 2022. 13 A Detailed Data Table 2: Average normalized D4RL scores of O2O methods shown in Figure 2. Outside parenthesis: scores at...
2022
-
[54]
However, our O2SAC still outperforms it with less computational cost during online fine-tuning and less requirements for offline policy
and the implementations in other papers [48] and [50]. However, our O2SAC still outperforms it with less computational cost during online fine-tuning and less requirements for offline policy. In 14 addition, our methods bridge different offline algorithms and three SOTA online...
-
[55]
For model-based O2O RL, [31] explores regions with high uncertainty and returns in learned model
utilize ensemble Q-learning to alleviate distribution shift, and implement optimistic exploration by some approaches about ensemble in online RL. For model-based O2O RL, [31] explores regions with high uncertainty and returns in learned model. [42] and [50] concatenate differe...
-
[56]
Moreover, [4] find that LayerNorm is favourable for efficient online RL with offline data
indicate that in offline RL, LayerNorm is a good solution to effectively avoid divergence without introducing detrimental bias, leading to superior performance. Moreover, [4] find that LayerNorm is favourable for efficient online RL with offline data. Therefore, for stable eva...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.