Pith. sign in

REVIEW 4 major objections 6 minor 11 cited by

Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that online RL fine-tuning can discard the offline pre-training dataset entirely, provided the replay buffer is first seeded with a few thousand rollouts from the frozen pre-trained policy.

desk verdict A solid, well-validated empirical paper that makes a real case for no-retention fine-tuning with a short warmup, but the universal claim is overstated and the method is conditional on decent pre-training and a tuned K. read the letter →

arxiv 2412.07762 v3 pith:XFRLHEB5 submitted 2024-12-10 cs.LG

Zhiyuan Zhou , Andy Peng , Qiyang Li , Sergey Levine , Aviral Kumar This is my paper · ORCID
classification cs.LG
keywords offline-to-onlineRLno-retentionfine-tuningwarm-startQ-valuedivergencecatastrophicforgettinghighupdate-to-dataratiosample-efficientreinforcementlearningreal-robot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement-learning systems are usually pre-trained on static offline data and then fine-tuned online, but practical recipes keep the offline data in the replay buffer throughout fine-tuning. This paper argues that retaining offline data is unnecessary: it is only needed at the very start of fine-tuning to stop the value function from diverging when online rollouts shift the state-action distribution. The paper's method, Warm-start RL (WSRL), replaces that data with a brief warm-up phase in which the frozen pre-trained policy collects about 5,000 online transitions before standard online RL begins. With this warm-up, the offline Q-function recalibrates to the online distribution and the pessimistic 'downward spiral' of value targets is avoided. The authors report that WSRL fine-tunes faster and reaches higher asymptotic performance than methods that keep the offline data, on simulated benchmarks and a real robot peg-insertion task.

What carries the argument

The load-bearing mechanism is the warm-up phase: K=5,000 transitions collected by the frozen pre-trained policy at the start of fine-tuning, seeded into the replay buffer. These transitions stand in for the offline dataset, but are drawn from the distribution the online policy actually visits, so they bridge the distribution shift and give the Q-function a set of state-action pairs whose targets are not pessimistically suppressed. The second component is the choice of a standard, high-update-to-data online RL algorithm after warm-up, so that pessimism or behavioral constraints do not slow asymptotic learning. The paper also relies on a value-function ensemble and layer normalization to stabilize the high-update-to-data regime.

What would settle it

Take one of the paper's own tasks, discard the offline dataset after pre-training, and run WSRL while monitoring the TD error on the discarded data as in the paper's Figure 3; if that error grows large and the policy's success rate collapses without recovering, the warm-up claim is wrong for that case. More sharply, find any environment with healthy pre-training where 5,000 warm-up steps are insufficient to keep Q-values from diverging, and the fixed-warmup recipe is falsified.

Watch

Extended reading notes

Core claim

The central claim is that the sole useful role of offline data during fine-tuning is to prevent a sudden divergence of the Q-function at the onset of online training. The paper identifies a mechanism: when online rollouts are used without offline data, Bellman backups query the pre-trained Q-function on out-of-distribution state-action pairs; because pessimistic pre-training suppressed those values, the targets are too low, propagating underestimation and eventually destroying the pre-trained initialization. WSRL avoids this by collecting K=5,000 warm-up rollouts from the frozen pre-trained policy, which places near-on-policy data in the replay buffer and 'recalibrates' the Q-function before the online algorithm updates. After warm-up, WSRL runs a standard online actor-critic (Soft Actor-Critic with a high update-to-data ratio, an ensemble of Q-functions, and layer normalization) with no pessimism or behavioral constraints. The paper's experiments indicate that this both prevents forgetting and speeds up learning, outperforming methods that retain offline data.

Load-bearing premise

The method rests on the empirical assumption that 5,000 rollouts collected by the pre-trained policy are enough to stabilize the learned value function before ordinary online training begins; if the pre-training itself is poor, those rollouts do not carry useful information and the method fails.

Editorial extensions

If this is right

  • Offline datasets can be discarded after pre-training, so large-scale RL fine-tuning no longer needs to store or batch through historical data.
  • Doing so is not merely tolerated: WSRL reports faster fine-tuning and higher final performance than algorithms that keep offline data across the benchmark tasks.
  • The value function recovers from an initial performance dip without losing the pre-trained initialization, as measured by KL divergence to the pre-trained policy and Q-function.
  • The recipe works with different offline pre-training algorithms (CQL, IQL, and CalQL), so WSRL is agnostic to the choice of offline initialization.
  • After warm-up, fine-tuning with a standard online RL algorithm is substantially better than continuing with a pessimistic offline RL algorithm, so the online algorithm choice is a deliberate part of the method.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: This suggests a broader design rule for offline-to-online transitions: a short on-policy calibration phase, not necessarily fixed at 5,000 steps, may be the missing ingredient whenever the pre-trained value function was learned under pessimism or behavioral constraints.
  • Inference: The method inherits the quality of pre-training, as Appendix O shows when pre-training fails on a hard task, warm-up cannot create a useful signal; a natural extension would be to detect low-confidence pre-training and fall back to pure online RL.
  • Inference: The same recalibration idea might apply to continual RL or domain adaptation, where a policy is fine-tuned to a new task distribution without access to past experience, and warm-up rollouts from the old policy could play the role of the offline data.
  • Inference: Because the paper's result is empirical, a theoretical account of how the required warm-up length scales with distribution shift would take the claim beyond the fixed K=5,000 recipe.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies online RL fine-tuning of an offline pre-trained policy and Q-function without retaining the offline dataset. It first analyzes why existing offline-to-online methods fail in this "no-retention" setting, attributing the failure to a Q-value underestimation spiral caused by distribution shift and pessimistic TD targets at the onset of fine-tuning. It then proposes WSRL, which initializes the online replay buffer with K=5,000 rollouts from the frozen pre-trained policy before switching to standard high-UTD SAC fine-tuning. The paper reports that WSRL outperforms prior methods on D4RL Antmaze, Kitchen, Adroit, and MuJoCo tasks, and shows a real-robot peg insertion result. Appendices provide ablations on warmup length, warmup data source, policy/Q initialization, offline algorithm choice, UTD ratio, and robustness to poor pre-training.

Significance. If the central claim holds, the paper makes a useful conceptual and practical contribution: it identifies Q-value recalibration as the key failure mode in no-retention fine-tuning and shows that a simple warmup procedure can avoid it without retaining offline data. The paper is strong on empirical breadth and transparency: code is released, ablations are extensive, the KL-divergence analysis attempts to separate unlearning from forgetting, and the real-robot result gives out-of-distribution credibility. The main weakness is that the universal claim in the abstract is stronger than the evidence: the method depends on a fixed and domain-sensitive warmup length K and on the pre-trained policy/Q-function being reasonably informative, and the main figures do not report error bars or seed counts. These issues make the paper suitable for major revision rather than acceptance in its current form.

major comments (4)
  1. [Abstract, Section 5.2, Appendix D (Figure 19)] The claim that WSRL works "irrespective of whether [previous methods] retain offline data or not" is not supported by the paper's own ablations. Appendix D Figure 19 shows that K=1,000 can be unstable on Door-binary and K=20,000 hurts Kitchen-mixed, so K=5,000 is a selected middle value rather than a robust default. Since Algorithm 1 takes K as a fixed input and no criterion is provided for choosing K on a new domain, the universal version of the central claim should be revised to a conditional statement about settings where the warmup length is adequate.
  2. [Algorithm 1, Section 4, Section 3.1] The mechanism of "recalibration" is underspecified. In Algorithm 1, no gradient updates occur during the warmup phase; the Q-function is frozen for the first K steps, so any recalibration is actually carried out by the first SAC updates after step K. The paper does not provide an argument, formal or diagnostic, that K transitions from the frozen policy will prevent the pessimistic TD-target spiral described in Section 3.1 (Figures 4 and 5). Figure 11 supports the claim on one environment, but a criterion for detecting or guaranteeing successful recalibration would be needed to substantiate the mechanism.
  3. [Appendix O, Figure 30, Section 5.2] Appendix O shows that WSRL completely fails on Adroit-binary-v0 and performs poorly on Kitchen-complete-v0 when CalQL pre-training is unsuccessful. The paper acknowledges this, but the main text and abstract do not carry the resulting scope restriction. As written, the claim that WSRL can fine-tune without retaining offline data is conditional on the pre-trained policy and Q-function being informative; this boundary condition should be stated explicitly in the main text.
  4. [Section 5, Figures 7, 9, 10, 12, 13] No error bars, standard deviations, or seed counts are reported in the main experimental figures. Fine-tuning comparisons in this setting are known to be high-variance, and the paper's central claims of "faster" and "higher asymptotic performance" rest on curve comparisons. The paper should report means and standard errors over at least 3-5 seeds, or explicitly state if the curves are single runs, and justify that the reported differences are robust.
minor comments (6)
  1. [Appendix O] There is a typo: "Recolate-binary-v0" should be "Relocate-binary-v0".
  2. [Appendix C and Appendix G] There are typos: "perfrom" should be "perform" and "paris" should be "pairs".
  3. [Figure 2] The label "Doffline" appears to be a rendering artifact of "Doffline" or "Offline"; please fix.
  4. [References] References [3] and [4] are duplicate entries for the same paper (Ball et al., arXiv:2302.02948); they should be merged.
  5. [Appendix I (Kitchen episode length)] The note that some ablation experiments use a Kitchen version with maximum episode length 1000 while main comparisons use 280 is important for reproducibility; please state this in the main text or in a clearly visible setup paragraph so readers do not misinterpret cross-figure differences.
  6. [Table 1 (real robot)] The real-robot comparison reports success counts over 20 initial poses but no indication of variance across repeats or training runs; a sentence on the number of runs and variability would strengthen the claim.

Circularity Check

0 steps flagged · score 1.0 of 10

No substantive circularity: WSRL's warmup claim is empirical and benchmarked externally; self-citations are not load-bearing.

full rationale

The paper does not derive its central result from its own fitted inputs. WSRL is fully specified by Algorithm 1: a K-step warmup with the frozen pre-trained policy, followed by standard high-UTD SAC fine-tuning with no offline data. The 'recalibration' mechanism is presented as a hypothesis supported by empirical diagnostics (Q-value divergence, TD-error, KL divergence in Figures 3-5, 8, 11), not as a parameter fitted to the benchmark outcomes and then renamed a prediction. The choice K=5000 is a hyperparameter, and Appendix D explicitly reports sensitivity to it (1k can be unstable, 20k can hurt), which qualifies the claim but does not make it circular. Appendix O likewise reports failure when pre-training itself is poor, again a scope condition rather than a self-referential reduction. Citations to CalQL, RLPD, and related work include overlapping authors, but they function as baselines, initializers, and implementation sources with external code and benchmarks, not as self-citation chains that force the paper's conclusion. No uniqueness theorem is invoked to prohibit alternative methods, and no equation in the paper is equal to its inputs by construction. The paper is self-contained against external D4RL and real-robot benchmarks, so any concern about generality belongs to correctness risk, not circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests primarily on empirical evidence rather than mathematical derivation. The main free parameter is the warmup length K=5,000, chosen from ablations. The key domain assumptions are that the pre-trained initialization is useful and that warmup rollouts from the pre-trained policy are sufficient to recalibrate the Q-function. No new entities are introduced.

free parameters (1)
  • Warmup length K = 5000
    Chosen based on ablations in Appendix D (Figure 19); the paper states 'we did not tune the lengths of the warmup phase beyond what is shown in Figure 19, and we use 5000 warmup steps for all environments.' The performance of WSRL depends on this value.
assumptions (4)
  • domain assumption The pre-trained policy and Q-function from offline RL are useful initializations for the online task.
    WSRL inherits the pre-trained initialization; Appendix O shows that when pre-training fails (e.g., Adroit-binary-v0), WSRL fails, so the method's success depends on this assumption.
  • domain assumption K=5,000 warm-up rollouts from the frozen pre-trained policy are sufficient to recalibrate the Q-function to the online distribution and prevent the pessimistic 'downward spiral'.
    This is the core empirical premise of WSRL; Appendix D ablates warmup length but does not provide a theoretical guarantee, and Figure 19 shows that too-short warmups can destabilize fine-tuning.
  • ad hoc to paper The 'downward spiral' mechanism (pessimistic OOD TD-targets propagating underestimation) is the true cause of catastrophic forgetting in no-retention fine-tuning.
    The paper supports this with correlations (Figures 3-5) but does not test it with intervention; if the true cause is different, the warm-up design may not be necessary or sufficient.
  • domain assumption High-UTD ensemble SAC with layer normalization remains stable and sample-efficient when initialized from offline RL parameters (borrowed from RLPD [3]).
    WSRL relies on the stability of this online RL recipe; Section 5.5 shows that using a pessimistic algorithm (CalQL) instead of SAC is significantly worse.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data." pith.science (2026). https://pith.science/paper/XFRLHEB5

@misc{pith2026241207762,
  author       = {Pith},
  title        = {Pith review of: Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XFRLHEB5}},
  note         = {Machine review of arXiv:2412.07762}
}
read the original abstract

The modern paradigm in machine learning involves pre-training on diverse data, followed by task-specific fine-tuning. In reinforcement learning (RL), this translates to learning via offline RL on a diverse historical dataset, followed by rapid online RL fine-tuning using interaction data. Most RL fine-tuning methods require continued training on offline data for stability and performance. However, this is undesirable because training on diverse offline data is slow and expensive for large datasets, and in principle, also limit the performance improvement possible because of constraints or pessimism on offline data. In this paper, we show that retaining offline data is unnecessary as long as we use a properly-designed online RL approach for fine-tuning offline RL initializations. To build this approach, we start by analyzing the role of retaining offline data in online fine-tuning. We find that continued training on offline data is mostly useful for preventing a sudden divergence in the value function at the onset of fine-tuning, caused by a distribution mismatch between the offline data and online rollouts. This divergence typically results in unlearning and forgetting the benefits of offline pre-training. Our approach, Warm-start RL (WSRL), mitigates the catastrophic forgetting of pre-trained initializations using a very simple idea. WSRL employs a warmup phase that seeds the online RL run with a very small number of rollouts from the pre-trained policy to do fast online RL. The data collected during warmup helps ``recalibrate'' the offline Q-function to the online distribution, allowing us to completely discard offline data without destabilizing the online RL fine-tuning. We show that WSRL is able to fine-tune without retaining any offline data, and is able to learn faster and attains higher performance than existing algorithms irrespective of whether they retain offline data or not.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SpikeATac: A Multimodal Tactile Finger with Taxelized Dynamic Sensing for Dexterous Manipulation

    cs.RO 2025-10 conditional novelty 7.0 of 10

    A fingertip with 16-taxel PVDF dynamic sensing plus capacitive static sensing enables fast delicate grasping and, with RLHF fine-tuning, in-hand manipulation of fragile objects.

  2. Adaptation of Generalist Robot Policies with Minimal Data

    cs.RO 2026-08 conditional novelty 6.0 of 10

    MiDAS, a two-stage recipe of one-demo behavior cloning plus residual online RL on a frozen VLA backbone, reaches high success from a single demonstration in simulation and improves real-world bimanual manipulation.

  3. Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?

    cs.LG 2026-07 conditional novelty 6.0 of 10

    On six robot-manipulation tasks, offline Q-pretraining does not accelerate online RL fine-tuning from a pretrained policy, while seeding the replay buffer with rollouts from an ensemble of policies (IPE) improves fina...

  4. OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policies

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A policy-agnostic two-stage real-world RL method learns tactile residual corrections on frozen visual policies, lifting contact-rich task success from 5–40% to 85–100% in under 80 minutes.

  5. The Three Regimes of Offline-to-Online Reinforcement Learning

    cs.LG 2025-10 conditional novelty 6.0 of 10

    Offline-to-online fine-tuning works best when the chosen method preserves the stronger of the pretrained policy or the offline dataset; a three-regime taxonomy organizes these choices.

  6. Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A single multi-task RL agent using a large regularized critic, categorical value loss, and task embeddings achieves state-of-the-art results across 283 tasks and transfers efficiently to new tasks.

  7. Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only

    cs.LG 2025-05 conditional novelty 6.0 of 10

    PORL fine-tunes a pre-trained policy online by first collecting epsilon-greedy interaction data and learning a fresh Q-function from scratch, removing the need for pre-trained critics or offline data.

  8. ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy

    cs.RO 2025-02 conditional novelty 6.0 of 10

    ConRFT fine-tunes VLA policies with an offline behavior-cloning plus Q-learning stage and an online consistency-policy RL stage with human corrections, reporting 96.3% average success on eight real-world tasks.

  9. Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuning

    cs.RO 2025-02 conditional novelty 6.0 of 10

    SGFT uses a simulation-trained value function to guide real-world exploration via potential-based reward shaping and short-horizon objectives, substantially improving fine-tuning sample efficiency.

  10. Value Flows

    cs.LG 2025-10 reject novelty 5.0 of 10

    Value Flows fits the full return distribution in RL with a flow-matching critic and reweights its learning objective by estimated return variance; the central theoretical guarantee does not follow from the stated equations.

  11. SimLauncher: Launching Sample-Efficient Real-world Robotic Reinforcement Learning via Simulation Pre-training

    cs.RO 2025-07 conditional novelty 5.0 of 10

    Simulation-pretrained policies, with digital-twin demos for critic bootstrapping and action proposals, cut real-world RL training time while reaching near-perfect success on three manipulation tasks.

Reference graph

Works this paper leans on

60 extracted references · 23 canonical work pages · cited by 11 Pith papers

  1. [1]

    Reincarnating reinforcement learning: Reusing prior computation to accelerate progress.Advances in neural information processing systems, 35:28955–28971, 2022

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress.Advances in neural information processing systems, 35:28955–28971, 2022

    2022

  2. [2]

    Meta reinforcement learning for sim-to-real domain adaptation

    Karol Arndt, Murtaza Hazara, Ali Ghadirzadeh, and Ville Kyrki. Meta reinforcement learning for sim-to-real domain adaptation. In2020 IEEE international conference on robotics and automation (ICRA), pages 2725–2731. IEEE, 2020

    work page 2020

  3. [4]

    Efficient online reinforcement learning with offline data.arXiv preprint arXiv:2302.02948, 2023

    Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data.arXiv preprint arXiv:2302.02948, 2023

    arXiv 2023

  4. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  5. [6]

    Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learning fast without a model. InInternational Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=AY8zfZm0tDd

    2021

  6. [7]

    Cheng, T

    C. Cheng, T. Xie, N. Jiang, and A. Agarwal. Adversarially Trained Actor Critic for Offline RL.ICML, 2022

    work page 2022

  7. [8]

    Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  8. [9]

    Offline meta reinforcement learning–identifiability challenges and effective data collection strategies.Advances in Neural Information Processing Systems, 34:4607–4618, 2021

    Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta reinforcement learning–identifiability challenges and effective data collection strategies.Advances in Neural Information Processing Systems, 34:4607–4618, 2021

    work page 2021

Show all 60 references
  1. [10]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023

    2023 arXiv

  2. [11]

    Rl2: Fast reinforcement learning via slow reinforcement learning.arXiv preprint arXiv:1611.02779, 2016

    Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl2: Fast reinforcement learning via slow reinforcement learning.arXiv preprint arXiv:1611.02779, 2016. 13 Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data

    2016 arXiv

  3. [13]

    Diagnosing bottlenecks in deep q-learning algorithms

    Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q-learning algorithms. InProceedings of the 36th International Conference on Machine Learning. PMLR, 2019

    2019

  4. [14]

    D4rl: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219, 2020

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219, 2020

    2004 arXiv

  5. [15]

    A minimalist approach to offline reinforcement learning

    Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2106.06860, 2021

    2021 arXiv

  6. [16]

    Amago: Scalable in-context reinforcement learning for adaptive agents

    Jake Grigsby, Linxi Fan, and Yuke Zhu. Amago: Scalable in-context reinforcement learning for adaptive agents. arXiv preprint arXiv:2310.09971, 2023

    2023 arXiv

  7. [17]

    Relaypolicylearning: Solving long-horizon tasks via imitation and reinforcement learning.arXiv preprint arXiv:1910.11956, 2019

    AbhishekGupta, VikashKumar, CoreyLynch, SergeyLevine, andKarolHausman. Relaypolicylearning: Solving long-horizon tasks via imitation and reinforcement learning.arXiv preprint arXiv:1910.11956, 2019

    1910 arXiv

  8. [18]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. InarXiv, 2018. URL https://arxiv.org/pdf/ 1801.01290.pdf

    2018 arXiv

  9. [19]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. InInternational conference on machine learning, pages 1861–1870. PMLR, 2018

    2018

  10. [20]

    Double q-learning

    Hado van Hasselt. Double q-learning. InProceedings of the 23rd International Conference on Neural Information Processing Systems - Volume 2, 2010

    2010

  11. [21]

    Dropout q-functions for doubly efficient reinforcement learning

    Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning. InInternational Conference on Learning Representations, 2022. URLhttps://openreview.net/forum?id=xCVJMsPv3RT

    2022

  12. [22]

    Imitation bootstrapped reinforcement learning

    Hengyuan Hu, Suvir Mirchandani, and Dorsa Sadigh. Imitation bootstrapped reinforcement learning. arXiv preprint arXiv:2311.02198, 2023

    2023 arXiv

  13. [23]

    Continual model-based reinforcement learning with hypernetworks

    Yizhou Huang, Kevin Xie, Homanga Bharadhwaj, and Florian Shkurti. Continual model-based reinforcement learning with hypernetworks. In2021 IEEE International Conference on Robotics and Automation (ICRA), pages 799–805. IEEE, 2021

    2021

  14. [24]

    Seizing serendipity: Exploiting the value of past success in off-policy actor-critic.arXiv preprint arXiv:2306.02865, 2023

    Tianying Ji, Yu Luo, Fuchun Sun, Xianyuan Zhan, Jianwei Zhang, and Huazhe Xu. Seizing serendipity: Exploiting the value of past success in off-policy actor-critic.arXiv preprint arXiv:2306.02865, 2023

    2023 arXiv

  15. [25]

    Efficient planning in a compact latent action space.arXiv preprint arXiv:2208.10291, 2022

    Zhengyao Jiang, Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktäschel, Edward Grefen- stette, and Yuandong Tian. Efficient planning in a compact latent action space.arXiv preprint arXiv:2208.10291, 2022

    2022 arXiv

  16. [26]

    Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of sciences, 114(13): 3521–3526, 2017

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of scienc...

    2017

  17. [27]

    Policysearchformotorprimitivesinrobotics

    JensKoberandJanPeters. Policysearchformotorprimitivesinrobotics. Advancesinneuralinformation processing systems, 21, 2008

    2008

  18. [28]

    Offline reinforcement learning with fisher divergence critic regularization.arXiv preprint arXiv:2103.08050, 2021

    Ilya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization.arXiv preprint arXiv:2103.08050, 2021

    2021 arXiv

  19. [29]

    Conservative q-learning for offline reinforcement learning.Advances in Neural Information Processing Systems, 33:1179–1191, 2020

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning.Advances in Neural Information Processing Systems, 33:1179–1191, 2020

    2020

  20. [30]

    Offline-to-online rein- forcement learning via balanced replay and pessimistic Q-ensemble

    Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online rein- forcement learning via balanced replay and pessimistic Q-ensemble. InConference on Robot Learning, pages 1702–1712. PMLR, 2022

    2022

  21. [32]

    Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning.arXiv preprint arXiv:2401.16013, 2024

    Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning.arXiv preprint arXiv:2401.16013, 2024

    2024 arXiv

  22. [33]

    Fmb: a functional manipulation benchmark for generalizable robotic learning.The Interna- tional Journal of Robotics Research, 44(4):592–606, 2025

    Jianlan Luo, Charles Xu, Fangchen Liu, Liam Tan, Zipeng Lin, Jeffrey Wu, Pieter Abbeel, and Sergey Levine. Fmb: a functional manipulation benchmark for generalizable robotic learning.The Interna- tional Journal of Robotics Research, 44(4):592–606, 2025

    2025

  23. [34]

    Finetuning from offline re- inforcement learning: Challenges, trade-offs and practical solutions.arXiv preprint arXiv:2303.17396, 2023

    Yicheng Luo, Jackie Kay, Edward Grefenstette, and Marc Peter Deisenroth. Finetuning from offline re- inforcement learning: Challenges, trade-offs and practical solutions.arXiv preprint arXiv:2303.17396, 2023

    2023 arXiv

  24. [35]

    Offline-boosted actor-critic: Adaptively blending optimal historical behaviors in deep off-policy rl.arXiv preprint arXiv:2405.18520, 2024

    Yu Luo, Tianying Ji, Fuchun Sun, Jianwei Zhang, Huazhe Xu, and Xianyuan Zhan. Offline-boosted actor-critic: Adaptively blending optimal historical behaviors in deep off-policy rl.arXiv preprint arXiv:2405.18520, 2024

    2024 arXiv

  25. [36]

    Awac: Accelerating online reinforce- ment learning with offline datasets.arXiv preprint arXiv:2006.09359, 2020

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforce- ment learning with offline datasets.arXiv preprint arXiv:2006.09359, 2020

    2006 arXiv

  26. [37]

    Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning.Advances in Neural Information Processing Systems, 36, 2024

    Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning.Advances in Neural Information Processing Systems, 36, 2024

    2024

  27. [38]

    The primacy bias in deep reinforcement learning

    Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. InInternational conference on machine learning, pages 16828– 16847. PMLR, 2022

    2022

  28. [39]

    The difficulty of passive learning in deep reinforcement learning.Advances in Neural Information Processing Systems, 34:23283–23295, 2021

    Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning.Advances in Neural Information Processing Systems, 34:23283–23295, 2021

    2021

  29. [40]

    CORA: Benchmarks, baselines, and metrics as a platform for continual reinforcement learning agents

    Sam Powers, Eliot Xing, Eric Kolve, Roozbeh Mottaghi, and Abhinav Gupta. CORA: Benchmarks, baselines, and metrics as a platform for continual reinforcement learning agents. InConference on Lifelong Learning Agents, pages 705–743. PMLR, 2022

    2022

  30. [41]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  31. [42]

    Moto: Offline pre-training to online fine-tuning for model-based robot learning

    Rafael Rafailov, Kyle Beltran Hatch, Victor Kolev, John D Martin, Mariano Phielipp, and Chelsea Finn. Moto: Offline pre-training to online fine-tuning for model-based robot learning. InConference on Robot Learning, pages 3654–3671. PMLR, 2023

    2023

  32. [43]

    Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.arXiv preprint arXiv:1709.10087, 2017

    Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.arXiv preprint arXiv:1709.10087, 2017

    2017 arXiv

  33. [44]

    Efficient off-policy meta-reinforcement learning via probabilistic context variables

    Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. InInternational conference on machine learning, pages 5331–5340. PMLR, 2019

    2019

  34. [45]

    Diffusion policy policy optimization.arXiv preprint arXiv:2409.00588, 2024

    Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization.arXiv preprint arXiv:2409.00588, 2024

    2024 arXiv

  35. [46]

    The University of Texas at Austin, 1994

    Mark Bishop Ring.Continual learning in reinforcement environments. The University of Texas at Austin, 1994

    1994

  36. [47]

    Promp: Proximal meta-policy search.arXiv preprint arXiv:1810.06784, 2018

    Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search.arXiv preprint arXiv:1810.06784, 2018

    2018 arXiv

  37. [48]

    Learning from demonstration.Advances in neural information processing systems, 9, 1996

    Stefan Schaal. Learning from demonstration.Advances in neural information processing systems, 9, 1996

    1996

  38. [49]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  39. [50]

    Hybrid RL: Using both offline and online data can make RL efficient.arXiv preprint arXiv:2210.06718, 2022

    Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL: Using both offline and online data can make RL efficient.arXiv preprint arXiv:2210.06718, 2022

    2022 arXiv

  40. [51]

    Hybrid RL: Using both offline and online data can make RL efficient

    Yuda Song, Yifei Zhou, Ayush Sekhari, Drew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL: Using both offline and online data can make RL efficient. InThe Eleventh International Conference on Learning Representations, 2023. URLhttps://openreview.net/forum?id=yyBis80iUuU

    2023

  41. [52]

    Some considerations on learning to explore via meta-reinforcement learning.arXiv preprint arXiv:1803.01118, 2018

    Bradly C Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reinforcement learning.arXiv preprint arXiv:1803.01118, 2018

    2018 arXiv

  42. [53]

    Revisiting the minimalist approach to offline reinforcement learning.Advances in Neural Information Processing Systems, 36, 2024

    Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning.Advances in Neural Information Processing Systems, 36, 2024

    2024

  43. [54]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  44. [55]

    Jump-startreinforcementlearning

    Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, ChuyuanFu, CongMa, JiantaoJiao, etal. Jump-startreinforcementlearning. InInternational Conference on Machine Learning, pages 34556–34583. PMLR, 2023. 16 Efficient Online Reinforce...

    2023

  45. [56]

    Continual world: A robotic benchmark for continual reinforcement learning.Advances in Neural Information Processing Systems, 34:28496–28510, 2021

    Maciej Wołczyk, Michał Zając, Razvan Pascanu, Łukasz Kuciński, and Piotr Miłoś. Continual world: A robotic benchmark for continual reinforcement learning.Advances in Neural Information Processing Systems, 34:28496–28510, 2021

    2021

  46. [57]

    Fine-tuning reinforcement learning models is secretly a forgetting mitigation problem.arXiv preprint arXiv:2402.02868, 2024

    Maciej Wołczyk, Bartłomiej Cupiał, Mateusz Ostaszewski, Michał Bortkiewicz, Michał Zając, Razvan Pascanu, Łukasz Kuciński, and Piotr Miłoś. Fine-tuning reinforcement learning models is secretly a forgetting mitigation problem.arXiv preprint arXiv:2402.02868, 2024

    2024 arXiv

  47. [58]

    Policy finetuning: Bridging sample-efficient offline and online reinforcement learning.Advances in neural information processing systems, 34:27395–27407, 2021

    Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning.Advances in neural information processing systems, 34:27395–27407, 2021

    2021

  48. [59]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023

    2023

  49. [60]

    Policy expansion for bridging offline-to-online reinforcement learning

    Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforcement learning. arXiv preprint arXiv:2302.00935, 2023

    2023 arXiv

  50. [61]

    Aperspective of q-value estimation on offline-to-online reinforcement learning

    YinminZhang, JieLiu, ChumingLi, YazheNiu, YaodongYang, YuLiu, andWanliOuyang. Aperspective of q-value estimation on offline-to-online reinforcement learning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16908–16916, 2024

    2024

  51. [62]

    Offline data enhanced on-policy policy gradient with provable guarantees.arXiv preprint arXiv:2311.08384, 2023

    Yifei Zhou, Ayush Sekhari, Yuda Song, and Wen Sun. Offline data enhanced on-policy policy gradient with provable guarantees.arXiv preprint arXiv:2311.08384, 2023

    2023 arXiv

  52. [63]

    no free lunch

    ZhiyuanZhou,PranavAtreya,AbrahamLee,HomerWalke,OierMees,andSergeyLevine. Autonomous improvement of instruction following skills via foundation models.arXiv preprint arXiv:2407.20635, 2024. 17 Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data Appe...

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.