Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

The apparently inconsistent results in offline-to-online reinforcement learning are explained by a stability-plasticity principle: fine-tuning should preserve whichever of the pretrained policy or the offline dataset is stronger, and the pa

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Offline-to-online fine-tuning works best when the chosen method preserves the stronger of the pretrained policy or the offline dataset; a three-regime taxonomy organizes these choices.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection The three-regime taxonomy is a genuinely useful organizing idea and the 63-setting sweep is substantial, but the headline counts are inflated by a select-and-test protocol; the directional claim likely holds, the exact numbers do not. the 4 major comments →

arxiv 2510.01460 v4 pith:CZFYMMBG submitted 2025-10-01 cs.LG cs.AI

The Three Regimes of Offline-to-Online Reinforcement Learning

Lu Li , Tianwei Ni , Yihao Sun , Pierre-Luc Bacon This is my paper
classification cs.LG cs.AI
keywords offline-to-online reinforcement learningstability-plasticity principlefine-tuningthree regimespretrained policyoffline datasetknowledge decompositionD4RL
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the puzzling variability in offline-to-online reinforcement learning comes down to a stability-plasticity trade-off. It defines three regimes by comparing the return of the pretrained policy (J(π0)) with the return of the offline dataset's behavior policy (J(πD)): Superior, Comparable, and Inferior. In the Superior regime, methods that protect the pretrained policy (π0-centric) tend to win; in the Inferior regime, methods that reuse the offline data (D-centric) tend to win. The authors test this on 63 settings and find 45/63 correct predictions with only 3 opposite mismatches. If right, practitioners can choose a fine-tuning method by simply comparing two scalar values.

Core claim

The paper introduces a knowledge decomposition: final performance equals the stronger offline prior, plus a non-positive stability term, plus a non-negative plasticity term. It then argues that the key is to anchor stability on whichever prior is stronger. This yields three regimes: Superior (J(π0) > J(πD)), Comparable (J(π0) ≈ J(πD)), and Inferior (J(π0) < J(πD)). In their large-scale study, π0-centric methods outperform D-centric ones in 24/32 Superior settings, and D-centric methods outperform π0-centric ones in 19/23 Inferior settings. The framework is intended to guide design choices, replacing trial-and-error with a regime-based selection.

What carries the argument

The central object is the stability-plasticity principle with two distinct forms of stability: stability around the pretrained policy π0 and stability around the offline dataset D. The key identity is Eq. (6), the knowledge decomposition: max_n J(π_n) = max(J(π0), J(πD)) + Stability(max(J(π0), J(πD))) + Plasticity. This identity makes the regime taxonomy operational: if π0 is stronger, stability should anchor on π0; if D is stronger, anchor on D.

Load-bearing premise

The paper assumes that offline RL regularization, when applied during fine-tuning, keeps the online policy near the pretrained policy π0 even though π0 itself is not stored or referenced; if this anchoring does not actually happen, then labeling such methods as π0-centric is wrong and the regime predictions do not test what they claim.

What would settle it

Run an Inferior-regime task (J(π0) < J(πD)) with offline RL regularization and measure the KL divergence or action distance between the online policy and π0 over time. If the online policy does not stay closer to π0 than to the behavior policy that generated the dataset, the implicit-anchoring assumption—and thus the π0-centric label—is falsified. Alternatively, find a setting where J(π0) > J(πD) but a D-centric method (offline data replay) consistently beats all π0-centric methods across a hyperparameter sweep; that would contradict the Superior-regime prediction.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners can choose a fine-tuning method by comparing J(π0) and J(πD): use π0-centric methods (online warmup, offline RL regularization) in the Superior regime and D-centric methods (offline data replay, reset) in the Inferior regime.
  • The framework explains why a single state-of-the-art algorithm fails across benchmarks: it often anchors stability on the weaker source of prior knowledge.
  • In the Comparable regime, π0-centric and D-centric methods are predicted to perform similarly when fully optimized, with small gaps sensitive to hyperparameters.
  • Parameter reset can be beneficial in the Inferior regime, where the pretrained policy provides little useful knowledge and plasticity is more important.
  • The decomposition provides a quantitative way to measure stability and plasticity empirically, enabling stage-dependent guidance (e.g., stability matters more early in fine-tuning, plasticity later).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would be to replace the discrete three-regime taxonomy with a continuous strength-ratio weighting, where the degree of π0- vs D-centricity is tuned by the margin J(π0) - J(πD).
  • The framework's underlying assumption that offline RL regularization implicitly anchors learning near π0 could be directly tested by measuring the online policy's divergence from π0 during fine-tuning; such a test would clarify the scope of the π0-centric category.
  • The same two-anchor stability logic might transfer beyond return comparisons, e.g., using data coverage or task-specific success rates to define regimes in sparse-reward or long-horizon settings.
  • If the implicit-anchoring assumption fails in some settings, the regime predictions could be re-derived using only explicitly stored anchors, which would change which methods are labeled π0-centric.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a stability–plasticity framework for offline-to-online RL. It defines stability as the worst-case performance drop below an offline baseline and plasticity as the range of performance improvement during fine-tuning, and presents Eq. (6) as a knowledge decomposition. Based on the relative returns of the pretrained policy J(π0) and the dataset behavior policy J(πD), it defines Superior, Comparable, and Inferior regimes and predicts that π0-centric methods (online warmup, offline RL regularization) should win in Superior, D-centric methods (offline data replay, reset) should win in Inferior, and the two families should be comparable in Comparable. The paper validates the framework on 21 dataset-task compositions, three pretraining algorithms, and six fine-tuning variants (63 settings), reporting 45/63 correct predictions and only 3 opposite mismatches. It also reports stability/plasticity measurements, δ-sensitivity analysis, and alternative taxonomies.

Significance. The main value is a simple, falsifiable decision rule: compare two scalars, J(π0) and J(πD), to decide whether to anchor fine-tuning to the pretrained policy or to the offline dataset. The study is large-scale (63 settings, 10 seeds, four D4RL domains, three pretraining algorithms) and includes sensitivity analyses and alternative taxonomies, which are strengths. The directional findings in Superior and Inferior regimes are nontrivial and likely informative. However, the theoretical decomposition is tautological, and the headline empirical counts rely on a select-and-test protocol that is statistically invalid; the Comparable-regime prediction is not actually confirmed. These issues are fixable and do not necessarily undermine the directional conclusion, but the manuscript as written overstates the strength of evidence.

major comments (4)
  1. [§B.2, Table 2] The headline counts (24/32, 19/23, 45/63) are obtained by per-setting selection of the stronger variant within each class using IQM on the same evaluation data, followed by a two-sided t-test between the two selected variants on the last-10 evaluations from the same seeds. This is a select-and-test procedure: the compared quantity is a per-setting maximum of two noisy estimators, so the expected gap is inflated and the nominal α=0.05 t-test is invalid. The counts therefore mix true regime effects with selection noise. Please report comparisons using a pre-specified representative method per class, or select variants on separate seeds, or use a proper multiple-testing/selection correction. This is load-bearing because the paper's practical advice rests on these counts.
  2. [§3.2, Eq. (6)] Equation (6) is an algebraic identity given definitions (2), (4), and (5): max_n J(π_n) = J*_off + (min_n J(π_n) − J*_off) + (max_n J(π_n) − min_n J(π_n)). Presenting it as a 'knowledge decomposition' attributes substantive content to a bookkeeping relation. The paper should either explicitly call it an identity and base the framework's value on the empirical regime-specific predictions, or derive a non-tautological decomposition (for example, under a model of policy updates). As written, the theoretical framing is circular.
  3. [§4.2] The classification of offline RL regularization as π0-centric relies on the claim that regularization 'implicitly anchors learning around the region visited by π0, even if π0 is not stored during fine-tuning.' No direct evidence is provided for this claim. Offline RL regularization typically anchors to the support of the offline dataset D, not necessarily to the policy π0, and these can differ substantially when π0 is obtained by BC or offline RL with distribution shift. Because this labeling determines the central comparison, please provide evidence (e.g., distance between occupancy measures, a behavior-cloning baseline, or an explicit policy-regularization ablation) or soften the claim and discuss how misclassification would affect the conclusions.
  4. [§5.4, Table 2] The Comparable-regime prediction is not supported by the reported statistics: only 2/8 settings are statistically indistinguishable, with 2/8 favoring π0-centric and 4/8 favoring D-centric methods. The paper explains this by noting that mean gaps are under 0.1 in 6/8 settings, but with the select-and-test protocol this is not a valid equivalence test. The Comparable regime requires a pre-specified equivalence test on fixed representatives; otherwise the claim that the framework 'predicts' comparable outcomes in this regime is not confirmed.
minor comments (5)
  1. [§5.2] Typo: 'identifying the the method' should read 'identifying the method'.
  2. [Appendix C.2, Table 15] The table caption says 'red cells: opposite mismatches (34/63); gray cells: adjacent mismatches (3/63)', but the text reports 3 opposite mismatches and 34 adjacent mismatches. The caption should be corrected.
  3. [Figures 3–5] The markers such as 'o : 0 D' and '× : 0 < D' are garbled and hard to read. Please render the regime symbols as proper mathematical notation or use explicit text labels.
  4. [§3.2, Eq. (6)] The term 'Final knowledge' is defined as max_n J(π_n), not J(π_N). This should be clarified, since the stated objective of fine-tuning is the final policy's return, and the distinction matters for interpreting stability and plasticity.
  5. [§5.1, Tables 1/5–8] Stability and plasticity are reported as averages over settings with different scales; consider reporting per-regime normalized values or medians to avoid the impression that the aggregate numbers are directly comparable.

Circularity Check

1 steps flagged

Eq. (6) is a definitional identity rather than an independent decomposition, but the three-regime claim and its empirical validation are not circular.

specific steps
  1. self definitional [Section 3.2, Equations (2), (5), (6)]
    "By relating these concepts through a knowledge decomposition, we have: max_{0≤n≤N} J(π_n) = J*_off + Stability(J*_off) + Plasticity. This equation states that the final knowledge an agent acquires after offline pretraining and online fine-tuning is the outcome of three interacting components."

    Substituting definitions (2) and (5) into the RHS: since π_0 is in {π_n}, min_j J(π_j) ≤ J(π_0) ≤ J*_off, so Stability(J*_off) = min_j J(π_j) − J*_off. Then J*_off + Stability + Plasticity = J*_off + (min_j J − J*_off) + (max_i J − min_j J) = max_i J(π_i). Thus Eq. (6) is the identity max = max; the 'decomposition' is imposed by the definitions and holds for every trajectory, so it is not an independent derivation. It is a mild, non-load-bearing circularity because the three-regime comparison is not derived from Eq. (6).

full rationale

The only self-definitional element is Eq. (6), which reduces to an algebraic identity once Stability and Plasticity are defined as in Eqs. (2) and (5). I do not treat it as load-bearing: the three-regime taxonomy in §3.2 is defined directly by comparing J(π_0) and J(π_D), and the validation in §5 compares pre-specified method classes (π0-centric vs D-centric) across 63 independently defined settings, reporting 45/63 agreements and only 3 opposite mismatches. No parameter is fitted to the outcome, and Appendix C shows that alternative taxonomies (Q-based and BC-based) perform worse, which is real evidence that the chosen taxonomy has empirical content. The only author-overlapping citation (Nikishin et al. 2022, which includes P.-L. Bacon) motivates parameter reset but is background empirical knowledge and is not used to force the central claim. The select-best-variant-then-t-test protocol noted by the skeptic is a statistical weakness rather than a circularity. Overall, the central claim has independent grounding, so the circularity score is low.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The central validation uses two hand-chosen thresholds (δ for regime, α for replay), an unverified mapping from offline-regularization to π0-stability, and a scalar-return abstraction of the dataset. These do not make the empirical claim circular, but they bound how much the 63-setting result can say beyond the specific protocol.

free parameters (3)
  • regime margin δ = 0.05
    Used in two one-sided t-tests to assign settings to Superior/Comparable/Inferior. δ=0 gives 46 correct, δ=0.1 gives 37 correct (Tables 3–4), so the headline 45/63 depends on this hand-chosen threshold.
  • offline data replay ratio α = 0.5
    All D-centric replay variants sample 50% offline data per batch. The performance of D-centric methods (and hence the regime comparisons) depends on this value, which is taken from RLPD rather than tuned or justified for each setting.
  • online warmup steps K = 5000
    The π0-centric warmup variant collects K steps before gradient updates; results in the Superior regime could change with K, and no sensitivity analysis is provided.
axioms (4)
  • domain assumption Dataset knowledge is summarized by scalar J(πD) = average trajectory return.
    §3.1; regime assignments and the whole framework inherit this simplification; the paper acknowledges coverage is ignored.
  • ad hoc to paper Preserving the stronger prior while maintaining plasticity is the necessary condition for efficient fine-tuning.
    This is the proposed principle, not derived from a theorem; §3.2.
  • ad hoc to paper Offline RL regularization implicitly anchors around π0 even when π0 is not stored.
    §4.2; required for classifying it as π0-centric.
  • domain assumption The 63 settings across D4RL are representative enough to validate the framework.
    The paper generalizes from these benchmarks in §6.
invented entities (2)
  • Abstract behavior policy πD no independent evidence
    purpose: Represents the offline dataset as a single policy with return J(πD), so the dataset can compete with π0 in regime classification.
    Introduced in §3.1 as an abstraction; estimated only through average trajectory return, with no independent behavioral characterization.
  • Three-regime taxonomy (Superior/Comparable/Inferior) no independent evidence
    purpose: Discretizes the continuous relative performance of J(π0) and J(πD) into categories that prescribe stability source.
    The categories are construction choices; the paper's own sensitivity analysis (δ=0/0.1) shows assignments can move settings between regimes, and §6 calls the discretization a simplification.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of The Three Regimes of Offline-to-Online Reinforcement Learning." pith.science (2026). https://pith.science/paper/CZFYMMBG

@misc{pith2026251001460,
  author       = {Pith},
  title        = {Pith review of: The Three Regimes of Offline-to-Online Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CZFYMMBG}},
  note         = {Machine review of arXiv:2510.01460}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Offline-to-online reinforcement learning (RL) has emerged as a practical paradigm that leverages offline datasets for pretraining and online interactions for fine-tuning. However, its empirical behavior is highly inconsistent: design choices of online fine-tuning that work well in one setting can fail completely in another. Guided by the stability--plasticity principle, we propose a framework that can explain this inconsistency: We argue that efficient fine-tuning must preserve the utility of the stronger offline prior, whether that is the pretrained policy or the offline dataset, while maintaining sufficient plasticity. This perspective identifies three regimes of online fine-tuning, each requiring distinct stability properties. We validate this framework through a large-scale empirical study, finding that the results strongly align with its predictions in 45 out of 63 cases, with only 3 opposite mismatches. This work provides a framework for guiding design choices in offline-to-online RL based on the relative performance of the offline dataset and the pretrained policy.

Figures

Figures reproduced from arXiv: 2510.01460 by Lu Li, Pierre-Luc Bacon, Tianwei Ni, Yihao Sun.

Figure 1
Figure 1. Figure 1: Comparison between WSRL (pre￾trained policy only) and RLPD (offline dataset only) on two representative offline￾to-online RL tasks. All learning curves are shown as mean ± 95% CI. While offline-to-online RL has led to promising re￾sults (Nair et al., 2020; Lee et al., 2022), online RL fine-tuning suffers from highly inconsistent empiri￾cal behavior: design choices that work well in one setting can fail com… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the three regimes in offline-to-online RL, defined based on the relative performance of the pretrained policy J(π0) and the offline dataset J(πD). For each regime, our framework indicates which property is most needed during fine-tuning. The boxes at the right show representative design choices that implement these enhancing stability or plasticity. Dashed arrows denote weaker connections than … view at source ↗
Figure 3
Figure 3. Figure 3: Representative fine-tuning results in the [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Representative results in the Inferior regime: the first six results are correct predictions, while the remaining two show an adjacent mismatch and an opposite mismatch. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Representative fine-tuning results for the [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of Q-function behavior in different regimes. Fine-tuning without offline data [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Full fine-tuning results in the Superior regime. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Full fine-tuning results in the Inferior regime. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Full fine-tuning results in the Comparable regime. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priors

    cs.LG 2026-06 unverdicted novelty 6.0

    Argues for shifting to diagnosis-driven tension management of offline priors in online RL, supported by a framework on prior roles, experiments showing help-or-hurt reversals, and cross-domain evidence.

  2. ROAD: Adaptive Data Mixing for Offline-to-Online Reinforcement Learning via Bi-Level Optimization

    cs.LG 2026-05 unverdicted novelty 6.0

    ROAD formulates data mixing as a bi-level optimization problem solved via multi-armed bandit to adaptively balance offline priors and online updates in RL.

Reference graph

Works this paper leans on

38 extracted references · 10 linked inside Pith · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Efficient online reinforcement learning with offline data

    Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pp.\ 1577--1594. PMLR, 2023

    2023

  3. [3]

    Data quality in imitation learning

    Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. Data quality in imitation learning. Advances in neural information processing systems, 36: 0 80375--80395, 2023

    2023

  4. [4]

    Magnetic control of tokamak plasmas through deep reinforcement learning

    Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602 0 (7897): 0 414--419, 2022

    2022

  5. [5]

    Loss of plasticity in deep continual learning

    Shibhansh Dohare, J Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632 0 (8026): 0 768--774, 2024

    2024

  6. [6]

    D4rl: Datasets for deep data-driven reinforcement learning

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

    Pith/arXiv arXiv 2004

  7. [7]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp.\ 1587--1596. PMLR, 2018

    2018

  8. [8]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.\ 1861--1870. Pmlr, 2018

    2018

  9. [9]

    Bayesian design principles for offline-to-online reinforcement learning

    Hao Hu, Yiqin Yang, Jianing Ye, Chengjie Wu, Ziqing Mai, Yujing Hu, Tangjie Lv, Changjie Fan, Qianchuan Zhao, and Chongjie Zhang. Bayesian design principles for offline-to-online reinforcement learning. In International Conference on Machine Learning, pp.\ 19491--19515. PMLR, 2024

    2024

  10. [10]

    Overcoming catastrophic forgetting in neural networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 0 (13): 0 3521--3526, 2017

    2017

  11. [11]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021

    Pith/arXiv arXiv 2021

  12. [12]

    Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble

    Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pp.\ 1702--1712. PMLR, 2022

    2022

  13. [13]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020

    Pith/arXiv arXiv 2005

  14. [14]

    Proto: Iterative policy regularized offline-to-online reinforcement learning

    Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, and Ya-Qin Zhang. Proto: Iterative policy regularized offline-to-online reinforcement learning. arXiv preprint arXiv:2305.15669, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Energy-guided diffusion sampling for offline-to-online reinforcement learning

    Xu-Hui Liu, Tian-Shuo Liu, Shengyi Jiang, Ruifeng Chen, Zhilong Zhang, Xinwei Chen, and Yang Yu. Energy-guided diffusion sampling for offline-to-online reinforcement learning. In International Conference on Machine Learning, pp.\ 31541--31565. PMLR, 2024

    2024

  16. [16]

    Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory

    James L McClelland, Bruce L McNaughton, and Randall C O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102 0 (3): 0 419, 1995

    1995

  17. [17]

    The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects

    Martial Mermillod, Aur \'e lia Bugaiska, and Patrick Bonin. The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in psychology, 4: 0 504, 2013

    2013

  18. [18]

    Human-level control through deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518 0 (7540): 0 529--533, 2015

    2015

  19. [19]

    Awac: Accelerating online reinforcement learning with offline datasets

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020

    Pith/arXiv arXiv 2006

  20. [20]

    Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning

    Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36: 0 62244--62269, 2023

    2023

  21. [21]

    The primacy bias in deep reinforcement learning

    Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp.\ 16828--16847. PMLR, 2022

    2022

  22. [22]

    An algorithmic perspective on imitation learning

    Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics , 7 0 (1-2): 0 1--179, 2018

    2018

  23. [23]

    Experience replay for continual learning

    David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019

    2019

  24. [24]

    Progressive neural networks

    Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016

    Pith/arXiv arXiv 2016

  25. [25]

    Learning from demonstration

    Stefan Schaal. Learning from demonstration. Advances in neural information processing systems, 9, 1996

    1996

  26. [26]

    Mastering the game of go without human knowledge

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550 0 (7676): 0 354--359, 2017

    2017

  27. [27]

    The dormant neuron phenomenon in deep reinforcement learning

    Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, pp.\ 32145--32168. PMLR, 2023

    2023

  28. [28]

    Hybrid rl: Using both offline and online data can make rl efficient

    Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint arXiv:2210.06718, 2022

    Pith/arXiv arXiv 2022

  29. [29]

    Feedback in imitation learning: The three regimes of covariate shift

    Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and J Andrew Bagnell. Feedback in imitation learning: The three regimes of covariate shift. arXiv preprint arXiv:2102.02872, 2021

    Pith/arXiv arXiv 2021

  30. [30]

    Revisiting the minimalist approach to offline reinforcement learning

    Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 36: 0 11592--11620, 2023

    2023

  31. [31]

    Jump-start reinforcement learning

    Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Jos \'e phine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. In International Conference on Machine Learning, pp.\ 34556--34583. PMLR, 2023

    2023

  32. [32]

    Empirical study of off-policy policy evaluation for reinforcement learning

    Cameron Voloshin, Hoang M Le, Nan Jiang, and Yisong Yue. Empirical study of off-policy policy evaluation for reinforcement learning. arXiv preprint arXiv:1911.06854, 2019

    Pith/arXiv arXiv 1911

  33. [33]

    Fine-tuning reinforcement learning models is secretly a forgetting mitigation problem

    Maciej Wolczyk, Bart omiej Cupia , Mateusz Ostaszewski, Micha Bortkiewicz, Micha Zaj a c, Razvan Pascanu, ukasz Kuci \'n ski, and Piotr Mi o \'s . Fine-tuning reinforcement learning models is secretly a forgetting mitigation problem. In International Conference on Machine Learning, pp.\ 53039--53078. PMLR, 2024

    2024

  34. [34]

    Policy expansion for bridging offline-to-online reinforcement learning

    Haichao Zhang, Wei Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforcement learning. In International Conference on Learning Representations ( ICLR ) , 2023

    2023

  35. [35]

    Efficient online reinforcement learning fine-tuning need not retain offline data

    Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforcement learning fine-tuning need not retain offline data. arXiv preprint arXiv:2412.07762, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  37. [37]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  38. [38]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv 2000

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.