Pith. sign in

REVIEW 3 major objections 6 minor 64 references

Naive Q-function pretraining rarely helps online RL fine-tuning of a pretrained policy; multi-policy rollouts do.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 11:00 UTC pith:U6QAUMQ3

load-bearing objection Clean negative result on Q-pretraining; the IPE gain is real-looking but the interaction budget may not be equalized against baselines. the 3 major comments →

arxiv 2607.27203 v1 pith:U6QAUMQ3 submitted 2026-07-29 cs.LG

Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?

Perry Dong , Ron Polonsky , Dorsa Sadigh , Chelsea Fin This is my paper
classification cs.LG
keywords online RL fine-tuningQ-function pretrainingpretrained policiesoffline-to-online RLpolicy ensemblecontinuous controlvalue mismatch
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When you already have a strong pretrained policy, conventional wisdom says you should also pretrain the Q-function on offline data before online RL fine-tuning. This paper shows that often does not help, and can even hurt, relative to starting the Q-function from scratch. The reason is a mismatch: offline pretraining learns the value of the base policy, not the value function that online RL actually converges to, and offline value-maximization tricks do not close that gap. The authors instead bootstrap the replay buffer with rollouts from several policies trained on the same data distribution, giving the critic broader action coverage around the base policy. On six hard continuous-control manipulation tasks, that simple change improves fine-tuning success by about 26% over naive Q pretraining.

Core claim

Naive offline pretraining of the Q-function, given a fixed pretrained base policy, usually yields little or no gain over a randomly initialized Q-function for online fine-tuning, because the pretrained critic converges to Q of the base policy rather than the Q that online RL needs; pooling environment rollouts from an ensemble of policies trained on the same action distribution lets online fine-tuning learn a better critic and raises final performance.

What carries the argument

Initialization via Policy Ensemble (IPE): train several policies on the base policy’s action distribution, roll them out in the environment, and use the pooled transitions to seed the online replay buffer so the Q-function sees diverse nearby actions instead of collapsing toward the base policy’s state values.

Load-bearing premise

The mismatch and the benefit of multi-policy rollouts are assumed to hold beyond the specific off-policy algorithm and sparse-reward continuous-control tasks studied here.

What would settle it

On the same tasks and pretrained policies, show that a carefully pretrained single-policy Q-function matches or beats IPE in online fine-tuning success, or that IPE’s gain disappears under a different off-policy or on-policy fine-tuning algorithm with matched data budget.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • For value-based fine-tuning of a fixed pretrained policy, random Q initialization is a strong default; spending compute on naive offline Q pretraining is often wasted.
  • Offline value maximization on top of the base policy does not substitute for online data when the target is the online RL Q-function.
  • Seeding online RL with rollouts from several policies on the same data distribution is a practical lever for better critics and higher fine-tuning success.
  • Fine-tuning performance generally improves as more distinct policies contribute to the initial rollout pool, at fixed total trajectory count.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Large supervised policies (including vision-language-action models) fine-tuned with offline-to-online RL may gain more from diverse near-policy rollouts than from fitting a critic solely to the base policy’s offline actions.
  • Methods that force the critic to see counterfactual actions near the base policy—without full multi-policy data collection—could capture much of IPE’s benefit at lower interaction cost.
  • If the base data distribution is far from optimal, the same mismatch may be larger, so the relative value of ensemble initialization versus pure online learning could shift.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies whether pretraining a Q-function on offline data helps online RL fine-tuning of a fixed, supervised-pretrained policy, using EXPO as the off-policy backbone on six Robomimic/OGBench manipulation tasks. Its negative result: naive TD pretraining of the critic on Doffline gives little or no benefit over random critic initialization (Fig. 2), and the authors attribute this to a mismatch between Q^{π_base} (what offline pretraining converges to) and Q^{π*_RL} (what online fine-tuning needs), probed via a "preference accuracy" metric against an ensemble of near-optimal reference critics (Fig. 3). Three offline value-maximization fixes are shown not to close the gap (Figs. 4–5). As a constructive contribution, the paper proposes IPE: train N additional policies on the same action distribution, roll out all N+1 policies in the environment to seed the replay buffer, then fine-tune. IPE yields an average 1.26× improvement over naive Q-pretraining (Fig. 7), with performance increasing in N at fixed total trajectory budget (Fig. 9). Appendix A replicates the negative result at UTD 20 for EXPO and RLPD.

Significance. If the results hold, the paper makes a useful negative contribution to a timely question (RL fine-tuning of pretrained/VLA-style policies): offline critic pretraining is not free head-start, and offline value-maximization patches do not fix it. Strengths worth naming: the negative result is replicated across two algorithms and two UTD regimes (Fig. 10); the preference-accuracy probe with an ensemble of 10 independently trained reference critics is a reasonable, non-circular diagnostic; the N-scaling study (Fig. 9) holds total trajectory count fixed, which is the right way to isolate diversity from data quantity; and the overall comparison is against external task success rather than a fitted quantity. The practical contribution (IPE) is potentially valuable but currently rests on a comparison whose interaction-budget fairness is not established.

major comments (3)
  1. [§5, Algorithm 1; §6.2, Figure 7] Algorithm 1 (steps 3–5) seeds the replay buffer with fresh environment rollouts from N+1 policies; IPE is therefore not an offline-pretraining method, and its comparison to the naive Q-pretraining and random-init baselines is interaction-fair only if those baselines receive an equal-budget warm-up from πbase alone (the N=0 case the text equates with 'the warm up phase'). The manuscript never states whether they do. §6.3 (Fig. 9) holds total trajectories fixed while varying N, which isolates diversity within IPE, but the missing control is 'naive/random init + equal-budget single-policy warm-up' vs. IPE in Fig. 7. Without it, part of the 26% gain may be attributable to extra online data that naive pretraining could equally exploit, and the framing of IPE as 'a better use of the offline data' (§5) is not supported. This ablation is feasible within the current setup and should be added, wit
  2. [§4.2, Figure 3; Appendix B.1] The central mechanistic claim (Takeaway 2) rests on preference accuracy against an ensemble of 10 reference critics treated as Qπ*_RL. Two issues. (a) The probe's validity is asserted, not argued: when πbase is itself trained to near-100% success (§4.2), it is unclear that a well-calibrated critic should systematically prefer πref actions over πbase actions at states where both succeed—preference then reflects return differences that may be within noise; a chance level (0.5) and the actual values behind 'quite close' in Fig. 3 are not reported in the text. (b) The Qπbase ≠ Qπ*_RL gap is, in sparse-reward tasks, close to definitional once fine-tuning changes the policy; the load-bearing question is whether the gap causes the Fig. 2 failure, and the paper's alternative explanation (Q collapse to Vπbase from narrow coverage, Fig. 8) is arguably more direct. The authors should either quantif
  3. [§6; §7 (limitations)] The negative result is shown to replicate across algorithms and UTD (Appendix A, Fig. 10: EXPO and RLPD at UTD 20), but the positive result—IPE's 1.26× improvement—is demonstrated only with EXPO at UTD 1. Since EXPO's edit policy and on-the-fly max over 8 action samples (Eqs. 2–3) make the fine-tuning dynamics unusually sensitive to the critic's action-ranking around πbase, the IPE benefit may partly be specific to this algorithm class. Running IPE with at least one generic off-policy algorithm (RLPD is already set up) would substantially de-risk the practical claim; if the benefit does not transfer, the abstract's unqualified 'across a suite of continuous control benchmarks' should be scoped accordingly.
minor comments (6)
  1. [Abstract; §6.2, Figure 7] The headline 'average 1.26×' (Abstract, Fig. 1) does not state the aggregation: mean of per-task ratios? Ratios of final success rates or of area-under-curve? Mixing Robomimic sparse success and OGBench normalized scores in one ratio average needs justification, and with three seeds per task a confidence interval on the ratio should be reported.
  2. [Table 1 (Appendix B.2)] Each row reports Nadd (additional generated rollouts) and pf (failure fraction), but the text never explains which policy generated these rollouts or whether they are added to Doffline. As written the table is not interpretable.
  3. [§5, Algorithm 1; Figure 6] Step 1 trains N policies 'via supervised learning on the πbase action distribution' using 'the identical dataset or different subsets thereof.' If the dataset is identical, diversity presumably comes only from seeds/architecture; Fig. 6 suggests seeds suffice, but this should be stated explicitly, and the Fig. 6 visualization method (PCA? t-SNE? raw action dims?) and axes should be labeled.
  4. [§4.2] Notation is inconsistent: π* and π*_RL are used interchangeably in §4.2, and Qπ*_RL is used before the reference-critic construction (Appendix B.1) is introduced. 'Ground truth RL policy' overstates an empirical estimate; 'reference policy' would be more accurate.
  5. [Various] Typos/grammar: 'all of which of trained' (§6.2); 'fine-tune pretrained policies policy' (§2); inconsistent hyphenation of fine-tune/finetune throughout; 'policies to collect diverse rollouts, IPE is able to better learn' (Fig. 7 caption) is a sentence fragment.
  6. [§6.1, Figure 8] Figure 8's claim that the single-policy Q-function is 'essentially deterministic' is strong; please report the variance values quantitatively and clarify whether variance is computed across actions at fixed states and averaged over states.

Circularity Check

0 steps flagged

Empirical RL study with external success metrics; no derivation reduces to its inputs by construction.

full rationale

The paper’s load-bearing claims are empirical comparisons (pretrained Q vs random Q vs IPE) measured by environment success rates and by preference accuracy against independently trained reference critics that solve the tasks to ~100%. Nothing is defined in terms of the quantity it is said to predict; no parameter is fitted on a subset and then reported as a closely related “prediction”; and no uniqueness theorem or ansatz is imported from prior author work to force the result. Self-citations to EXPO/EXPO-FT supply the off-policy testbed algorithm, which is normal and non-circular: Appendix A also reports the same negative Q-pretraining finding under RLPD, and the mismatch diagnosis plus IPE gains are new measurements on external benchmarks, not restatements of those citations. Experimental-design concerns (e.g., whether IPE’s multi-policy environment rollouts equalize interaction budget with baselines) affect correctness risk, not circularity of a derivation chain. Score 0; steps empty.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The work is empirical. It inherits the standard MDP/TD-learning setting and the EXPO actor-critic machinery from prior work. Load-bearing modeling choices are the preference-accuracy construction, the decision to hold total pretraining trajectories fixed while varying N, and the assumption that multi-policy supervised clones on the same action distribution supply useful action diversity. No new physical entities are postulated; free parameters are ordinary RL hyperparameters and the ensemble size N.

free parameters (5)
  • N (number of ensemble policies for IPE) = best at N=5 in experiments
    Chosen by ablation; N=5 best in reported range. Performance claim depends on this choice and on holding total trajectories fixed.
  • Edit action scale β / edit bound = 0.1
    EXPO hyperparameter (0.1 in Table 2) that shapes the action set the critic sees during fine-tuning.
  • Offline-to-online batch ratio = 0.5
    Fixed at 0.5; affects how much offline data continues to regularize online updates.
  • Number of Q-functions / min-Q ensemble size = 10 Qs, min 2
    10 Q-functions, min over 2; standard but chosen, not derived.
  • Nadd and failure fraction pf per environment = task-specific (e.g., square 200/0.3)
    Table 1 sets environment-specific extra rollout counts and failure fractions used to build IPE buffers.
axioms (5)
  • standard math Standard discounted MDP and TD learning objective for the critic (Eq. 1).
    Background RL formalism assumed throughout Sections 3–4.
  • domain assumption EXPO’s base+edit+on-the-fly max policy is a valid and representative off-policy fine-tuning algorithm for studying Q initialization.
    All main experiments and the mismatch diagnosis are run inside EXPO; Appendix A adds RLPD but does not cover on-policy methods.
  • ad hoc to paper Preference accuracy against an ensemble of near-100% RL reference critics is a faithful probe of whether a critic ranks actions like Qπ*RL.
    Defined in Section 4.2 / Appendix B.1; central to the mismatch claim.
  • domain assumption Multiple supervised policies trained on the same action distribution produce sufficiently diverse rollouts to escape Q≈V collapse.
    Premise of IPE, supported by Figures 6 and 8 but not proved in general.
  • domain assumption Offline datasets and sparse/subtask rewards in Robomimic and OGBench are representative enough for the negative pretraining conclusion.
    Six tasks, all continuous-control manipulation; generalization beyond this regime is assumed in the discussion.
invented entities (2)
  • Initialization via Policy Ensemble (IPE) no independent evidence
    purpose: Procedure that trains N extra policies on the base action distribution, rolls them out, and pools transitions to bootstrap the online critic.
    Core proposed method (Algorithm 1). It is an engineering construct, not a new physical or mathematical object; independent evidence is the reported success-rate gains on the six tasks.
  • Preference accuracy metric no independent evidence
    purpose: Fraction of states where a critic prefers a reference optimal action over a base-policy action; used to show Qπbase ≠ Qπ*RL.
    Introduced in Section 4.2 to diagnose the mismatch. Falsifiable on any task with a strong reference policy, but defined for this paper’s analysis.

pith-pipeline@v1.2.0-grok45-kimik3 · 17431 in / 3516 out tokens · 56403 ms · 2026-07-30T11:00:22.727617+00:00 · methodology

0 comments
read the original abstract

Pre-training followed by fine-tuning has become the dominant recipe for learning performant policies, and in value-based reinforcement learning (RL) this raises a natural question: given a pretrained policy, should the Q-function be pretrained on offline data too? Conventional wisdom suggests it should, but recent results show that online RL with a randomly-initialized Q-function can result in highly performant and reliable policies without needing to pretrain the Q-function. In this paper, we systematically study whether pretraining the Q-function actually helps when fine-tuning on top of a pretrained base policy. We find, surprisingly, that naive Q-function pretraining often provides little benefit over random initialization. We show this stems from a fundamental mismatch: the Q-function learned during pretraining targets the pretrained policy's Q-function, not the Q-function that online fine-tuning converges to, and this gap persists even after offline value maximization. Motivated by this finding, we propose Initialization via Policy Ensemble (IPE), a simple method that trains multiple diverse policies and uses their pooled rollouts to bootstrap the Q-function learning in online RL. Across a suite of challenging continuous control benchmarks, IPE yields an average 1.26x improvement in fine-tuning performance over naive Q-function pre-training.

Figures

Figures reproduced from arXiv: 2607.27203 by Chelsea Fin, Dorsa Sadigh, Perry Dong, Ron Polonsky.

Figure 1
Figure 1. Figure 1: IPE. Left: naively pretraining does not help RL fine-tuning performance as Q πbase ̸= Q π ∗ RL . Mid: IPE enables online RL fine-tuning to better learn Q π ∗ RL through diverse action coverage. Right: IPE results in an average of 26% improvement over directly pretraining the Q-function. Pretraining followed by using reinforcement learning (RL) to finetune has become the predominant approach of learning rel… view at source ↗
Figure 2
Figure 2. Figure 2: Pretraining the Q-function does not improve performance for downstream fine-tuning. Success rate over steps of online fine-tuning for a pretrained Q-function vs a randomly initialized Q-function, both sharing the same pretrained policy. policy, and then dig into why Q-function pretraining fails to help (Section 4.2) even under several natural attempts to fix it. Setup. We evaluate on six challenging manipu… view at source ↗
Figure 3
Figure 3. Figure 3: Preference of Q πbase and Q π ∗ RL . Q πbase is not the same as Q π ∗ RL even for a near-optimal base policy, but the Q-functions have preferences close to each other. policy (see Appendix B.1 for details). A well-calibrated Q-function should generally prefer π ∗ ’s actions. Why does pretraining not help? We hypothesize that the central issue is a mismatch between Qπbase , the value function the offline da… view at source ↗
Figure 4
Figure 4. Figure 4: Performance of online RL after offline value maximization. Performing offline value maximization does not improve the performance over directly pretraining, suggesting online data and learning is necessary [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Preference score analysis of offline value maximization. The preference of actions after performing offline value maximization stays similar to pretraining without value maximization. Takeaway 3: Fixing pretraining post hoc through offline value maximization is insufficient. Offline value-maximization techniques do not close the gap between Qπbase and Qπ ∗ RL ; more data helps incrementally but does not re… view at source ↗
Figure 6
Figure 6. Figure 6: Actions from a single policy vs multiple policies. Actions sampled from one pretrained policy are significantly less diverse than actions pooled across three different policies trained on the same action distribution, the latter giving the Q-function meaningful coverage to learn from [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: IPE. Through using multiple policies to collect diverse rollouts, IPE is able to better learn Q π ∗ RL from RL finetuning. is not that the data is uninformative, but that pretraining directly on πbase overly constrains the Q-function to Qπbase . This problem is compounded by a second issue: a single trained policy, even an expressive one such as a diffusion or flow-matching policy, tends to produce a narro… view at source ↗
Figure 9
Figure 9. Figure 9: Scaling the number of policies for IPE. Increasing the number of policies used to collect rollouts results in better performance. At N = 0, this recovers the warm up phase for online RL. As we show in Section 6.3, larger N typically yields higher performance. We summarize IPE in Algorithm 1. 6 EXPERIMENTS Our experiments study how well IPE enables effective fine-tuning on top of a pretrained base policy. C… view at source ↗
Figure 8
Figure 8. Figure 8: Variance of Q-functions. The Q-function trained on data from 1 policy exhibits very low variance, where the Q￾function is essentially deterministic. We first test the premise behind IPE: that a single pretrained policy yields insufficiently diverse actions for the Q-function to learn beyond V πbase [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Evaluation success rates of pretraining versus no pretraining of Q-functions. Pretraining Q-functions for both EXPO with UTD 20 and RLPD with UTD 20 do not result in improvement in performance compared to a randomly initialized Q-function. B EXPERIMENT DETAILS B.1 PREFERENCE ACCURACY We use a preference accuracy metric to evaluate whether a learned critic recovers action preferences identified by a strong… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 28 linked inside Pith

  1. [1]

    Advances in Neural Information Processing Systems , volume=

    Hybrid policy optimization from imperfect demonstrations , author=. Advances in Neural Information Processing Systems , volume=

  2. [2]

    2025 , eprint=

    Reinforcement Learning via Implicit Imitation Guidance , author=. 2025 , eprint=

    2025

  3. [3]

    Proceedings of the 40th International Conference on Machine Learning , pages =

    Jump-Start Reinforcement Learning , author =. Proceedings of the 40th International Conference on Machine Learning , pages =. 2023 , editor =

    2023

  4. [5]

    2025 , eprint=

    Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer , author=. 2025 , eprint=

    2025

  5. [6]

    2024 , eprint=

    Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning , author=. 2024 , eprint=

    2024

  6. [7]

    2018 , eprint=

    Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards , author=. 2018 , eprint=

    2018

  7. [8]

    2025 , eprint=

    Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning , author=. 2025 , eprint=

    2025

  8. [9]

    2025 , eprint=

    Flow Q-Learning , author=. 2025 , eprint=

    2025

  9. [10]

    2023 , eprint=

    IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies , author=. 2023 , eprint=

    2023

  10. [11]

    2026 , eprint=

    Value Flows , author=. 2026 , eprint=

    2026

  11. [12]

    2022 , eprint=

    Training language models to follow instructions with human feedback , author=. 2022 , eprint=

    2022

  12. [14]

    2026 , eprint=

    TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse , author=. 2026 , eprint=

    2026

  13. [15]

    2019 , eprint=

    Off-Policy Deep Reinforcement Learning without Exploration , author=. 2019 , eprint=

    2019

  14. [16]

    2021 , eprint=

    What Matters in Learning from Offline Human Demonstrations for Robot Manipulation , author=. 2021 , eprint=

    2021

  15. [17]

    2018 , eprint=

    Residual Reinforcement Learning for Robot Control , author=. 2018 , eprint=

    2018

  16. [18]

    2021 , eprint=

    A Minimalist Approach to Offline Reinforcement Learning , author=. 2021 , eprint=

    2021

  17. [19]

    2025 , eprint=

    OGBench: Benchmarking Offline Goal-Conditioned RL , author=. 2025 , eprint=

    2025

  18. [20]

    2019 , eprint=

    Residual Policy Learning , author=. 2019 , eprint=

    2019

  19. [21]

    2018 , eprint=

    RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation , author=. 2018 , eprint=

    2018

  20. [22]

    2021 , eprint=

    AWAC: Accelerating Online Reinforcement Learning with Offline Datasets , author=. 2021 , eprint=

    2021

  21. [24]

    2017 , eprint=

    Proximal Policy Optimization Algorithms , author=. 2017 , eprint=

    2017

  22. [25]

    Learning from Demonstration , url =

    Schaal, Stefan , booktitle =. Learning from Demonstration , url =

  23. [26]

    2018 , eprint=

    Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations , author=. 2018 , eprint=

    2018

  24. [27]

    2024 , eprint=

    Diffusion Policy Policy Optimization , author=. 2024 , eprint=

    2024

  25. [28]

    2025 , eprint=

    ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy , author=. 2025 , eprint=

    2025

  26. [29]

    2026 , eprint=

    EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models , author=. 2026 , eprint=

    2026

  27. [30]

    2025 , eprint=

    Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data , author=. 2025 , eprint=

    2025

  28. [31]

    2023 , eprint=

    Policy Expansion for Bridging Offline-to-Online Reinforcement Learning , author=. 2023 , eprint=

    2023

  29. [32]

    2023 , eprint=

    Efficient Online Reinforcement Learning with Offline Data , author=. 2023 , eprint=

    2023

  30. [33]

    2025 , eprint=

    EXPO: Stable Reinforcement Learning with Expressive Policies , author=. 2025 , eprint=

    2025

  31. [34]

    Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine

    Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data, 2023. URL https://arxiv.org/abs/2302.02948

    Pith/arXiv arXiv 2023

  32. [35]

    Conrft: A reinforced fine-tuning method for vla models via consistency policy, 2025

    Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, and Dongbin Zhao. Conrft: A reinforced fine-tuning method for vla models via consistency policy, 2025. URL https://arxiv.org/abs/2502.05450

    Pith/arXiv arXiv 2025

  33. [36]

    Lessing, Annie S

    Perry Dong, Alec M. Lessing, Annie S. Chen, and Chelsea Finn. Reinforcement learning via implicit imitation guidance, 2025 a . URL https://arxiv.org/abs/2506.07505

    Pith/arXiv arXiv 2025

  34. [37]

    Expo: Stable reinforcement learning with expressive policies, 2025 b

    Perry Dong, Qiyang Li, Dorsa Sadigh, and Chelsea Finn. Expo: Stable reinforcement learning with expressive policies, 2025 b . URL https://arxiv.org/abs/2507.07986

    Pith/arXiv arXiv 2025

  35. [38]

    Expo-ft: Sample-efficient reinforcement learning finetuning for vision-language-action models, 2026 a

    Perry Dong, Kuo-Han Hung, Tian Gao, Dorsa Sadigh, and Chelsea Finn. Expo-ft: Sample-efficient reinforcement learning finetuning for vision-language-action models, 2026 a . URL https://arxiv.org/abs/2605.25477

    Pith/arXiv arXiv 2026

  36. [39]

    Tql: Scaling q-functions with transformers by preventing attention collapse, 2026 b

    Perry Dong, Kuo-Han Hung, Alexander Swerdlow, Dorsa Sadigh, and Chelsea Finn. Tql: Scaling q-functions with transformers by preventing attention collapse, 2026 b . URL https://arxiv.org/abs/2602.01439

    arXiv 2026

  37. [40]

    Value flows, 2026 c

    Perry Dong, Chongyi Zheng, Chelsea Finn, Dorsa Sadigh, and Benjamin Eysenbach. Value flows, 2026 c . URL https://arxiv.org/abs/2510.07650

    Pith/arXiv arXiv 2026

  38. [41]

    A minimalist approach to offline reinforcement learning, 2021

    Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning, 2021. URL https://arxiv.org/abs/2106.06860

    Pith/arXiv arXiv 2021

  39. [42]

    Off-policy deep reinforcement learning without exploration, 2019

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration, 2019. URL https://arxiv.org/abs/1812.02900

    Pith/arXiv arXiv 2019

  40. [43]

    Idql: Implicit q-learning as an actor-critic method with diffusion policies, 2023

    Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies, 2023. URL https://arxiv.org/abs/2304.10573

    Pith/arXiv arXiv 2023

  41. [44]

    Imitation bootstrapped reinforcement learning

    Hengyuan Hu, Suvir Mirchandani, and Dorsa Sadigh. Imitation bootstrapped reinforcement learning. arXiv preprint arXiv:2311.02198, 2023

    Pith/arXiv arXiv 2023

  42. [45]

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch...

  43. [46]

    Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, Danny Driess, Michael Equi, Adnan Esmail, Yunhao Fang, Chelsea Finn, Catherine Glossop, Thomas Godden, Ivan Goryachev, Lachy Groom, Hunter Hancock, Karol Hausman, Gashon Hussein, Brian Ichter, Szym...

    Pith/arXiv arXiv 2025

  44. [47]

    Residual reinforcement learning for robot control, 2018

    Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Residual reinforcement learning for robot control, 2018. URL https://arxiv.org/abs/1812.03201

    Pith/arXiv arXiv 2018

  45. [48]

    Roboturk: A crowdsourcing platform for robotic skill learning through imitation, 2018

    Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, Silvio Savarese, and Li Fei-Fei. Roboturk: A crowdsourcing platform for robotic skill learning through imitation, 2018. URL https://arxiv.org/abs/1811.02790

    Pith/arXiv arXiv 2018

  46. [49]

    What matters in learning from offline human demonstrations for robot manipulation, 2021

    Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation, 2021. URL https://arxiv.org/abs/2108.03298

    Pith/arXiv arXiv 2021

  47. [50]

    Offline retraining for online rl: Decoupled policy learning to mitigate exploration bias

    Max Sobol Mark, Archit Sharma, Fahim Tajwar, Rafael Rafailov, Sergey Levine, and Chelsea Finn. Offline retraining for online rl: Decoupled policy learning to mitigate exploration bias. arXiv preprint arXiv:2310.08558, 2023

    Pith/arXiv arXiv 2023

  48. [51]

    Awac: Accelerating online reinforcement learning with offline datasets, 2021

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets, 2021. URL https://arxiv.org/abs/2006.09359

    Pith/arXiv arXiv 2021

  49. [52]

    Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning, 2024

    Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning, 2024. URL https://arxiv.org/abs/2303.05479

    Pith/arXiv arXiv 2024

  50. [53]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback,...

    Pith/arXiv arXiv 2022

  51. [54]

    Ogbench: Benchmarking offline goal-conditioned rl, 2025 a

    Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl, 2025 a . URL https://arxiv.org/abs/2410.20092

    Pith/arXiv arXiv 2025

  52. [55]

    Flow q-learning, 2025 b

    Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning, 2025 b . URL https://arxiv.org/abs/2502.02538

    Pith/arXiv arXiv 2025

  53. [56]

    Learning complex dexterous manipulation with deep reinforcement learning and demonstrations, 2018

    Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations, 2018. URL https://arxiv.org/abs/1709.10087

    Pith/arXiv arXiv 2018

  54. [57]

    Ren, Justin Lidard, Lars L

    Allen Z. Ren, Justin Lidard, Lars L. Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization, 2024. URL https://arxiv.org/abs/2409.00588

    Pith/arXiv arXiv 2024

  55. [58]

    Learning from demonstration

    Stefan Schaal. Learning from demonstration. In M.C. Mozer, M. Jordan, and T. Petsche (eds.), Advances in Neural Information Processing Systems, volume 9. MIT Press, 1996. URL https://proceedings.neurips.cc/paper_files/paper/1996/file/68d13cf26c4b4f4f932e3eff990093ba-Paper.pdf

    1996

  56. [59]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    Pith/arXiv arXiv 2017

  57. [60]

    Residual policy learning, 2019

    Tom Silver, Kelsey Allen, Josh Tenenbaum, and Leslie Kaelbling. Residual policy learning, 2019. URL https://arxiv.org/abs/1812.06298

    Pith/arXiv arXiv 2019

  58. [61]

    Gemini Robotics Team, Abbas Abdolmaleki, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Ashwin Balakrishna, Nathan Batchelor, Alex Bewley, Jeff Bingham, Michael Bloesch, Konstantinos Bousmalis, Philemon Brakel, Anthony Brohan, Thomas Buschmann, Arunkumar Byravan, Serkan Cabi, Ken Caluwaerts, Federico Casarini, Christ...

    Pith/arXiv arXiv 2025

  59. [62]

    Jump-start reinforcement learning

    Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Jos\' e phine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, Sergey Levine, and Karol Hausman. Jump-start reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International...

    2023

  60. [63]

    Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards, 2018

    Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards, 2018. URL https://arxiv.org/abs/1707.08817

    Pith/arXiv arXiv 2018

  61. [64]

    Posterior behavioral cloning: Pretraining bc policies for efficient rl finetuning, 2025

    Andrew Wagenmaker, Perry Dong, Raymond Tsao, Chelsea Finn, and Sergey Levine. Posterior behavioral cloning: Pretraining bc policies for efficient rl finetuning, 2025. URL https://arxiv.org/abs/2512.16911

    arXiv 2025

  62. [65]

    Hybrid policy optimization from imperfect demonstrations

    Hanlin Yang, Chao Yu, Siji Chen, et al. Hybrid policy optimization from imperfect demonstrations. Advances in Neural Information Processing Systems, 36: 0 4653--4663, 2023

    2023

  63. [66]

    Policy expansion for bridging offline-to-online reinforcement learning, 2023

    Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforcement learning, 2023. URL https://arxiv.org/abs/2302.00935

    Pith/arXiv arXiv 2023

  64. [67]

    Efficient online reinforcement learning fine-tuning need not retain offline data, 2025

    Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforcement learning fine-tuning need not retain offline data, 2025. URL https://arxiv.org/abs/2412.07762

    Pith/arXiv arXiv 2025