REVIEW 19 cited by
Hybrid RL: Using Both Offline and Online Data Can Make RL Efficient
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We consider a hybrid reinforcement learning setting (Hybrid RL), in which an agent has access to an offline dataset and the ability to collect experience via real-world online interaction. The framework mitigates the challenges that arise in both pure offline and online RL settings, allowing for the design of simple and highly effective algorithms, in both theory and practice. We demonstrate these advantages by adapting the classical Q learning/iteration algorithm to the hybrid setting, which we call Hybrid Q-Learning or Hy-Q. In our theoretical results, we prove that the algorithm is both computationally and statistically efficient whenever the offline dataset supports a high-quality policy and the environment has bounded bilinear rank. Notably, we require no assumptions on the coverage provided by the initial distribution, in contrast with guarantees for policy gradient/iteration methods. In our experimental results, we show that Hy-Q with neural network function approximation outperforms state-of-the-art online, offline, and hybrid RL baselines on challenging benchmarks, including Montezuma's Revenge.
Forward citations
Cited by 19 Pith papers
-
On the Complexity of Offline Reinforcement Learning with $Q^\star$-Approximation and Partial Coverage
Q*-realizability plus Bellman completeness is insufficient for sample-efficient offline RL under partial coverage, and a new decision-estimation framework recovers and improves existing bounds.
-
Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysis
A unified framework proves that augmenting any confidence-based online RL algorithm with offline data yields order-optimal suboptimality-gap and regret bounds, with a new concentrability coefficient that separates the...
-
A Unified Algorithmic Framework for Hybrid Reinforcement Learning in Tabular MDPs with Shifted Transition Dynamics
A tabular hybrid RL framework with shifted transition dynamics, claiming near-optimal regret and suboptimality bounds under a known per-state-action bias bound.
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
Learning Upper Lower Value Envelopes to Shape Online RL: A Principled Approach
A two-stage RL framework learns value-function envelopes from offline data and uses them to shape online exploration, yielding regret bounds that improve as offline data grows.
-
The Three Regimes of Offline-to-Online Reinforcement Learning
Offline-to-online fine-tuning works best when the chosen method preserves the stronger of the pretrained policy or the offline dataset; a three-regime taxonomy organizes these choices.
-
Decentralized Relaxed Smooth Optimization with Gradient Descent Methods
A decentralized gradient descent method with adaptive clipping is claimed to reach best-known convergence rates for convex and nonconvex problems under (L0,L1)-smoothness without knowing the constants.
-
Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Review
A survey proposing adaptability as a three-part taxonomy (learning, policy, scenario-driven) for organizing and evaluating MARL under changing conditions.
-
Online Pre-Training for Offline-to-Online Reinforcement Learning
A new 'online pre-training' phase trains a second value function that is then blended with the offline one during fine-tuning, improving offline-to-online RL across D4RL benchmarks.
-
Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach
IRO trains a sequence of small value functions that reweight a frozen LLM's own samples, achieving test-time alignment gains that outperform Best-of-N search on AlpacaEval 2.0 and TL;DR.
-
Reinforcement Learning via Implicit Imitation Guidance
A reinforcement learning method that learns a state-dependent covariance from expert-policy action differences and uses it as exploration noise, improving sample efficiency on sparse-reward continuous control tasks.
-
Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment
SquareχPO, a square-loss variant of χPO, achieves optimal 1/sqrt(n) suboptimality under label privacy and Huber corruption for offline direct alignment with general function classes.
-
Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only
PORL fine-tunes a pre-trained policy online by first collecting epsilon-greedy interaction data and learning a fresh Q-function from scratch, removing the need for pre-trained critics or offline data.
-
From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignment
FOREWARN steers a diffusion robot policy at runtime by using a world model to predict latent futures and a vision-language model to narrate and rank those futures in natural language.
-
Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration
A hybrid online-plus-offline preference optimization algorithm, HPO, provably needs fewer samples than pure online or offline RLHF in linear MDP settings.
-
Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
Warm-start RL (WSRL) fine-tunes offline-pretrained RL agents online with no offline data retention, using 5,000 warm-up rollouts from the frozen pre-trained policy followed by standard high-UTD SAC.
-
Coverage Analysis for Digital Cousin Selection -- Improving Multi-Environment Q-Learning
A coverage-coefficient based rule for choosing synthetic environments in multi-environment Q-learning is derived, implemented as CCQ, and shown to outperform partial ordering in random graph experiments.
-
Improving the Data-efficiency of Reinforcement Learning by Warm-starting with LLM
Warm-starting RL with a small LLM-generated buffer and then fine-tuning online (LORO) improves cumulative reward over pure RL in six Gym environments, though the theoretical guarantee is borrowed and the coverage assu...
-
Dynamic Action Interpolation: A Universal Approach for Accelerating Reinforcement Learning with Expert Guidance
Dynamic Action Interpolation linearly blends expert and RL actions with a time-decaying weight and claims faster learning and higher final rewards, but the supporting theory is asserted rather than derived.
Discussion (0). Continue with ORCID to comment.