REVIEW 7 cited by
RvS: What is Essential for Offline RL via Supervised Learning?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent work has shown that supervised learning alone, without temporal difference (TD) learning, can be remarkably effective for offline RL. When does this hold true, and which algorithmic components are necessary? Through extensive experiments, we boil supervised learning for offline RL down to its essential elements. In every environment suite we consider, simply maximizing likelihood with a two-layer feedforward MLP is competitive with state-of-the-art results of substantially more complex methods based on TD learning or sequence modeling with Transformers. Carefully choosing model capacity (e.g., via regularization or architecture) and choosing which information to condition on (e.g., goals or rewards) are critical for performance. These insights serve as a field guide for practitioners doing Reinforcement Learning via Supervised Learning (which we coin "RvS learning"). They also probe the limits of existing RvS methods, which are comparatively weak on random data, and suggest a number of open problems.
Forward citations
Cited by 7 Pith papers
-
Nonreciprocal current induced by dissipation in time-reversal symmetric systems
Dissipation induces nonreciprocal current in time-reversal symmetric noncentrosymmetric systems via interband processes, inversely proportional to lifetime and linked to the shift vector.
-
Freeform Preference Learning for Robotic Manipulation
Language-conditioned multi-axis human preferences yield denser rewards and steerable robot policies that outperform sparse and binary-preference baselines by 38 points on long-horizon manipulation.
-
When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play
Supervised fine-tuning collapses LLM action diversity in board-game play beyond what the accuracy–diversity tradeoff requires; augmenting SFT data with all optimal actions per state partially prevents this.
-
Generative Sequential Notification Optimization via Multi-Objective Decision Transformers
A Decision Transformer with quantile-regression return prompts improved notification decisions at LinkedIn, boosting sessions by 0.72% over the deployed CQL baseline in a live A/B test.
-
Hybrid Sequence Modeling and Reinforced Verification for Controllable Target-Conditioned Decision Making
Doctor is a transformer-based offline RL method that samples candidate actions near a desired return and verifies them with a learned Q-function, improving target-return alignment.
-
Behavioral Exploration: Learning to Explore via In-Context Adaptation
A coverage-conditioned behavioral cloning policy adapts in-context to its own history, making robots explore new expert-like behaviors online without online reinforcement learning.
-
Reinforcement Learning: From Algorithms To Foundation Models
A dissertation uniting the author's published results: non-exploitable Nash-DQN policies and the FightLadder benchmark for games, plus diffusion/consistency-model world models for RL — a compilation rather than new results.
Discussion (0). Sign in to comment.