Pith. sign in

REVIEW 2 cited by

Value-Based Deep RL Scales Predictably

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.04327 v2 pith:GPJMLUMF submitted 2025-02-06 cs.LG

Oleh Rybkin , Michal Nauman , Preston Fu , Charlie Snell , Pieter Abbeel , Sergey Levine , Aviral Kumar This is my paper
classification cs.LG
keywords computedatagivenperformancebudgetpredictablescalingwhen
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Scaling data and compute is critical to the success of modern ML. However, scaling demands predictability: we want methods to not only perform well with more compute or data, but also have their performance be predictable from small-scale runs, without running the large-scale experiment. In this paper, we show that value-based off-policy RL methods are predictable despite community lore regarding their pathological behavior. First, we show that data and compute requirements to attain a given performance level lie on a Pareto frontier, controlled by the updates-to-data (UTD) ratio. By estimating this frontier, we can predict this data requirement when given more compute, and this compute requirement when given more data. Second, we determine the optimal allocation of a total resource budget across data and compute for a given performance and use it to determine hyperparameters that maximize performance for a given budget. Third, this scaling is enabled by first estimating predictable relationships between hyperparameters, which is used to manage effects of overfitting and plasticity loss unique to RL. We validate our approach using three algorithms: SAC, BRO, and PQL on DeepMind Control, OpenAI gym, and IsaacGym, when extrapolating to higher levels of data, compute, budget, or performance.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A single multi-task RL agent using a large regularized critic, categorical value loss, and task embeddings achieves state-of-the-art results across 283 tasks and transfers efficiently to new tasks.

  2. Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies

    cs.LG 2025-08 conditional novelty 4.0 of 10

    A survey that categorizes deep reinforcement learning scaling strategies into data, network, and training budget dimensions and outlines challenges for scaling DRL systems.

Pith tools