REVIEW 3 major objections 3 minor 1 cited by
Compute-Optimal Scaling for Value-Based Deep RL
T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read For fixed compute, online value-based deep RL has an optimal split between model capacity and update-to-data ratio, governed by a TD-overfitting effect in which batch size harms small models but not large ones.
desk verdict I can't judge the evidence because the wrong full text was provided, but the TD-overfitting question is real and the Q-accuracy-to-return proxy needs to be the referee's first check. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the TD-overfitting effect—the degradation in Q-function accuracy caused by large batches in small Q-networks—along with the two compute-allocation axes of model capacity and update-to-data (UTD) ratio. The fitted relationship between batch size, capacity, and Q-accuracy carries the argument, turning the observed interplay into a recipe for compute-optimal training.
What would settle it
Train a small Q-network and a large one on the same online RL task, sweep batch sizes under a fixed compute budget, and measure both held-out Q-accuracy and final policy returns. The central claim fails if the small model's Q-accuracy does not degrade with increasing batch size, or if the large model shows the same degradation.
Extended reading notes
Core claim
For online value-based deep RL, compute allocation is governed by two axes: model capacity and the update-to-data ratio. The paper's central claim is that these axes interact with batch size through a mechanism called TD-overfitting: large batches quickly harm Q-function accuracy in small models, but this harm disappears in large models, enabling effective large-batch training at scale. Given a fixed compute budget, this creates a specific optimal split between model size and UTD ratio that maximizes sample efficiency. The authors offer a mental model for the effect and translate it into concrete guidelines for choosing batch size and UTD in deep RL training.
Load-bearing premise
The guidelines assume that the relationship between batch size, model capacity, and Q-function accuracy measured on the paper's benchmarks is stable enough to transfer to other environments and compute budgets, and that Q-function accuracy is the right proxy for final policy performance.
Editorial extensions
If this is right
- Scaling batch size with model capacity becomes a principled default in online value-based RL instead of a per-environment guess.
- Large-model training can safely use large batches, improving throughput and compute efficiency.
- Small-model regimes should pair small batches with adjusted UTD ratios to avoid Q-accuracy collapse.
- Compute-budget planning in RL can be reframed as a two-axis allocation problem analogous to supervised scaling laws.
- Given a measured scaling relationship, practitioners can derive a specific batch-size and UTD recommendation from a compute budget.
Reading between the lines
- The same batch-capacity interaction may appear in actor-critic and offline RL methods that use TD-style targets, though the paper does not test those settings explicitly.
- Q-function accuracy is the paper's proxy for success; an independent check would be whether the recommended splits also maximize final policy returns across varied tasks.
- The guidelines are fitted on the authors' benchmarks, so porting them to new environments should start by re-measuring the batch-size/capacity curve rather than assuming the coefficients transfer.
- If TD-overfitting stems from target instability, then target-network smoothing strength should modulate the batch-size harm for small models—a testable extension of the proposed mental model.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The submission consists of an abstract (arXiv:2508.14881) claiming a compute-optimal scaling analysis for online value-based deep RL, identifying a phenomenon called 'TD-overfitting' and deriving guidelines for batch size and update-to-data (UTD) ratio. The full text that follows is, however, an unrelated manuscript (arXiv:2508.14882) on nonparametric knockoff generation and variable importance. The abstract asserts that increasing batch size quickly harms Q-function accuracy for small models but not for large models, and that this enables effective use of large batch sizes at scale. No equations, experiments, benchmark list, seeds, or error bars are provided in the abstract, and the body contains no reinforcement-learning content. The central claims are therefore unsupported in the submitted document.
Significance. If the abstract's claims were established, the paper would address a genuinely understudied axis of scaling for value-based deep RL and could provide practical guidance analogous to supervised-learning scaling laws. The proposed interaction between model size and batch size is a falsifiable and potentially useful empirical hypothesis, and naming 'TD-overfitting' gives the community a convenient handle. However, the submitted manuscript contains none of the supporting evidence: no fitted scaling curves, no benchmarks, no validation of Q-function accuracy as a proxy for final policy returns, and no comparison with alternative operating points. The body is on a different topic entirely. The potential significance cannot be assessed from this submission, and readers have no basis for using the proposed guidelines.
major comments (3)
- [Full text (all sections)] The body of the submission, from the title through the appendix, is a knockoff variable-selection paper (Mason and Fei, 'Novel Knockoff Generation and Importance Measures with Heterogeneous Data via Conditional Residuals and Local Gradients'). It contains no experiments with value-based deep RL, no TD learning, and no scaling curves. The abstract's central claims about TD-overfitting and compute-optimal UTD/batch-size allocation are therefore entirely unsupported by the submitted text. This is a load-bearing mismatch: no part of the claimed contribution can be checked.
- [Abstract] The abstract defines the harmful effect of batch size in terms of Q-function accuracy ('increasing the batch quickly harms Q-function accuracy') and then derives batch-size and UTD guidelines. In online value-based RL, Q-function accuracy is not shown to be monotone-aligned with policy returns; TD-error reductions can coexist with policy degradation through overestimation bias or value-instability-induced churn. The submitted document provides no evidence that maximizing Q-accuracy is equivalent to maximizing episodic return across model sizes and batch sizes. This proxy assumption is load-bearing because every downstream guideline inherits it.
- [Abstract] The findings are presented without equations, a task list, number of seeds, or error bars, and the 'mental model' is introduced after the experiments that revealed the effect. This creates a post-hoc-narrative risk: the scaling curves and guidelines may be fitted to the same data that motivated them. A concrete test would be out-of-domain validation on held-out environments or a pre-specified prediction of when the TD-overfitting effect appears. No such evidence is present in the submission.
minor comments (3)
- [Abstract] The term 'compute-optimal' is used without a formal definition of the compute budget or the optimization objective; the reader cannot tell whether the objective is sample efficiency, wall-clock time, or final performance at a fixed budget.
- [Abstract] The term 'TD-overfitting' is introduced without a precise mathematical definition, making it difficult to distinguish from ordinary optimization noise or the TD bias-variance trade-off.
- [Full text] The full text's title, keywords, references, and empirical sections are entirely unrelated to reinforcement learning. The editor should verify whether a wrong file was uploaded, but as submitted the document is not internally coherent with its abstract.
Circularity Check
No circularity established from the abstract; full-text mismatch prevents equation-level verification.
full rationale
The claimed paper is represented only by its abstract, which contains no equations, no named fitted parameters, no self-citations, and no invoked uniqueness theorems. The TD-overfitting phenomenon is presented as an empirical observation, and the 'mental model' and 'guidelines' are explicitly post-hoc ('We provide a mental model for understanding this phenomenon and build guidelines'), so they are not dressed up as independent predictions. No specific reduction of the form 'Eq. X = Eq. Y by construction' or 'fitted parameter renamed as prediction' can be exhibited. The concern that Q-function accuracy may not align with policy return is a validity and extrapolation risk, not a circularity: optimizing Q-accuracy does not, by definition, make the guidelines identical to the compute-optimal performance claim. The supplied full text is a different manuscript (arXiv:2508.14882 on knockoffs), so the claimed derivation chain is not present; this is a serious provenance/missing-support issue, but it is not evidence of circularity. Without the actual equations, experiments, or validation protocol, no circular step can be substantiated.
Assumptions & free parameters
free parameters (1)
- scaling-curve exponents and offsets for compute-optimal allocation =
not reported in abstract
assumptions (2)
- domain assumption Q-function accuracy during training is a reliable proxy for final policy performance
- domain assumption Empirical scaling relations measured on the authors' benchmark suite generalize across environments and compute budgets
invented entities (1)
-
TD-overfitting
Cite this review
Pith. "Pith review of Compute-Optimal Scaling for Value-Based Deep RL." pith.science (2026). https://pith.science/paper/WUCPB37L
@misc{pith2026250814881,
author = {Pith},
title = {Pith review of: Compute-Optimal Scaling for Value-Based Deep RL},
year = {2026},
howpublished = {\url{https://pith.science/paper/WUCPB37L}},
note = {Machine review of arXiv:2508.14881}
}
read the original abstract
As models grow larger and training them becomes expensive, it becomes increasingly important to scale training recipes not just to larger models and more data, but to do so in a compute-optimal manner that extracts maximal performance per unit of compute. While such scaling has been well studied for language modeling, reinforcement learning (RL) has received less attention in this regard. In this paper, we investigate compute scaling for online, value-based deep RL. These methods present two primary axes for compute allocation: model capacity and the update-to-data (UTD) ratio. Given a fixed compute budget, we ask: how should resources be partitioned across these axes to maximize sample efficiency? Our analysis reveals a nuanced interplay between model size, batch size, and UTD. In particular, we identify a phenomenon we call TD-overfitting: increasing the batch quickly harms Q-function accuracy for small models, but this effect is absent in large models, enabling effective use of large batch size at scale. We provide a mental model for understanding this phenomenon and build guidelines for choosing batch size and UTD to optimize compute usage. Our findings provide a grounded starting point for compute-optimal scaling in deep RL, mirroring studies in supervised learning but adapted to TD learning.
Forward citations
Cited by 1 Pith paper
-
ReBRAC-v2: The Return of the King
A fixed-recipe offline RL method combining normalizing-flow actors, categorical critics, staged training, and test-time refinement beats recent flow-based baselines by 22.5 points averaged over ten OGBench categories.
Reference graph
Works this paper leans on
-
[4]
Nonparametric Bayesian Knockoff Generators for Feature Selection Under Complex Data Structure
URL https://github.com/jazoller96/mammalian-methyl-clocks/ tree/main. Michael J Martens, Anjishnu Banerjee, Xinran Qi, and Yushu Shi. Bayesian knockoff gener- ators for robust inference under complex data structure. arXiv preprint arXiv:2111.06985 ,
-
[9]
Proof of Theorem 1 Theorem 2 (Exchangeability of Conditional Residual Knockoffs) Let X = (X1,
24 Heterogeneous Nonparametric Knockoffs Appendix A. Proof of Theorem 1 Theorem 2 (Exchangeability of Conditional Residual Knockoffs) Let X = (X1, . . . , Xp) ∈ Rp be a random vector, and the observed data are X ∈ Rn×p. Denote term-wise conditional expectation predictors bX·j = bgj (X·,−j), j = 1, 2, . . . , pfit on the observed data X, and assume that bg...
work page 2021
-
[2007]
A power and prediction analysis for knockoffs with lasso statistics
Asaf Weinstein, Rina Barber, and Emmanuel Candes. A power and prediction analysis for knockoffs with lasso statistics. arXiv preprint arXiv:1712.06465 ,
-
[2011]
Adaptive Non-Parametric Regression With the $K$-NN Fused Lasso
Oscar Hernan Madrid Padilla, James Sharpnack, Yanzhen Chen, and Daniela M Wit- ten. Adaptive non-parametric regression with the k-nn fused lasso. arXiv preprint arXiv:1807.11641,
-
[2017]
A power analysis for knockoffs with the lasso coefficient-difference statistic
Asaf Weinstein, Weijie J Su, Ma lgorzata Bogdan, Rina F Barber, and Emmanuel J Candes. A power analysis for knockoffs with the lasso coefficient-difference statistic. arXiv preprint arXiv:2007.15346, 2(7):1–22,
arXiv 2007
-
[2020]
Auto-Encoding Knockoff Generator for FDR Controlled Variable Selection
23 Mason and Fei Ying Liu and Cheng Zheng. Auto-encoding knockoff generator for fdr controlled variable selection. arXiv preprint arXiv:1809.10765 ,
-
[2022]
ISSN 2050-084X. doi: 10.7554/eLife.75244. URL https://doi.org/10.7554/eLife.75244. Kristin K Nicodemus. On the stability and ranking of predictors from random forest variable importance measures. Briefings in bioinformatics , 12(4):369–373,
-
[2023]
URL https://www.ncbi.nlm.nih.gov/books/ NBK538534/. Updated 2023 Mar
work page 2023
Show all 9 references
-
[2024]
Sequential knockoffs for continuous and categorical predictors: with ap- plication to a large psoriatic arthritis clinical trial pool
Matthias Kormaksson, Luke J Kelly, Xuan Zhu, Sibylle Haemmerle, Luminita Pricop, and David Ohlssen. Sequential knockoffs for continuous and categorical predictors: with ap- plication to a large psoriatic arthritis clinical trial pool. arXiv preprint arXiv:2010.14026 ,
2010 arXiv
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.