{"id":"258955f6-c1a9-495d-9711-d90850213a2d","arxiv_id":"2508.03194","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey that categorizes deep reinforcement learning scaling strategies into data, network, and training budget dimensions and outlines challenges for scaling DRL systems.","lead":"This review paper organizes recent deep reinforcement learning scaling research into three categories: data, network, and training budget. It provides a structured taxonomy and a set of open problems intended to guide future work in scalable DRL.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 2.3's Eq. (5) presents an additive, separable multi-resource scaling law as the formal basis for the survey's three-axis taxonomy, but the equation is not derived or cited from the scaling-law literature and conflicts with the coupled N-D-C forms in the cited sources.","rationale":"The reader's weakest assumption identifies the independence and exhaustiveness of the three-axis taxonomy as the key vulnerability. My concern is a sharper version of the same issue: Eq. (5) in Section 2.3 is the only formal statement that encodes the independent-contribution assumption, and it is presented without derivation or citation. The paper's own Section 7.3 acknowledges that interdependencies are poorly understood, which directly contradicts the separable additive form in Eq. (5). If Eq. (5) is removed, the survey still has value as a structured literature review, but the concluding claim about 'principled pathways' loses its formal grounding. This does not change the reader's CONDITIONAL verdict; it reinforces it. I credit the paper for its broad coverage, useful comparison tables, and honest discussion of open problems, but the unverified formal equation and related citation issues justify revision before the survey is treated as a definitive reference.","tokens_in":34076,"tokens_out":4478,"duration_ms":55501,"concrete_test":"Independently reconstruct Eq. (5) from the four cited sources (Kaplan et al. 2020; Henighan et al. 2020; Hoffmann et al. 2022; Marafioti et al. 2025). Specifically: (1) search each paper for a three-term additive law in N, D, and C with independent exponents; (2) fit Eq. (5) to the loss-vs-N,D data in Hoffmann et al. (2022) with C = 6ND, checking whether the εC^{-ζ} term is identifiable and whether the fitted exponents match reported values such as δ ≈ 0.34; (3) check whether Marafioti et al. 2025 contains any scaling-law formula at all. If no source supports Eq. (5), the paper should either replace it with the coupled Chinchilla/Kaplan forms or explicitly label Eq. (5) as the authors' proposed model rather than an established scaling law.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'the systematic application of scaling laws has emerged as a transformative force in DRL' is carried by the formalization in Section 2.3, Eq. (5): P(N,D,C) = αN^{-β} + γD^{-δ} + εC^{-ζ} + L0, described as a 'multi-resource power-law equation.' No source is cited for this specific additive separable form, and the cited scaling-law papers do not contain an independent compute term of this kind. Kaplan et al. (2020) fit L(N,D) ≈ (N_c/N)^{α_N} + (D_c/D)^{α_D} + L0; Hoffmann et al. (2022) fit a coupled loss L(N,D) = E + A/N^α + B/D^β under compute-optimal pairing of N and D, with FLOPs C ≈ 6ND. An additive εC^{-ζ} term is therefore not identifiable and is inconsistent with the data-generation model. In DRL specifically, Section 2.3 concedes that 'very few works systematically analyzed the scaling phenomenon,' and Section 7.3 admits that 'interdependencies are poorly understood.' Thus the equation that licenses the three independent scaling axes is an unsupported assumption rather than an established law. If Eq. (5) is invalid or cannot be sourced, the survey's claim that scaling laws provide principled pathways is substantially weakened, even though the taxonomy may still be useful as an organizing heuristic.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This survey proposes a three-axis taxonomy for scaling deep reinforcement learning (data, network, and training budget), surveys representative methods within each axis, and connects the discussion to scaling laws in supervised learning and to recent RL-based LLM post-training. The central formal claim is Equation (5), an additive separable multi-resource power law P(N,D,C) = αN^{-β} + γD^{-δ} + εC^{-ζ} + L0, and the conclusion asserts that the systematic application of scaling laws is a transformative force in DRL. The paper also identifies open problems, particularly the poorly understood interdependencies among the three scaling dimensions.","tokens_in":34434,"tokens_out":4977,"duration_ms":57914,"significance":"If the formalization in Section 2.3 were sound, the paper would provide a principled framework for organizing the DRL scaling literature and for guiding resource-aware algorithm design. Even without that formalization, the survey is useful as a broad, well-structured map of methods in distributed data collection, synthetic experience replay, network width and depth scaling, ensembles, evolutionary RL, replay ratios, batch sizes, and auxiliary objectives. The inclusion of LLM post-training as a scaling application is a valuable contemporary extension, and the explicit enumeration of open problems in Section 7.3 is a strength. The paper is not an empirical contribution and does not derive new scaling laws; its value is bibliographic synthesis and hypothesis generation.","major_comments":[{"comment":"Equation (5) is presented as the formal basis for the survey's three-axis taxonomy, but it is neither derived nor cited. The cited scaling-law sources do not support an independent additive compute term: Kaplan et al. (2020) fit L(N,D) approximately as (N_c/N)^{α_N} + (D_c/D)^{α_D} + L0, and Hoffmann et al. (2022) fit a coupled form L(N,D) = E + A/N^α + B/D^β under compute-optimal training with C ≈ 6ND. In both cases, compute is not an independent resource whose effect enters additively. The εC^{-ζ} term is therefore not identifiable from the standard data-generation model and is inconsistent with the coupled scaling behavior in the cited literature. Since the conclusion in Section 8 that scaling laws offer 'principled pathways' relies on this equation, the paper must either provide a citation and derivation for Eq. (5) or explicitly demote it to a heuristic illustration, with the central claims adjusted accordingly.","section":"Section 2.3, Eq. (5)"},{"comment":"The paper contains a load-bearing internal inconsistency. Section 2.3 states that the separable formulation of Eq. (5) implies that each resource contributes independently to performance improvement, while Section 7.3 states that 'Most existing works treat data, network, and training budget scaling as independent axes, but their interdependencies are poorly understood.' These two statements cannot both be asserted without qualification. If the three axes are independent by assumption, the survey's taxonomy is a useful organizing device but not a demonstrated scientific claim; if they are interdependent, Eq. (5) is contradicted. The authors should state explicitly that the separability is an assumption to be tested and temper the formal language in Sections 2.3 and 8 accordingly.","section":"Section 7.3 vs. Section 2.3"},{"comment":"There is a concrete citation error in the replay-ratio comparison table. Table 3 lists 'Simba[Nauman et al., 2024]', but the Nauman et al. (2024) reference is BRO, not SimBa; the text in Section 6.2 correctly attributes SimBa to Lee et al. (2024), and Table 1 also lists 'Simba[Lee et al., 2024]'. Since Table 3 is the evidence table for the replay-ratio scaling section, this misattribution must be corrected and the table regenerated to ensure all entries correspond to the correct references.","section":"Table 3 and Section 6.2"},{"comment":"The paper cites Marafioti et al. (2025) as one of the four foundational scaling-law references in the opening of Sections 1 and 2.3. The cited work, SmolVLM, is a small multimodal model recipe, not a scaling-law study. Including it among Kaplan et al. (2020), Henighan et al. (2020), and Hoffmann et al. (2022) is misleading about the established basis of scaling laws and weakens the survey's foundational premise. The citation should be replaced with an actual scaling-law reference or removed.","section":"Sections 1 and 2.3, References"}],"minor_comments":[{"comment":"The text refers to 'The modified RL algorithm in Algorithm 3', but the displayed pseudocode is labeled Algorithm 1; the cross-reference should be corrected.","section":"Section 3, Algorithm 1"},{"comment":"The reference 'Sutton and Barto, 1988' is dated incorrectly: the cited book 'Reinforcement Learning: An Introduction' was published in 1998, not 1988.","section":"References"},{"comment":"The use of '/' in the Critic Ensemble and Policy Ensemble columns is ambiguous; for example, a reader cannot tell whether '/' means 'not applicable' or 'no entry'. Use explicit check marks or written entries such as 'No' and 'Yes' for clarity.","section":"Table 2"},{"comment":"The caption 'APE-X PQN SAPG' lists the methods, but the figure itself does not visually distinguish the three techniques; adding labeled panels or direct annotations would improve readability.","section":"Figure 2"},{"comment":"The phrase 'a novel taxonomy for the first time' is an overclaim; the paper does not establish that no prior scaling taxonomy exists, and Section 7.3 partially undercuts the taxonomy's independence. Suggest rewording to 'a three-axis taxonomy'.","section":"Section 1"},{"comment":"The discussion of LLM RL post-training relies heavily on non-archival industry technical reports and blog posts (OpenAI, Kimi, DeepSeek, Qwen). The paper should state access dates or version identifiers, and ideally separate peer-reviewed results from technical reports.","section":"Section 7.2"}],"recommendation":"major_revision","confidential_remarks":"The survey's breadth is a genuine strength, but the unsupported formalization in Eq. (5) and the citation errors in the foundational references and tables should be corrected before publication. The self-citations (ERL-Re2, EvoRainbow, CHAIN) appear descriptive and appropriate, but the authors may wish to add a brief note on related prior work to avoid any perception of over-weighting their own contributions. The recommendation is major revision because the central formal claim needs rework, but the organizational contribution is defensible and the paper can be fixed within its current scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful organizing survey, not a research contribution. The three-axis taxonomy (data, network, training budget) is a reasonable lens, and the survey gathers a lot of recent DRL scaling work in one place, including connections to LLM RL fine-tuning that most DRL surveys lack. It deserves to be published after revision, but it should not be cited as evidence for a multi-resource scaling law.\n\nWhat it does well: coverage is current and broad; the tables are handy; the discussion of interdependencies in Section 7.3 is honest and correct. The authors clearly know the literature, and their descriptive summaries of individual methods are mostly accurate.\n\nSoft spots, in rough order of importance. First, Eq. (5) in Section 2.3 presents an additive separable multi-resource power law with no derivation or citation. The cited scaling-law papers use coupled forms (Kaplan, Chinchilla), and an independent additive compute term is not what those papers fit. Since the survey's organizing framework treats the three axes as independent, this equation is load-bearing rhetoric; the authors should either source it, derive it, or clearly label it as a heuristic. Section 7.3 actually concedes interdependencies are poorly understood, which partially undermines the independence assumed earlier. Second, there are concrete citation errors: SimBa is attributed to Nauman et al. 2024 in Table 3 and in Section 6.2, but SimBa is Lee et al. 2024. Marafioti et al. 2025 (SmolVLM) is also cited as a scaling-law reference in the introduction and preliminaries, which is questionable since it is a multimodal model paper, not a scaling-law study. These errors are fixable but they reduce trust in the tables. Third, the \"transformative force\" language in Section 8 outruns the evidence: the survey itself says very few DRL works have systematically analyzed scaling laws, so the empirical support is mostly from individual method papers rather than established DRL scaling laws.\n\nNone of this kills the survey. The taxonomy is useful as an organizing heuristic, and the practitioner guidelines in Section 7.1 are sensible. The paper is for people entering the area or for researchers who want a quick map of levers. I would not cite Eq. (5), but I would cite the taxonomy and the tables. Send it to review with a request to fix the citation errors and reframe Eq. (5) as a heuristic, not a law.","headline":"Useful organizing survey with a load-bearing but unsourced scaling-law equation and fixable citation errors; worth reviewing after revision.","tokens_in":34914,"tokens_out":1895,"would_cite":true,"duration_ms":21463,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep reinforcement learning scales predictably when data, network capacity, and training budget are grown together.","keywords":["scaling laws","deep reinforcement learning","data scaling","network scaling","training budget scaling","replay ratio","sample efficiency","survey"],"falsifier":"A factorial study on a fixed benchmark set that varies data volume, network width, and replay ratio independently and fits the additive power-law model would settle the matter: if two-way interactions are large enough that the additive form systematically mispredicts performance, or if a fourth resource such as test-time inference compute produces gains that cannot be absorbed into any of the three axes, the survey's central organizing assumption fails.","tokens_in":33890,"feed_emoji":"📈","tokens_out":7320,"duration_ms":79119,"temperature":0.7,"pith_summary":"This survey argues that the scaling laws driving progress in supervised deep learning can be brought to deep reinforcement learning, and that doing so requires thinking about three levers at once: more and better data, larger and more expressive networks, and smarter allocation of the training budget. It organizes the field into data scaling (parallel collection and synthetic generation), network scaling (width and depth growth, ensembles, mixtures of experts, and agent populations), and training budget scaling (distributed training, replay ratios, batch sizes, and auxiliary tasks). The paper's working hypothesis is that scaling these three dimensions in a coordinated way can relieve DRL's long-standing bottlenecks of sample inefficiency, reward sparsity, and unstable training. A sympathetic reader takes from this survey a roadmap: DRL scaling is not a single knob but a set of interacting levers whose joint design determines whether larger models and datasets actually help.","feed_headline":"Deep RL scales when data, networks, and compute grow together","feed_subtitle":"A survey of deep RL shows scaling gains depend on co-designing data, networks, and training budgets.","key_machinery":"The organizing device is the three-axis taxonomy of scaling (data, network, training budget), together with the additive power-law model $P(N,D,C) = \\alpha N^{-\\beta} + \\gamma D^{-\\delta} + \\epsilon C^{-\\zeta} + L_0$ borrowed from supervised learning. The survey maps each phase of the RL training pipeline onto one of the three axes: environment interaction feeds data scaling, policy and value network initialization and architecture feed network scaling, and the update loop feeds training budget scaling through replay ratios (the number of gradient updates per collected experience), batch sizes, distributed gradient computation, and auxiliary losses. This mapping is what lets the survey treat scale as a design variable in RL rather than a fixed hardware constraint.","core_discovery":"The paper's central claim is that DRL is entering a regime in which performance improves predictably with scale, just as in vision and language, but the mapping is more complex because data, network capacity, and compute interact through the agent's own behavior. The proposed taxonomy has three axes, and the survey maps representative algorithms onto it: data scaling via parallel sampling and generative augmentation, network scaling via monolithic expansion, ensembles and mixtures of experts, and agent-number scaling, and training budget scaling via distributed training, replay ratio, batch size, and auxiliary objectives. The evidence is read as showing that width scaling with normalization is the most reliable architectural lever, that high replay ratios work when paired with bias suppression and plasticity preservation, and that synthetic data can substitute for expensive real interactions when generation is filtered by value or dynamics consistency. The paper does not claim to have measured a single universal scaling law for RL; it claims that the components of such a law are now visible across the literature and that their interdependencies are the main open problem.","pith_inferences":["I would read the survey as evidence that the field needs a compute-efficient frontier for RL analogous to loss-versus-FLOP curves in language modeling, where the x-axis is total compute and the y-axis is policy return; the paper stops short of proposing such a metric, but its synthesis makes it the natural next step.","A testable extension suggested by the paper's open problems is a factorial study varying data volume, network width, and replay ratio together on a fixed benchmark suite to measure interaction terms; if those interactions are large, the additive power-law model needs a coupled or multiplicative term.","The survey's discussion of LLM post-training hints that test-time scaling, including longer rollouts and multi-agent verification, may be a fourth dimension that the three-axis taxonomy does not fully absorb; if test-time compute keeps improving results independently of training-time scaling, the taxonomy may need revision."],"forward_implications":["If the survey's synthesis is correct, the default recipe for pushing DRL performance is not just bigger models but the joint scaling of data throughput and update-to-data ratio, with architectural normalization to keep training stable.","High replay ratios cease to be a trick and become a first-class scaling lever, but only when coupled with ensembles, normalization, or periodic resets that control primacy bias and plasticity loss.","Synthetic data becomes a legitimate substitute for real environment interactions in domains like robotics and offline RL, provided generation is filtered by value or dynamics consistency.","The same scaling logic already operates in LLM post-training, where model size, data quality, rollout length, batch size, and test-time compute are being scaled in coordination.","Scaling choices cannot be made axis by axis; the paper predicts that ignoring cross-axis interactions, such as critic width versus replay ratio, will lead to overfitting or wasted compute."],"supporting_citations":[{"why":"Supplies the neural scaling law formalism that the survey extends to reinforcement learning.","marker":"[Kaplan et al., 2020]"},{"why":"Establishes compute-optimal scaling and the data-scaling exponent in large language models, the template for DRL budget analysis.","marker":"[Hoffmann et al., 2022]"},{"why":"Documents scaling laws across autoregressive and vision models, giving the paper its cross-domain evidence that scaling generalizes.","marker":"[Henighan et al., 2020]"},{"why":"Extends neural scaling laws to single-agent RL through the concept of intrinsic performance, the direct precedent for RL scaling.","marker":"[Hilton et al., 2023]"},{"why":"Shows that value-based DRL scales predictably with hyperparameters and budget allocation, the empirical anchor for training-budget scaling.","marker":"[Rybkin et al., 2025]"},{"why":"Provides the canonical decoupled actor-learner architecture that the data scaling section builds on.","marker":"[Horgan et al., 2018]"},{"why":"REDQ supplies the main evidence that ensemble critics unlock high replay ratios.","marker":"[Chen et al., 2021]"},{"why":"Shows that regularized critic scaling can outperform pure replay ratio scaling and that the two dimensions interact.","marker":"[Nauman et al., 2024]"},{"why":"Demonstrates that mixture-of-experts modules enable parameter scaling in value-based DRL, a key network scaling result.","marker":"[Obando-Ceron et al., 2024]"}],"fun_headline_variants":["Deep RL scales when data, nets, and budgets align","Three-way scaling: data, network, and budget for DRL","Survey maps scaling levers for deep RL across three axes","Co-design data, networks, and compute to scale deep RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey's three-way split of scaling into data, network, and training budget is assumed to be the right and complete way to organize the field, even though the paper admits these axes interact in ways that are poorly understood.","fun_headline_variants_meta":{"raw":{"variants":["Deep RL scales when data, nets, and budgets align","Three-way scaling: data, network, and budget for DRL","Survey maps scaling levers for deep RL across three axes","Co-design data, networks, and compute to scale deep RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000353,"raw_usage":{"total_tokens":1965,"prompt_tokens":1035,"completion_tokens":930,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":860}},"tokens_in":651,"tokens_out":930,"duration_ms":9877,"temperature":1.0,"reasoning_tokens":860,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T04:34:47.404863+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A factorial study on a fixed benchmark set that varies data volume, network width, and replay ratio independently and fits the additive power-law model would settle the matter: if two-way interactions are large enough that the additive form systematically mispredicts performance, or if a fourth resource such as test-time inference compute produces gains that cannot be absorbed into any of the three axes, the survey's central organizing assumption fails.","supporting_citations":[],"review_version":1}