Pith. sign in

REVIEW 4 major objections 5 minor 20 references

Fixed-budget RL post-training is a resource-allocation problem: the best split of compute among model size, search, learning, and feedback changes with each of those choices.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:18 UTC pith:IW76GTUG

load-bearing objection Useful framing and honest reporting, but the quantitative frontiers hinge on an unvalidated constant a=0.85 and need artifacts before the numbers can be trusted. the 4 major comments →

arxiv 2607.13389 v1 pith:IW76GTUG submitted 2026-07-15 cs.LG cs.CL

Where Should RL Post-Training Compute Go? Model Size, Search, Learning, and Feedback

Patrick Wilhelm , Odej Kao This is my paper
classification cs.LG cs.CL
keywords reinforcement learning post-trainingcompute allocationFLOP accountingGRPOLoRAreward designprocess reward modelsallocation frontiers
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that reporting a single total FLOP budget for RL post-training is under-specified. It introduces a FLOP-accounting framework that splits post-training compute into rollout/search, policy-update/learning, and reward/feedback evaluation, and defines the update fraction rho as the central allocation variable. Across LoRA-adapted Qwen2.5 policies trained with GRPO on math reasoning, the best observed allocation varies with model size, compute budget, reward system, and evaluation target: rule-based rewards favor high-update regimes while PRM-style feedback shifts toward lower update fractions because reward-model inference consumes budget; downstream accuracy and process-quality judges often select different regimes than native reward. The paper proposes RACE, a diagnostic pilot-grid protocol to identify allocation regimes before expensive validation, explicitly not claiming held-out improvement guarantees. A sympathetic reader should care because if the claim is right, post-training papers and practitioners must report and tune the distribution of compute, not just the total.

Core claim

The central discovery is that RL post-training under a fixed budget exhibits conditional allocation frontiers: the best observed balance between search, learning, and reward evaluation is not a single point but a surface that shifts with model size, budget band, reward system, and evaluation metric. Using C_total = C_search + C_learning + C_reward and rho = C_learning / C_total, the paper shows same-FLOP comparisons are mechanistically meaningful only when the allocation is held fixed; larger policies consume more per-token compute and therefore buy fewer updates or rollouts, coupling model choice to allocation. It documents that rule-based rewards spend almost all non-update compute on roll

What carries the argument

The load-bearing object is the FLOP-accounting identity C_total = C_search + C_learning + C_reward and the derived update fraction rho = C_learning / C_total, which turns 'where should compute go?' into a measurable allocation variable. Rollout compute is parameter-token autoregressive cost, update compute uses an effective update fraction feff = a + (1-a)*fLoRA with a = 0.85 to fold frozen-backbone forward/activation costs into LoRA updates, and reward compute is zero for rule-based rewards but substantial for a 7B process reward model. The RACE protocol fits a quadratic response model in log rho over a small IsoFLOP pilot grid and recommends validation allocations, falling back to pilot-be

Load-bearing premise

The entire matched-FLOP comparison rests on a fixed constant a = 0.85 in the effective update fraction feff = a + (1-a)*fLoRA and on a linear per-token scaling for step schedules; if either is inaccurate, the same-FLOP bands are not truly matched and the reported frontiers could shift.

What would settle it

Measure the actual FLOPs of a LoRA update step (forward, backward, activations) across model sizes and ranks and compare with the assumed feff = a + (1-a)*fLoRA with a = 0.85; if feff deviates systematically, the IsoFLOP bands are mismatched. Alternatively, a single held-out run at a different allocation that beats the claimed best observed frontier under a correctly recomputed budget would break the central claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Reporting only total post-training FLOPs is under-specified; allocation numbers (rollout, update, reward FLOPs and rho) must accompany totals for meaningful comparisons.
  • Model size and training allocation are coupled: the same FLOP budget buys fewer updates and rollouts for larger policies, so pooled allocation prescriptions are misleading.
  • Reward systems change the accounting: rule-based rewards are rollout-heavy, PRM-style feedback spends a visible share on reward-model inference, shifting the best observed rho.
  • Evaluation targets disagree: native reward, downstream pass@1, and common process-judge scores can each select a different allocation regime, so allocation decisions must name their target.
  • Small pilot grids (RACE) can identify high- versus lower-update regimes with perfect same-regime rates in validation, though they do not guarantee held-out reward improvement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If allocation is the right frame, scaling-law analyses for RL post-training should be revised to model the three-term decomposition rather than total compute; a testable extension is whether the allocation-response surface generalizes beyond math reasoning to other post-training tasks.
  • The fixed constant a = 0.85 in the effective update fraction is a natural first falsification target: measuring actual LoRA update costs across model sizes and ranks would confirm whether matched-FLOP bands are truly matched.
  • The framework transfers to robot-learning pipelines as a budget-planning tool: before paying for expensive evaluation, one could run a RACE-style pilot to decide whether to buy a larger planner, more rollouts, or a stronger feedback model.
  • Reward-model inference cost may force a sweet spot: stronger feedback can degrade downstream pass@1 if it crowds out rollout exploration; that tension might be testable by holding rho fixed and varying reward quality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies fixed-budget RL post-training as a resource-allocation problem over model size, rollout/search, policy updates, and reward-feedback compute. It introduces a FLOP-accounting decomposition for GRPO with LoRA adapters, formalizes the update fraction rho, and empirically sweeps allocations across Qwen2.5 1.5B/3B/7B policies on Polaris-53K, comparing sparse/structured/dense/proxy-PRM/real-PRM rewards. The central claim is that the best observed allocation is conditional on model size, compute budget, reward system, and evaluation target, so reporting only total FLOPs is insufficient. The paper also proposes RACE, a pilot-grid diagnostic for identifying allocation regimes, while explicitly disclaiming that it guarantees held-out improvement.

Significance. If the allocation-dependence result holds, it is a useful corrective to the common practice of summarizing RL post-training by a single FLOP budget. The paper makes a concrete, falsifiable claim: two runs with matched total FLOPs can differ in performance because of how compute is split among search, learning, and reward evaluation, and the best split is not universal. The strengths are the explicit FLOP decomposition, the systematic multi-stage experimental design, and the honest treatment of RACE as a diagnostic rather than an optimizer. However, the evidential weight of the empirical claims is currently weakened by an uncalibrated constant in the compute model, the absence of any uncertainty quantification, and the small downstream evaluation sets.

major comments (4)
  1. [Appendix B, Eq. (B.2)] The entire same-FLOP comparison and every rho value depend on the effective update fraction feff = a + (1-a)*fLoRA with a fixed a = 0.85. No calibration, measurement, or sensitivity analysis is reported for a. If a varies across model sizes, LoRA ranks, or reward systems, the IsoFLOP bands are not matched and RQ1/RQ2 comparisons conflate allocation with total-compute differences. Please add a sensitivity analysis over a plausible range (e.g., 0.5-0.95) and, if possible, measure a on at least one configuration, or explicitly restrict the claims to the assumed accounting.
  2. [Table 1, Section 5 (RQ3)] Downstream pass@1 accuracy is computed on only 250 examples per dataset, so the reported differences between the best settings (e.g., 0.698 vs 0.714 for structured vs real PRM at a180) are within one or two standard errors. No confidence intervals, error bars, or seed variation are reported anywhere in the paper. The claim that evaluation targets select different allocations needs uncertainty quantification; otherwise the observed target dependence may be sampling noise.
  3. [Appendix G, Section 5 (RQ4)] Appendix G states that on the Stage A pilot grid the constant baseline obtains the lowest average LOOCV RMSE and that the quadratic log-rho model is not statistically preferred. Yet RQ4 claims that RACE 'recovers the correct high- versus lower-update regime' with same-regime rates of 1.0. These two statements are in tension: if the fitted response model is not better than a constant mean, the regime recovery may be an artifact of the coarse grid rather than evidence that RACE diagnoses a real allocation response. Please clarify what evidence supports regime recovery and temper the RQ4 conclusion accordingly.
  4. [Appendix B, Eq. (B.1)] The decomposition in Eq. (1) and the narrative distinguish F_reward as a separate compute channel, but Appendix B defines F_rollout = c_fwd(N+R)*T_rollout, placing reward-model forward passes inside rollout compute, and never defines F_reward. This makes it unclear how the 'visible fraction to reward-model evaluation' in Figure 3 is computed and whether the denominator in rho double-counts reward compute. Please give the explicit formula for F_reward and reconcile it with F_rollout.
minor comments (5)
  1. [References] Reference [20] lists 'A. Authors' and 'OpenReview, 2025'; a complete citation is needed.
  2. [Appendix A.2] The step schedule S(N) = round(S_1.5B * 1.5/NB) is stated without empirical justification. A line on why this linear-inverse scaling is used would help.
  3. [Figure 2] The right panel labels rho values in a way that is hard to read; consider a table or clearer callouts.
  4. [Section 7] The limitations say 'limited seeds' but no seed count is given. Please specify the number of seeds and which results are seed-averaged.
  5. [Appendix F] The same-regime rate of 1.0 is reported without a confusion matrix or per-model-size breakdown; a small table would strengthen the diagnostic claim.

Circularity Check

1 steps flagged

Main empirical claims are self-contained; minor in-sample circularity in the RACE diagnostic.

specific steps
  1. fitted input called prediction [Section 5 RQ4; Appendix E (RACE); Appendix G]
    "RACE fits a local response model ˆR_N,R,C(ρ) over the update fraction ρ. ... RACE uses this fit only as a decision rule. If the fitted best-response point is stable and lies inside the observed range, RACE recommends validation runs near the predicted point. ... RACE recovers the correct high- versus lower-update regime in inside-grid and leave-rank-out validation, with same-regime rates of 1.0 in both settings."

    The inside-grid component evaluates the fitted quadratic on the same pilot grid used to fit it; the curve is constructed from those rewards, so its argmax regime is a restatement of the input data rather than an independent confirmation. Appendix G adds that the constant baseline has the lowest average LOOCV RMSE on this grid, so the allocation-dependent model is not even preferred. This makes the in-sample 'recovery' evidence self-referential. The leave-rank-out part is a genuine holdout prediction, and the paper explicitly disclaims held-out improvement, so the circularity is confined to the diagnostic's in-sample validation.

full rationale

The paper's central claims RQ1-RQ3 rest on direct experiments: runs with logged token counts, FLOP estimates, and held-out evaluations. No central 'prediction' is derived from a fitted parameter; the allocation frontiers are descriptive observations from the grid. The FLOP accounting uses a hand-set constant a=0.85 and a linear step schedule S(N)=round(S_1.5B*1.5/NB); these are uncalibrated assumptions that affect whether same-FLOP bands are truly matched, but they are not circular because they are not fitted to the reward outcomes being predicted. The only true self-referential element is RACE's inside-grid regime recovery: the quadratic response model is fit on a pilot grid and then 'recovers' the regime on that same grid. Appendix G explicitly acknowledges that a constant baseline has lower LOOCV RMSE, weakening the diagnostic's statistical support. Since RACE is presented as a diagnostic rather than a guarantee and is not the basis of the main allocation-frontier conclusions, the overall circularity is minor.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The central claim rests on an operational FLOP-accounting model with a hand-set effective-update constant (a=0.85), hand-designed step schedules, and unspecified reward hyperparameters. The empirical conclusions are data-driven rather than derived, and no new physical or architectural entities are introduced.

free parameters (5)
  • a_eff = 0.85
    Effective update-side frozen-backbone cost fraction in feff = a + (1-a)*fLoRA (Appendix B). A single hand-set constant controls every rho value and all IsoFLOP bands; no sensitivity analysis is given.
  • alpha_proxy_prm = unspecified
    Weight in r_proxyPRM = alpha*r_proc + (1-alpha)*r_out (Appendix D). The value is not reported, so the proxy PRM reward is not exactly reproducible.
  • lambda_fmt = unspecified
    Formatting weight in the structured reward (Appendix D); value not reported.
  • lambda_shape = unspecified
    Answer-shape weight in the structured reward (Appendix D); value not reported.
  • base_step_schedules = 65/62/59/53/46; 105/100/95/85/74; 140/133/127/113/99 (r=8..128, bands C1p1e16/C1p8e16/C2p4e16)
    Hand-set step schedules for the 1.5B reference model that define the IsoFLOP bands; other sizes are derived by the 1.5/NB scaling rule. These choices determine which runs count as 'same-FLOP'.
axioms (6)
  • domain assumption Per-token FLOP cost scales linearly with backbone parameter count, so update steps can be matched across model sizes by S(N) = round(S_1.5B * 1.5/NB).
    Used in Section A.2 to construct IsoFLOP bands; if this scaling is inaccurate (e.g., due to attention or overhead), the budgets are not actually matched.
  • ad hoc to paper LoRA update compute is captured by feff = a + (1-a)*fLoRA with a constant a = 0.85.
    Appendix B. The constant is hand-set and no sensitivity analysis is provided; it is load-bearing for all rho values.
  • domain assumption GRPO with K=2 and L=2048 on Polaris-53K math reasoning is a representative proxy for RL post-training of reasoning and feedback modules.
    This is the controlled setting used throughout; the paper acknowledges in Limitations that it is not direct embodied robot learning.
  • domain assumption Downstream pass@1 on 250-example GSM8K and MATH-500 subsets provides reliable held-out signal.
    Sections 4 and 5 use 250 examples per dataset with no confidence intervals, so small differences may be noise.
  • domain assumption Including reward-model inference FLOPs in C_reward is the correct accounting for PRM-style feedback.
    Core to the framework's 'reward compute matters' message; reasonable, but the accounting ignores hardware-dependent efficiency differences.
  • domain assumption Within-reward min-max normalization preserves meaningful ordering for cross-reward comparisons.
    Appendix C; it assumes native reward scales are not directly comparable and that relative ordering within each condition is the useful signal.

pith-pipeline@v1.3.0-alltime-deepseek · 3963 in / 4216 out tokens · 136325 ms · 2026-08-02T05:18:22.184090+00:00 · methodology

0 comments
read the original abstract

Reinforcement Learning (RL) post-training is increasingly used to adapt foundation models for reasoning, planning, and feedback-driven robot-learning pipelines, but constrained post-training resources are often summarized by a single total FLOP budget. We study the fixed-budget decision problem behind this practice: under the same post-training budget, should one use a larger policy, train a smaller policy longer, generate more rollout search, or spend compute on stronger reward feedback? We introduce a FLOP-accounting framework for GRPO post-training that decomposes compute into rollout/search, policy-update/learning, and reward- or feedback-model evaluation. Across LoRA-adapted Qwen2.5 policies, we find conditional allocation frontiers: the best observed allocation changes with model size, compute budget, reward system, and evaluation target. Same-FLOP model-size comparisons show that model choice and training allocation are coupled because larger policies consume more per-token compute and therefore buy fewer updates or rollouts under the same budget. Reward systems also change the accounting: rule-based rewards spend nearly all non-update compute on policy rollouts, while PRM-style feedback allocates a visible part of the budget to reward-model inference. We present RACE as a diagnostic pilot-grid protocol, not a guarantee of held-out improvement, for identifying allocation regimes before expensive validation runs; our results suggest that RL post-training papers should report total FLOPs together with how compute is divided among model size, search, learning, and feedback.

Figures

Figures reproduced from arXiv: 2607.13389 by Odej Kao, Patrick Wilhelm.

Figure 1
Figure 1. Figure 1: Search–learning allocation plane. Under a fixed RL post-training budget, runs differ in how much compute is spent on rollout search, policy updates, and reward evaluation. The best observed allocation is a point on this conditional resource-allocation plane, not a function of total compute alone. This result is the main empirical reason to avoid pooled allocation prescriptions. A smaller policy trained lon… view at source ↗
Figure 2
Figure 2. Figure 2: Same-FLOP model-size trade-off. Under matched pilot budgets, changing policy size changes how much training the budget buys. Larger policies consume more per-token compute, so the same FLOP budget can imply fewer rollout tokens and fewer update steps; the best observed downstream run therefore depends jointly on model size and allocation. Sparse Structured Dense Proxy PRM Real PRM 0.4 0.5 0.6 0.7 0.8 B est… view at source ↗
Figure 3
Figure 3. Figure 3: Feedback cost changes allocation. For best observed native-reward runs in the matched 3B setting, rule-based rewards spend the budget on policy rollout and update, while PRM-style feedback allocates a visible fraction to reward-model evaluation. This is the mechanism hidden by reporting only total FLOPs. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Evaluation target changes allocation. Native reward, downstream pass@1 symbolic accuracy, and common process-quality scoring can select different allocation regimes. The answer to “where should compute go?” therefore depends on the target being measured. RQ4: Can RACE diagnose allocation regimes from small pilot grids? RACE recovers the correct high- versus lower-update regime in inside-grid and leave-rank… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 11 linked inside Pith

  1. [1]

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022

    Pith/arXiv arXiv 2022

  2. [2]

    Huang, F

    W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y . Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. InProceedings of The 6th Conference on Robot Learning, pages 1769–1782, 2023

    2023

  3. [3]

    Driess, F

    D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

    Pith/arXiv arXiv 2022

  5. [5]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. InProceedings of The 7th Conference on Robot Learning, 2023

    2023

  6. [6]

    Open x-embodiment: Robotic learning datasets and rt-x models

    Open X-Embodiment Collaboration. Open x-embodiment: Robotic learning datasets and rt-x models. InIEEE International Conference on Robotics and Automation, 2024

    2024

  7. [7]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  9. [9]

    Hoffmann, S

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    Pith/arXiv arXiv 2022

  10. [10]

    Khatri, L

    D. Khatri, L. Madaan, R. Tiwari, R. Bansal, S. S. Duvvuri, M. Zaheer, I. S. Dhillon, D. Brand- fonbrener, and R. Agarwal. The art of scaling reinforcement learning compute for llms.arXiv preprint arXiv:2510.13786, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Tan et al

    Z. Tan et al. Scaling behaviors of llm reinforcement learning post-training: An empirical study in mathematical reasoning.arXiv preprint arXiv:2509.25300, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. V . Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems, 2022

    2022

  13. [13]

    X. Wang, J. Wei, D. Schuurmans, Q. V . Le, E. H. Chi, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. InInternational Conference on Learning Representations, 2023

    2023

  14. [14]

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y . Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.arXiv preprint arXiv:2305.10601, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Huang, P

    W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. InProceedings of the 39th International Conference on Machine Learning, pages 9118–9147, 2022. 9

    2022

  16. [16]

    Liang, W

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control.arXiv preprint arXiv:2209.07753, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. InAdvances in Neural Information Processing Systems, 2017

    2017

  18. [18]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. InAdvances in Neural Information Processing Systems, 2022

    2022

  19. [19]

    Y . Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Pith/arXiv arXiv 2022

  20. [20]

    A. Authors. Scaling laws for generative reward models. InOpenReview, 2025. A Experimental Details A.1 Overview of Experimental Stages We organize the experiments into three main empirical stages. Stage A studies the core search– learning allocation frontier under an IsoFLOP design. Stage B extends the same compute-allocation protocol to compare reward sys...

    2025