Pith. sign in

REVIEW 2 major objections 4 minor 24 references

Disabling z-score normalization makes binary-reward ES with N=2 improve on capable 0.5B–7B models, where the normalized variant collapses.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:10 UTC pith:2ESKPHPY

load-bearing objection A neat closed-form explanation for the ES population puzzle, but the normalization attribution needs mid-training q re-probing. the 2 major comments →

arxiv 2607.19408 v1 pith:2ESKPHPY submitted 2026-07-08 cs.LG

Reward-Aware Population Scaling of Evolutionary Strategies in LLM Fine-Tuning

Sung Cho , Gyubin Han This is my paper
classification cs.LG
keywords evolutionary strategiespopulation sizeLLM fine-tuningbinary rewardadvantage normalizationzero-advantage probabilityavailability thresholdreward granularity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to resolve a sharp contradiction in population-size findings for evolutionary-strategy fine-tuning of LLMs: one antithetic pair is enough with a dense log-loss reward, while binary accuracy reward reportedly needs about thirty pairs. It argues that the discrepancy is a scaling law set by reward granularity and normalization, not an intrinsic population limit. It derives a closed-form probability q for a perturbation pair yielding exactly zero advantage under binary reward, and shows the availability threshold N_avail is small for capable instruction-tuned models. The core experimental claim is that with population size two, z-score advantage normalization erases the absolute size of the advantage gap (exactly so at N=2), causing collapse; disabling normalization recovers improvement on two benchmarks across model sizes 0.5B–7B. If true, the reported N≈30 requirement in this regime is an implementation artifact, not a fundamental constraint.

Core claim

The central discovery is that z-score advantage normalization, not population size, causes a class of N=2 failures in binary-reward ES. The paper proves that normalizing two pair-level advantages maps them to a fixed sign vector, (±1,∓1) or (±1/√2,∓1/√2), so the absolute advantage gap vanishes entirely: a tiny advantage and a large advantage produce the same update magnitude. Under binary reward, where most advantages are exactly zero and survivors can be very small near degeneracy, raw ES self-anneals (the update shrinks as signal weakens), while normalization promotes the one or two surviving non-zero seeds to fixed-size updates and estimates the scale from a handful of sparse values. Empi

What carries the argument

The zero-advantage probability q = P(A=0), approximated in closed form for binary accuracy reward as q ≈ (4πB·p0(1−p0)(1−ρ))^{-1/2}, where p0 is base accuracy, B the batch size, and ρ the intra-pair per-example correctness correlation. Through K_N = #{i: A_i≠0}, it sets the availability threshold N_avail = ⌈log δ / log q⌉, the smallest population with at least one non-degenerate perturbation direction at confidence 1−δ. The second load-bearing object is Proposition 4: two-sample z-score normalization at N=2 collapses any two advantages to a sign-only vector, erasing absolute reward scale and removing the self-annealing property of raw advantages. Together they separate when small N is at ris

Load-bearing premise

The availability probability q is estimated only from the initial checkpoint (θ0); if q grows as training changes base accuracy and correlation, then availability could become the binding constraint during training, and the paper's claim that normalization—not population size—causes the N=2 collapse would be unproven.

What would settle it

Run the same zero-training degeneracy probe at the initial checkpoint and again after, say, 50 and 100 training steps of a normalized-off N=2 binary-reward run on a 1.5B instruction-tuned model on a math benchmark with batch size 16. If q rises above ≈0.27 (so P(K_N=0)=q² exceeds ≈0.07) while the normalized-on run still collapses, the availability explanation can no longer be excluded at N=2, and the normalization attribution fails. If q stays below ~0.2 throughout, the collapse is confirmed to be about normalization.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A zero-training probe costing about 6,400 forward passes (K=200, B=16) estimates N_avail and can set a population floor before training, avoiding an empirical sweep over N.
  • Cross-entropy reward has zero probability of zero advantage (q=0), so N_avail=1 and the cumulative drift and covariance at fixed perturbation budget are N-independent in the local linearization.
  • With normalization enabled, N=2 binary-reward ES collapses in every tested model/task cell (0.5B–7B on two benchmarks); with normalization disabled, the same setup improves wherever the base model is already capable at the task.
  • The paper argues the previously reported N≈30 floor for binary-reward ES is consistent with a combination of availability regime and advantage normalization, and does not assert which dominates there.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper measures q only at the initial checkpoint; if q grows as base accuracy and correlation evolve during training, N_avail could become binding mid-run, and the normalization attribution would weaken. A direct test is re-probing q at intermediate checkpoints in a normalized-off N=2 run.
  • The scale-erasure mechanism is not specific to ES: any algorithm that z-scores sparse advantages from a handful of samples would inherit the same failure mode, so first-order policy-gradient methods with tiny batches and terminal binary rewards may show analogous instability.
  • Because q grows as p0 approaches 0 or 1 (very weak or very strong base accuracy) and as correlation rises, the success of N=2 with raw advantages is inherently capability-dependent; the formula gives a testable boundary where a weak model should still need N>2 even without normalization.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper studies population-size requirements for evolutionary-strategy fine-tuning of LLMs. It distinguishes reward-induced availability limits from implementation choices. For binary accuracy rewards it derives a closed-form zero-advantage probability q ≈ (4πB p0(1-p0)(1-ρ))^{-1/2} (Eq. 4) and defines an availability threshold N_avail = ceil(log δ / log q). It validates q against a forward-pass probe at the initial checkpoint on Qwen2.5-Instruct/GSM8K (MAE 0.020 over 12 configurations). It then argues that z-score advantage normalization erases reward-scale information at N=2 (Proposition 4) and that disabling normalization lets binary-reward ES with N=2 improve on GSM8K/TREC across model sizes, while the normalized variant collapses. The paper concludes that small-population failure in capable-model binary ES can be an implementation artifact rather than an intrinsic population limit.

Significance. If the central attribution is correct, the paper reconciles the conflicting N=1 CE-reward result and N≈30 binary-reward result and gives a practical probe for choosing N. The derivation of Eq. (4) is clean under the stated homogeneous-batch, small-perturbation approximation, and Proposition 4 is a crisp, exact statement about two-sample z-scoring. The empirical design is also a strength: a forward-pass-only probe and a fixed-everything normalization ablation are directly falsifiable. However, the validation of Eq. (4) is weaker than reported, and the attribution to normalization rests on a q measurement at a single checkpoint. These are correctable with additional probing and uncertainty reporting.

major comments (2)
  1. [Section 4 and Appendix E (Table 1)] The claim that 'pure availability cannot explain the N=2 collapse' uses P(K_N=0)≈0.036 computed from q estimated only at the initial checkpoint θ0 (Appendix E protocol). Eq. (4) makes q depend on p0(1-p0)(1-ρ), all of which can evolve as ES updates move θ. If the normalized run drifts into a region with larger q (e.g., accuracy saturates or ρ→1), availability could become binding during training, and the Figure 1 on/off difference would not uniquely implicate normalization. The manuscript never re-probes q during or after the Figure 1 runs, so the central 'implementation artifact' conclusion is not fully established. Please re-run the probe at multiple checkpoints along both the normalized-on and normalized-off trajectories and report q(θ_t), or otherwise bound q over the training path.
  2. [Section 3, Table 1, Eq. (4)] The validation is underpowered. With K=200, the per-cell binomial standard error is about sqrt(q(1-q)/200) ≈ 0.03, so the reported MAE of 0.020 across 12 cells is within sampling noise; moreover, N_avail computed from predicted q disagrees with the empirical-q N_avail on 3 of 12 rows. Thus 'matches the formula with MAE 0.020' overstates the evidence for Eq. (4). Please report per-cell confidence intervals (or increase K substantially), and reconcile the N_avail discrepancies before using predicted q as a quantitative threshold.
minor comments (4)
  1. [Abstract and Section 4] The abstract says binary-reward ES with N=2 improves on GSM8K and TREC across 0.5B–7B, but Section 4 states that on 0.5B/GSM8K raw ES only avoids collapse without improving. Please qualify the abstract to match the 'avoids collapse vs improves' distinction.
  2. [Notation] K is overloaded: K denotes the number of probe pairs in Appendix E, the number of training forward passes in Figure 1, and K_N is the availability count. This makes equations such as Eq. (3) and the figure axis confusing. Use distinct symbols.
  3. [Figure 1 / Appendix E] No dataset sample sizes, task split details, or number of evaluation batches are given for GSM8K/TREC. Please add these for reproducibility, and state whether the probe uses a fixed batch that is also used for training or a fresh batch.
  4. [References] Reference [13] has a corrupted author list ('Qwen, :, An Yang, ...'); please fix. Also, Appendix H cites Liang et al. [7] as the source of β≈2, r≈100, but notes these are not verified for Qwen2.5/GSM8K; this caveat should be in the main text if spectral stability is advertised.

Circularity Check

0 steps flagged

No significant circularity identified; the main derivation is self-contained and the normalization ablation is an independent controlled experiment.

full rationale

The paper's central derivation chain does not reduce to its own inputs. Equation (4), q ≈ [4πB p0(1−p0)(1−ρ)]^(−1/2), is an analytic approximation derived from a local-CLT argument; the parameters p0 and ρ are measured quantities, and the probe compares this closed-form expression against the empirical zero-advantage rate. No constant is fitted to the q curve, so the MAE 0.020 agreement is a genuine test of the functional form rather than a tautology. Equation (3), P(K_N=0)=q^N and N_avail = ⌈log δ / log q⌉, is definitional probability, not a fitted prediction. The attribution that normalization—not population size—causes N=2 collapse is supported by Figure 1's fixed-everything ablation, which varies only whether z-score normalization is applied; this is a controlled experiment, not a quantity derived from the probe. Proposition 4 is a direct algebraic fact about two-sample z-scoring. The paper's citations of prior work [7,12] are from different research groups and are used for context or comparison, not as load-bearing proof of the present claims; the authors explicitly disclaim asserting which mechanism dominates in Qiu et al.'s setting. The valid scientific weakness—that q is probed only at θ0 and not re-probed during training—is a limitation on the strength of the causal attribution, but it is not circular reasoning: the ablation and the probe are independent, and the conclusion is not forced by definition or by a self-citation chain. Accordingly, no circular step is present.

Axiom & Free-Parameter Ledger

6 free parameters · 7 axioms · 0 invented entities

The paper's theoretical contribution rests on standard CLT/lattice facts and three domain approximations (homogeneous batch, iid seeds, analytic CE); its empirical claims rest on measured inputs p_0 and rho rather than fitted constants. Free parameters are limited to hand-chosen hyperparameters (sigma, eta, B, delta, K) and two borrowed spectral parameters (beta, r) that the authors flag as unverified for their models. No new physical or mathematical entities are postulated; N_avail is an operational diagnostic index (Eq. 3), not an asserted entity, so there is no 'graviton problem.'

free parameters (6)
  • perturbation scale sigma = 1e-3
    Chosen by hand; all probe and training results use sigma=1e-3 with no sensitivity sweep.
  • learning rate eta = 1e-3
    Chosen by hand; the normalize-on collapse vs normalize-off recovery comparison is reported only at eta=1e-3.
  • batch size B = 16 (training), 4-32 (probe)
    Training ablation fixed at B=16; probe covers B in {4,8,16,32}.
  • probe confidence delta = 0.05
    Chosen by hand; sets N_avail via ceil(log delta / log q).
  • probe pair count K = 200
    Chosen by hand; sets sampling noise (~0.03 binomial standard error for q in 0.2-0.5).
  • spectral decay parameters beta, r = beta=2, r=100
    Borrowed from Liang et al. [7]; Appendix H admits they are not verified for Qwen2.5/GSM8K; sensitivity table shows factor-4 variation in L_true.
axioms (7)
  • domain assumption Homogeneous-batch approximation: all B examples share the same base accuracy p_0 and intra-pair correlation rho
    Invoked in Appendix D to derive Eq. (4); the empirical probe is preferred when this is doubtful (Appendix E).
  • standard math Validity of the local CLT / Edgeworth expansion for S_B = sum Y_j with the O(B^-1) fourth-cumulant leading term
    Appendix D uses Gnedenko/Petrov local CLT and the vanishing kappa_3 to claim q ~ (4*pi*B*p0*(1-p0)*(1-rho))^(-1/2) + O(B^-1).
  • domain assumption IID seeds: K_N = sum 1[A_i != 0] is Binomial(N, 1-q)
    Used in Eq. (3) for N_avail(delta) = ceil(log delta / log q).
  • domain assumption CE reward R_CE is real analytic in theta, so its antithetic difference is zero with probability zero
    Corollary 2 relies on analyticity of CE under 'standard transformer architectures (GELU/SwiGLU)'.
  • domain assumption Local linearization: theta stays near its reference point so grad f_sigma and Sigma are constant across T steps
    Proposition 1's drift/diffusion N-independence is stated under this approximation; the authors explicitly decline a descent-level claim (Discussion, caveat c).
  • domain assumption The zero-training probe at theta_0 characterizes q during training
    Section 3/Appendix E probe q only at initialization, then Section 4 uses P(K_N=0) ~ 0.036 to rule out availability as the N=2 failure cause.
  • domain assumption Spectral decay lambda_k ~ k^-beta with beta=2 and effective rank r=100 applies to Qwen2.5/GSM8K
    Appendix H acknowledges these values come from Liang et al. [7] and are not verified for the models in this paper.

pith-pipeline@v1.3.0-alltime-deepseek · 12322 in / 22134 out tokens · 202923 ms · 2026-08-02T08:10:49.465876+00:00 · methodology

0 comments
read the original abstract

Using Evolutionary Strategies (ES) for fine-tuning large language models is attractive because it is memory-efficient, parallel, and compatible with black-box or discrete rewards. Yet its population-size conclusions conflict sharply: fine-tuning with cross-entropy (CE) reward succeeds with $N=1$, while binary-reward training often needs $N \approx 30$. We show this gap is largely about reward design and normalization, not population size. In the capable-model regime we study, z-score advantage normalization can cause $N=2$ to fail. Disabling normalization lets binary-reward ES with $N=2$ improve on GSM8K and TREC across capable models spanning 0.5B-7B, where the normalized variant collapses or degrades. This small-$N$ risk is set by reward granularity: binary accuracy reward induces a zero-advantage probability $q$ that depends in closed form on base accuracy, batch size, and intra-pair correctness correlation; a zero-training probe on Qwen2.5-Instruct/GSM8K matches the formula with mean absolute error 0.020 across 12 configurations and finds the availability threshold $N_{\mathrm{avail}}$ to be small in this capable-model regime. The implication is not that $N=2$ is universally sufficient, but that small-population failure in capable-model binary ES can be an implementation artifact rather than an intrinsic population limit.

Figures

Figures reproduced from arXiv: 2607.19408 by Gyubin Han, Sung Cho.

Figure 1
Figure 1. Figure 1: Turning off advantage normalization recovers small-population ES across model sizes and tasks. Advantage-normalization ablation at N = 2 with binary reward (seeds 42/43/44; mean ± std), σ = 10−3 , η = 10−3 , B = 16, 200 iterations; for Qwen2.5-{0.5B, 1.5B, 7B} (columns) on GSM8K (top row) and TREC (bottom row). In every panel, z-score normalization (on, orange) collapses (i.e. accuracy degrades toward zero… view at source ↗
Figure 2
Figure 2. Figure 2: below shows population-scaling trajectories under CE versus binary reward. The binary runs all use the default advantage-normalized ES implementation; the normalization ablation is reported separately in Section 4. 0 100 200 300 400 500 600 Training forward passes (K) 0.0 0.2 0.4 0.6 0.8 Validation accuracy N=1 N=2 N=4 N=8 N=16 0 2 4 6 8 10 12 14 16 Training forward passes (K) 0.0 0.1 0.2 0.3 0.4 0.5 Valid… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 1 canonical work pages

  1. [1]

    Dense reward for free in reinforcement learning from human feedback

    Alex James Chan, Hao Sun, Samuel Holt, and Mihaela Van Der Schaar. Dense reward for free in reinforcement learning from human feedback. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, ed- itors,Proceedings of the 41st International Conference on Machine Learning, volume 235 of P...

    2024

  2. [2]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URLhttps://arxiv. org/abs/2110.14168

    Pith/arXiv arXiv 2021

  3. [3]

    Process reinforcement through implicit rewards, 2025

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards, 2025. URLh...

    Pith/arXiv arXiv 2025

  4. [4]

    On designing effective rl reward at training time for llm reasoning, 2024

    Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for llm reasoning, 2024. URLhttps://arxiv.org/abs/2410.15115

    Pith/arXiv arXiv 2024

  5. [5]

    To- ward semantics-based answer pinpointing

    Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran. To- ward semantics-based answer pinpointing. InProceedings of the First International Con- ference on Human Language Technology Research, 2001. URLhttps://www.aclweb. org/anthology/H01-1069

    2001

  6. [6]

    Learning question classifiers

    Xin Li and Dan Roth. Learning question classifiers. InCOLING 2002: The 19th Inter- national Conference on Computational Linguistics, 2002. URLhttps://www.aclweb. org/anthology/C02-1150

    2002

  7. [7]

    The blessing of dimensionality in llm fine-tuning: A variance-curvature perspective,

    Qiyao Liang, Jinyeop Song, Yizhou Liu, Jeff Gore, Ila Fiete, Risto Miikkulainen, and Xin Qiu. The blessing of dimensionality in llm fine-tuning: A variance-curvature perspective,

  8. [8]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, 2024

    2024

  9. [9]

    Improve mathematical reasoning in language models by automated process supervision, 2024

    Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models by automated process supervision, 2024. URLhttps:// arxiv.org/abs/2406.06592

    Pith/arXiv arXiv 2024

  10. [10]

    Lee, Danqi Chen, and Sanjeev Arora

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. InAdvances in Neural Information Processing Systems, 2023. 7 REWARD-AWAREPOPULATIONSCALING OFES

    2023

  11. [11]

    Random gradient-free minimization of convex func- tions.Foundations of Computational Mathematics, 17(2):527–566, 2017

    Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex func- tions.Foundations of Computational Mathematics, 17(2):527–566, 2017

    2017

  12. [12]

    Hayes, Qiyao Liang, Yinggan Xu, Roberto Dailey, Elliot Mey- erson, Babak Hodjat, and Risto Miikkulainen

    Xin Qiu, Yulu Gan, Conor F. Hayes, Qiyao Liang, Yinggan Xu, Roberto Dailey, Elliot Mey- erson, Babak Hodjat, and Risto Miikkulainen. Evolution strategies at scale: Llm fine-tuning beyond reinforcement learning, 2026. URLhttps://arxiv.org/abs/2509.24372

    Pith/arXiv arXiv 2026

  13. [13]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

    Pith/arXiv arXiv 2025

  14. [14]

    Evolution strategies as a scalable alternative to reinforcement learning, 2017

    Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning, 2017. URLhttps://arxiv.org/ abs/1703.03864

    Pith/arXiv arXiv 2017

  15. [15]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sen- timent treebank. InProceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1631–1642, 2013

    2013

  16. [16]

    Black-box tun- ing for language-model-as-a-service

    Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tun- ing for language-model-as-a-service. InProceedings of the 39th International Conference on Machine Learning, volume 162 ofProceedings of Machine Learning Research, pages 20841– 20855. PMLR, 2022

    2022

  17. [17]

    Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 9426–9439. Association for Computational Linguistics, 2024

    2024

  18. [18]

    TLCR: Token-level continuous reward for fine-grained reinforcement learning from human feedback

    Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Nam, Daejin Jo, Kyoung- Woon On, Mark Hasegawa-Johnson, Sungwoong Kim, and Chang Yoo. TLCR: Token-level continuous reward for fine-grained reinforcement learning from human feedback. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Association for Com- putational Linguis...

    work page doi:10.18653/v1/2024.findings-ac 2024

  19. [19]

    Opt: Open pre-trained transformer language models, 2022

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. 8 REWARD-AWAREPOPULATION...

    2022

  20. [21]

    SampleKperturbation pairs{ε i}K i=1 fromN(0, Id)at theσintended for training

  21. [22]

    Total cost:2KB forward passes

    EvaluateR(θ 0 +σε i,B)andR(θ 0 −σε i,B)on a fixed batch of sizeB. Total cost:2KB forward passes

  22. [23]

    Estimate the zero-advantage rateˆq=K −1 |{i:R(θ 0 +σε i,B) =R(θ 0 −σε i,B)}|, the base accuracyp 0, and the intra-pair correctness correlationˆρfrom the paired indicators

  23. [24]

    ComputeN avail(δ) =⌈logδ/log ˆq⌉at the desired confidence (we useδ= 0.05throughout). ForK= 200andB= 16, the probe costs6,400forward passes (on the order of a handful of training iterations) and returns a data-driven population floor that bypasses the homogeneous- batch approximation in Proposition 3. We recommend it as a cheaper alternative to sweepingN e...

  24. [2026]

    URLhttps://arxiv.org/abs/2602.00170

    arXiv