{"id":"9c8574ff-d512-451c-aab6-8bb59cb55003","arxiv_id":"2505.07271","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Adding a batch-wise sum-to-zero penalty to Bradley-Terry reward modeling makes reward models more robust to unseen prompts and responses, according to experiments across multiple model families and benchmarks.","lead":"This paper proposes a simple training penalty for reward models used in RLHF: force the average reward in each batch to zero. The authors report that this makes reward models more robust to new prompts and response styles, and improves downstream RLHF win rates.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BSR's gradient (Eq. 28) is misderived: the loss centers the batch mean, not per-example outliers, so the paper's central norm-dispersion causal claim is unsupported.","rationale":"The reader's CONDITIONAL verdict is appropriate, but my most load-bearing concern differs from the reader's stated weakest assumption. The reader identified ArmoRM as the gold preference model and worried that its biases could invalidate the controlled evaluations. That is a real external-validity concern, but it does not touch the internal logic of the causal claim: even with a biased gold model, the paper's proposed mechanism could in principle be correct. The gradient error in Eq. (28) is more fundamental because it breaks the internal logic. The paper claims BSR constrains extreme reward magnitudes by penalizing outliers, yet the actual loss is a squared batch mean, whose gradient is uniform across the batch. The paper's own derivation of the mechanism is therefore incorrect, independent of any choice of gold reward model or benchmark. This matters because the novelty of the paper rests substantially on the diagnosis that hidden-state norm dispersion is the main source of over-optimization and that BSR acts on that dispersion. If BSR only centers the mean reward, the link between the regularizer and norm dispersion is at best indirect and unexplained. The empirical results may still support BSR as a useful regularizer, so rejection would be too harsh, but the paper needs a corrected derivation and a revised causal narrative. Since the reader already assigned CONDITIONAL and explicitly noted the Eq. (28) error in the rationale, my read does not change the verdict; it sharpens the reason. I therefore mark agreement as partial: the reader's chosen weakest assumption is not the one I would emphasize, but the reader did independently flag the derivation issue.","tokens_in":25500,"tokens_out":9362,"duration_ms":92822,"concrete_test":"Recompute the gradient of Eq. (25) analytically: if ∂L_BSR/∂h(x_i,y_{i,j}) equals (sum of all rewards in the batch)/(2|B|^2) * W_p rather than r(x_i,y_{i,j})/|B| * W_p, then Eq. (28) is wrong and the stated outlier-penalty mechanism is false. As a complementary check, inspect the released training code to confirm which loss is actually implemented; if the code implements Eq. (25), log per-example gradients during one training step and verify that all examples in a batch receive the same hidden-state update direction scaled by the batch mean, not by their individual rewards.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim has two linked parts: (i) BT training inflates hidden-state norm dispersion and this is the main source of over-optimization; (ii) BSR reduces that dispersion by penalizing outlier rewards. The second part is falsified by the paper's own loss. From Eq. (25), L_BSR = (1/(2|B|) * sum_{i,j} r(x_i,y_{i,j}))^2. By the chain rule, the gradient with respect to any hidden state is (sum_{k,l} r(x_k,y_{k,l}))/(2|B|^2) * W_p, a quantity that is identical for every example in the batch. Eq. (28) instead claims the gradient is (1/|B|) * r(x_i,y_{i,j}) * W_p, which would be the gradient of a per-example squared-reward penalty, not of Eq. (25). Thus BSR is a batch mean-centering regularizer: it shifts all hidden states along the same direction (and updates W_p accordingly) to drive the average reward toward zero. It does not 'penalize outliers with large magnitude,' and the gradient is not proportional to the individual reward. The reduction in hidden-state norm variance in Figure 3 may be an indirect side effect of mean-centering, but it is not the mechanism the paper describes. Consequently, the proposed intervention does not actually target the claimed cause, and the causal inference that hidden-state norm dispersion is the 'main source of over-optimization' is unsupported by the paper's own formalism. The empirical gains on RM-Bench and AlpacaEval could still be real, but they are not evidence for the proposed mechanism unless the method is re-described and the causal claim re-derived.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes batch-wise sum-to-zero regularization (BSR) as an add-on to the Bradley-Terry (BT) reward modeling loss. The authors argue that BT training induces excessive dispersion of hidden-state norms, which they claim is the main source of reward-model over-optimization. BSR is introduced to penalize extreme reward magnitudes, reduce this norm dispersion, and improve robustness to unseen prompts and response distributions. The paper evaluates BSR against several baselines in four generalization scenarios using ArmoRM as a gold preference model, assesses downstream RLHF training with RLOO, and reports experiments with an 8B model on RM-Bench and AlpacaEval 2.0, claiming consistent improvements including a 40% reduction in generation length and a 7% relative gain in win rate. Code, data, and models are released.","tokens_in":25803,"tokens_out":6310,"duration_ms":61083,"significance":"If the central claims hold, BSR is a simple, inexpensive regularizer with meaningful practical benefits for reward-model robustness and downstream RLHF alignment. The empirical study is broad: it covers multiple model families and sizes, defines a clean four-scenario evaluation of generalization, and includes external benchmarks (RM-Bench, AlpacaEval) in addition to the ArmoRM-based controlled setup. The release of code, data, and models strengthens reproducibility. However, the paper's core mechanistic account of BSR is undermined by an incorrect gradient derivation, and the causal claim that hidden-state norm dispersion is the main source of over-optimization is not established by the experiments. The empirical results remain valuable, but the theoretical framing requires substantial revision.","major_comments":[{"comment":"The gradient of L_BSR in Eq. (25) is miscomputed. With N = 2|B| rewards in the batch, L_BSR = ( (1/N) * sum_{k} r_k )^2, so for any example m, dL_BSR/dh_m = (2 * mean_r / N) * W_p = ( (1/|B|) * mean_r ) * W_p, which is identical for all examples in the batch. Eq. (28) instead states dL_BSR/dh(x_i,y_{i,j}) = (1/|B|) * r(x_i,y_{i,j}) * W_p, which corresponds to a per-example squared-reward penalty and is not the gradient of Eq. (25). Because the paper repeatedly justifies BSR as 'penalizing outliers with large magnitude' and attributes the reduced norm dispersion to gradient contributions proportional to individual rewards, this is not a minor typo: the described mechanism is not the mechanism of the implemented loss. The authors must either correct the derivation and reinterpret the regularizer as a batch-mean-centering term, or change the loss (e.g., to a per-example squared penalty) if the intended behavior is indeed outlier penalization.","section":"Section 4.2, Eq. (28)"},{"comment":"The causal claim that excessive dispersion of hidden-state norms is the 'main source of over-optimization' is not supported by the evidence presented. Figure 2 shows a correlation between BT training and growing ||h(x,y_w) - h(x,y_l)||, and Figure 3 shows that BSR reduces norm dispersion, but no experiment directly manipulates norm dispersion while holding other factors fixed. Moreover, because the true BSR gradient is a common batch-level direction (see previous comment), the variance reduction in Figure 3 is an indirect side effect of mean-centering rather than a direct penalty on high-norm outliers. In addition, the four-scenario evaluation in Section 3 uses ArmoRM as the gold preference model r*; if ArmoRM carries its own verbosity or self-preference biases, then 'over-optimization' as measured here is alignment to ArmoRM, not necessarily to human preference. The authors should weaken the causal wording (e.g., to 'BSR empirically improves robustness') or add a controlled test that isolates norm dispersion, such as comparing BSR against an explicit per-example norm penalty or an L2 regularization on hidden states.","section":"Sections 4.1 and 3"}],"minor_comments":[{"comment":"The table header contains 'BTBT-BSR' in the first column; this should read 'RMBT-BSR' for consistency with the rest of the paper.","section":"Section 5.3, Table 2"},{"comment":"The sentence 'RMBT-BSR experiences around 40% increase' appears to be a typo: the table shows that RMBT (the baseline) has the large train-to-eval effective-rank increase (+10.42), while RMBT-BSR is nearly stable (+0.14).","section":"Section 5.3, Table 1"},{"comment":"The definition of over-optimization as an accuracy gap between in-domain and out-of-domain sets is non-standard; a brief sentence connecting this to the usual notion of reward over-optimization (reward increases while true reward degrades) would help the reader.","section":"Section 2.2"},{"comment":"The average response length is reported without units; AlpacaEval typically reports token counts, and the normalization in Figure 5 is also not defined, so the plots are hard to interpret quantitatively.","section":"Section 5.3, Table 3"},{"comment":"The phrase 'with 8B RMs' could be misread as the policy being 8B; the RLHF experiments use a 1.5B policy (Qwen2.5-1.5B) with an 8B reward model, so the abstract would benefit from explicit clarification.","section":"Abstract and Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is best viewed as an empirical contribution: BSR appears to help in several benchmarks. The gradient error in Eq. (28) is a significant correctness issue that directly contradicts the method's stated mechanism, and the causal claim about hidden-state norm dispersion is over-stated relative to the evidence. These are fixable within the manuscript's scope by re-deriving the gradient, softening the causal language, and/or adding an ablation that directly tests norm dispersion. The external benchmarks (RM-Bench, AlpacaEval) provide some independent grounding, so I do not see this as a reject. The authors should also check Table 1 text and Table 2 headers for typos before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The batch sum-to-zero regularizer (BSR) is simple and the reported OOD gains are plausible, but the central mechanistic story has a derivation error that undercuts the causal claim. The four-scenario decomposition of OOD shifts (in-domain, prompt-disjoint, response-disjoint, mutual-disjoint) is a useful evaluation template, and the empirical scope is solid: multiple model families and sizes, four seeds, external benchmarks (RM-Bench, AlpacaEval), and released code, data, and models. The RLHF transfer results—40% shorter generations and a higher AlpacaEval win rate—are interesting even without error bars.\n\nThe soft spot is load-bearing. Equation (28) misstates the gradient of BSR. From Eq. (25), the gradient with respect to each hidden state is proportional to the batch mean reward, identical for every example in the batch, not proportional to the individual reward as claimed. So BSR centers the batch mean; it does not penalize per-example outliers. That changes the interpretation of Figure 3: the reduced hidden-state norm dispersion is at best an indirect side effect of mean-centering, not evidence for the proposed mechanism. The paper's claim that excessive hidden-state norm dispersion is \"the main source of over-optimization\" is therefore unsupported by its own equations. Also minor: lambda is tuned on RM-Bench in Section 5.3, the RLHF curves appear to be single seeds with no error bars, and Yang et al. (2024b) on hidden-state regularization is cited but not used as a baseline.\n\nThe empirical contribution may survive a re-description—a simple regularizer that improves OOD preference prediction—but the paper's story about why it works needs a rewrite and re-derivation. It deserves a serious referee, and I would bring it to a reading group to dissect the discrepancy. I would not cite the mechanism as-is.","headline":"BSR's empirical gains look real, but the paper's gradient derivation (Eq. 28) is wrong, so the norm-dispersion causal story doesn't hold.","tokens_in":26401,"tokens_out":3080,"would_cite":false,"duration_ms":27671,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that runaway hidden-state norm spread causes reward-model over-optimization and that a sum-to-zero batch regularizer cures it.","keywords":["reward modeling","RLHF","Bradley-Terry model","over-optimization","hidden state norm","sum-to-zero regularization","reward robustness","synthetic preference data"],"falsifier":"Train pooled reward models with and without BSR and evaluate them on freshly collected human-labeled preference pairs spanning several response styles; if the BSR model does not beat the plain Bradley-Terry model on those human labels, the claim that hidden-state norm dispersion is the main cause of over-optimization toward real preferences would be falsified.","tokens_in":25266,"feed_emoji":"🎯","tokens_out":5610,"duration_ms":48528,"temperature":0.7,"pith_summary":"Reward models trained with the standard Bradley-Terry loss lose their power on data they have not seen, a failure called over-optimization. This paper argues that the root cause is not the projection head but runaway dispersion in hidden-state norms: as training pushes chosen and rejected rewards apart, the norms of the last hidden states spread out, and that spread does not transfer to unseen prompts or response styles. The proposed remedy is batch-wise sum-to-zero regularization (BSR), which adds a penalty that keeps the average reward in each mini-batch near zero and thereby suppresses extreme reward magnitudes. Across four controlled generalization scenarios, several model families and sizes, and downstream RLHF training, the regularized reward model is claimed to align better with the gold preference model, score higher on hard preference pairs, and produce policies that are shorter and preferred more often.","feed_headline":"One batch regularizer stops reward-model over-optimization","feed_subtitle":"Added to Bradley-Terry loss, it adds 7 percent AlpacaEval wins while cutting response length 40 percent.","key_machinery":"Batch-wise sum-to-zero regularization (BSR): an additional squared term $\\mathcal{L}_{\\mathrm{BSR}} = \\left(\\frac{1}{2|B|}\\sum_{i=1}^{|B|}\\sum_{j\\in\\{w,l\\}} r(x_i,y_{i,j})\\right)^2$ added to the Bradley-Terry loss. Because its gradient is proportional to the reward itself, it pushes extreme positive and negative reward outliers back toward zero, damping the hidden-state norm dispersion that the paper identifies as the carrier of over-optimization; unlike logit normalization, it does not discard the magnitude information in reward prediction.","core_discovery":"Starting from the factorization $r(x,y) = \\|W_p\\| \\, \\|h(x,y)\\| \\cos\\psi$, the paper observes that the projection head norm stays near its initialization while the variance of $\\|h(x,y)\\|$ grows throughout training. It therefore claims that the BT objective's reward-margin maximization is effectively carried by inflating hidden-state norm differences, which is the same mechanism as over-confidence in classifiers. Adding BSR, a squared penalty on the batch sum of rewards, creates a gradient that pulls outlier rewards back toward zero proportionally to their magnitude; the paper shows empirically that this keeps hidden-state norms in a consistent range on unseen data. The claimed consequence is that BSR-trained reward models beat plain BT, hinge, logit-normalized, and margin-boosted baselines in all four over-optimization scenarios, and the robustness propagates to RLHF: the policy stays aligned to the gold reward model instead of stagnating, generation length drops by 40 percent, and the length-controlled AlpacaEval win rate rises by about 7 percent.","pith_inferences":["The verbosity angle can be pushed further: since BSR keeps hidden-state norms stable and the paper ties norm outliers to reward gaming, a direct testable prediction is that BSR-trained reward models show a smaller reward-length correlation on held-out style distributions.","The same norm-dispersion mechanism likely afflicts direct alignment objectives with implicit rewards, such as DPO-style losses, so a sum-to-zero-style regularizer could be adapted there; the paper does not test this.","Because the controlled evaluation uses ArmoRM as the gold judge, the causal claim would be strengthened by repeating the four-scenario comparison with human preference labels; if the BSR advantage shrinks, part of the measured robustness is alignment to that judge's biases.","A practical selection rule left implicit is to pick the regularization weight by hard-task accuracy, as the 8B setup does; that rule could transfer directly to other data scales and model families."],"forward_implications":["Adding BSR to the standard Bradley-Terry loss makes reward models more robust across unseen prompts, unseen response generators, and both simultaneously, on Llama-3 and Qwen2.5 backbones from 1B to 8B scale.","Robustness transfers to RLHF: policies trained with RLOO on a BSR-regularized reward model keep improving on the gold preference model instead of stagnating, indicating less reward hacking.","On RM-Bench, BSR's accuracy gain concentrates on hard preference pairs with subtle differences; at a suitable weight it exceeds the plain BT model by more than 5 percent on that harder subset.","The regularized model is less verbose: RLOO with BSR produces 40 percent shorter generations than the SFT model while raising the length-controlled AlpacaEval 2.0 win rate by about 7 percent.","The advantage of BSR grows with model size, suggesting that the regularization becomes more important as hidden dimensions and backbone capacity increase."],"supporting_citations":[{"why":"Supplies the Bradley-Terry paired comparison model that defines the reward modeling objective being regularized.","marker":"Bradley & Terry (1952)"},{"why":"Defines reward model over-optimization and establishes the synthetic gold reward model evaluation setup that the four-scenario protocol follows.","marker":"Gao et al. (2023)"},{"why":"Provides the logit normalization and over-confidence analysis that motivates the hidden-state norm dispersion hypothesis and the LBT-Norm baseline.","marker":"Wei et al. (2022)"},{"why":"Introduces ArmoRM, which is used as the gold preference model r* for all controlled over-optimization measurements.","marker":"Wang et al. (2024b)"},{"why":"Provides RLOO, the REINFORCE-style RLHF algorithm used to test propagation of reward robustness into policy training.","marker":"Ahmadian et al. (2024)"},{"why":"Introduces RM-Bench, whose hard-task accuracy is the main real-world metric for subtle preference prediction in the 8B experiments.","marker":"Liu et al. (2025b)"},{"why":"Provides Skywork-Reward-Preference data and the official 8B reward model checkpoint used as the state-of-the-art baseline.","marker":"Liu et al. (2024a)"},{"why":"Introduces length-controlled AlpacaEval 2.0, the evaluation used to measure win rate and verbosity of the RLHF-trained policies.","marker":"Dubois et al. (2024)"}],"fun_headline_variants":["Sum-to-zero batch regularizer tames reward over-optimization","Zero-sum batch loss reduces reward over-confidence","BSR: one-line regularizer for stable reward models","Reward models get robust via zero-sum batch trick","Batch sum-to-zero penalty improves reward robustness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that ArmoRM's preferences are a faithful stand-in for true human preferences; if ArmoRM carries the same verbosity or self-preference biases that the method targets, the measured robustness may be robustness to the wrong judge.","fun_headline_variants_meta":{"raw":{"variants":["Sum-to-zero batch regularizer tames reward over-optimization","Zero-sum batch loss reduces reward over-confidence","BSR: one-line regularizer for stable reward models","Reward models get robust via zero-sum batch trick","Batch sum-to-zero penalty improves reward robustness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000721,"raw_usage":{"total_tokens":3300,"prompt_tokens":1071,"completion_tokens":2229,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":687,"completion_tokens_details":{"reasoning_tokens":2153}},"tokens_in":687,"tokens_out":2229,"duration_ms":16089,"temperature":1.0,"reasoning_tokens":2153,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:21:49.983367+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train pooled reward models with and without BSR and evaluate them on freshly collected human-labeled preference pairs spanning several response styles; if the BSR model does not beat the plain Bradley-Terry model on those human labels, the claim that hidden-state norm dispersion is the main cause of over-optimization toward real preferences would be falsified.","supporting_citations":[{"cited_title":"Scaling laws for reward model overoptimization","cited_arxiv_id":null,"evidence_quote":"Defines reward model over-optimization and establishes the synthetic gold reward model evaluation setup that the four-scenario protocol follows."},{"cited_title":"Mitigating neural network overconfidence with logit normalization","cited_arxiv_id":null,"evidence_quote":"Provides the logit normalization and over-confidence analysis that motivates the hidden-state norm dispersion hypothesis and the LBT-Norm baseline."},{"cited_title":"Back to basics: Revisiting REINFORCE -style optimization for learning from human feedback in LLM s","cited_arxiv_id":null,"evidence_quote":"Provides RLOO, the REINFORCE-style RLHF algorithm used to test propagation of reward robustness into policy training."}],"review_version":1}