{"id":"12eadcec-9bcc-4070-96fc-4528f44e41cd","arxiv_id":"2507.07375","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Jointly training a Bradley-Terry preference head and a multi-attribute regression head on a shared embedding improves reward-model robustness to reward hacking and boosts multi-objective scoring performance.","lead":"Researchers propose SMORM, a reward model that trains a single-objective preference head and a multi-objective attribute head together on one shared embedding. The method is reported to reduce reward hacking in out-of-distribution settings and to improve reward-model scoring, including a 7B model matching a model trained on much more data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's proof replaces the actual logistic BT loss with a squared loss and asserts that the population minimizers coincide; this is false, so the central theoretical result is not established.","rationale":"The reader's weakest-assumption analysis is exactly right: the invalid equivalence in Appendix C.2 is the load-bearing flaw in the paper's theoretical narrative. The paper's central claim has an empirical component (joint training improves robustness and scoring) and a theoretical component (Theorem 1 and Theorem 2). Theorem 1's proof is not merely missing an epsilon; the proof derives w_S from a squared-loss objective that the model is not trained with, so Eq. (11), Eq. (13), and the constants c and ε do not follow for the trained model. The quoted surrogate-loss justification conflates consistency of a classification surrogate with equality of population minimizers, which is false for linear models except in restrictive cases. Moreover, Theorem 1 assumes the positive-correlation condition (3), so the nontrivial step is precisely the one that is unproved. Theorem 2 has a related under-specification: its strict Fisher-information increase relies on positive correlation between gradients, but formal assumption (3) concerns α = μ_S^⊤ Σ_M^{-1} C_M, not gradient alignments. Separately, the headline 7B-vs-70B result in Table 3 is a 0.2-point RewardBench difference (89.0 vs 88.8) with no error bars, and the RLHF curves in Figs. 2–5 are single runs, so the empirical complementarity claim is weaker than the prose suggests. These issues justify a CONDITIONAL posture: the recipe may work and the multi-dataset, multi-backbone experiments provide useful evidence, but the theory and the strongest headline claims need repair before the paper can be accepted as stated. Since the reader already reached CONDITIONAL, no verdict change is needed.","tokens_in":30233,"tokens_out":6922,"duration_ms":88197,"concrete_test":"Re-derive the population first-order conditions for w_S using the actual logistic loss in Eq. (5) instead of replacing it with squared loss, and check whether Theorem 1's conclusion follows. A minimal numerical test: let u be a one-dimensional feature with a skewed distribution, set P(chosen | u) = σ(2u), and choose a multi-objective head satisfying assumption (3); compare the logistic minimizer with Σ^{-1} μ_S, then test whether r_m ≥ c r_s - ε holds with the paper's c over the support of u. If the inequality fails on this example, Theorem 1 as stated is false; if it holds, the proof must be rewritten using the true logistic minimizer rather than the squared-loss minimizer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The proof of Theorem 1 (Appendix C.2) does not analyze the SMORM objective as defined in Eq. (5). It replaces the logistic BT loss with a squared loss, asserting that 'any strictly convex proper surrogate has the same first-order optimality conditions up to a positive scalar factor,' and then obtains w_S = Σ_S^{-1} μ_S (Eq. 11). This step is not valid for the two losses at hand. For a linear head on features u = f_c - f_r, the squared-loss minimizer is indeed Σ_S^{-1} μ_S, but the logistic/BT minimizer solves E[u(σ(w_S^⊤u) - p(u))] = 0, where p(u) = P(chosen | u). These minimizers coincide only in special cases (e.g., Gaussian class-conditional features with equal covariance); for skewed feature distributions or feature-dependent label noise, their directions differ. Consequently, the derivation of w_S, the decomposition in Eq. (13), and the constants c and ε used in the claimed inequality r_m ≥ c r_s - ε all refer to a head that the actual training objective does not produce. Since Theorem 1 is the paper's stated theoretical basis for the complementary-benefit claim — including the claims that BT training improves the multi-objective head and that SMORM-F can substitute for SMORM-M — the central theoretical result is not established by the given proof. The empirical recipe may still be valuable, but it cannot carry the 'theoretically grounded' claim in the abstract.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SMORM, a unified reward-modeling framework that jointly trains a Bradley-Terry single-objective head and a multi-objective regression head on a shared transformer embedding. The authors claim that this joint training yields complementary benefits: the regression task improves the single-objective head's robustness against reward hacking in out-of-distribution settings, while the BT task improves the scoring capability of the multi-objective head, allowing a 7B model to outperform a 70B baseline. The paper supports these claims with Theorem 1 (a lower bound relating the multi-objective average score to the single-objective score), Lemma 1 (an MSE-to-preference-error bound), Theorem 2 (an asymptotic MSE reduction under joint training), and experiments on RewardBench, RM-Bench, PPO, and Best-of-N sampling.","tokens_in":1913,"tokens_out":2115,"duration_ms":99161,"significance":"The problem addressed is timely and important: reward hacking under out-of-distribution prompts is a real limitation of current reward models, and the idea of sharing an embedding between a BT head and a multi-attribute regression head is simple and plausible. The experimental scope is broad, covering multiple base models, data scales, and evaluation protocols, and the paper is honest about the difficulty of obtaining high-quality multi-attribute annotations. However, the theoretical results are currently the main weak point: the proof of Theorem 1 rests on an invalid equivalence between logistic and squared losses, and the proof of Theorem 2 is incomplete. If the theoretical claims were repaired or appropriately softened, the empirical recipe would still be a useful contribution; as written, the 'theoretically grounded' framing is not supported.","major_comments":[{"comment":"The proof of Theorem 1 replaces the logistic BT loss with a squared loss and asserts that 'any strictly convex proper surrogate has the same first-order optimality conditions up to a positive scalar factor.' This claim is false for the losses in question. For a linear head w on features u = f_c - f_r, the logistic/BT population minimizer satisfies E[u(σ(w^T u) - p(u))] = 0, whereas the squared-loss minimizer satisfies E[uu^T]w = E[u t] with t = 1; these minimizers coincide only under special distributional assumptions, such as Gaussian class-conditional features with equal covariance. The derivation of w_S = Σ_S^{-1} μ_S, the decomposition in Eq. (13), and the constants c and ε in Eq. (14) therefore apply to a different objective than the SMORM objective in Eq. (5). Since Theorem 1 is the paper's stated basis for the 'implicit multi-attribute effect' and for the claim that SMORM-F matches SMORM-M, the central theoretical result is not established by the proof. In addition, the theorem's conclusion is structurally dependent on assumption (3), 1^T α ≥ 0, because c is defined directly from α; the lower bound is therefore closer to a restatement of the assumption than an independent derivation, and the paper does not verify this assumption on the trained models.","section":"Appendix C.2, Eq. (11)-(14)"},{"comment":"The proof of Theorem 2 is incomplete in several steps. First, the argument that the hybrid Fisher matrix is strictly larger than the single-head Fisher matrix shows only that g0^T I_hybrid g0 > g0^T I_single g0 for one direction g0; this does not imply the Loewner inequality I_hybrid - I_single ≻ 0, which would require v^T(I_hybrid - I_single)v > 0 for all v. Second, the proof invokes 'positive correlation ρ0k > 0' without defining ρ0k or proving that the relevant inner products are strictly positive. Third, the step from Cov(θ̂_hybrid) ≺ Cov(θ̂_single) to the head-specific MSE inequalities in Eq. (9) assumes that the head weights w_S and w_M are fixed, whereas in SMORM they are jointly estimated; the standard Cramér-Rao argument does not directly compare MSE of predictions under two different models with different parameter spaces. The inequalities in Eq. (9) are therefore not established.","section":"Appendix C.4, Theorem 2 proof"},{"comment":"The RLHF experiments report no error bars, no number of random seeds, and no statistical significance tests. Claims such as 'SMORM-F and SMORM-M exhibit a consistent increase in gold score throughout training' and 'SMORM significantly outperforms all baselines' are based on single trajectories, which are known to be noisy in PPO and BoN experiments. This is a load-bearing empirical claim, because the paper's main narrative is that SMORM mitigates reward hacking more reliably than prior methods.","section":"Section 5.3 and Figures 2-5"},{"comment":"The abstract's claim that SMORM 'enables a 7B model to outperform a 70B baseline' rests on the RewardBench average 89.0 for SMORM-L 7B versus 88.8 for Llama-3-70B-RM, a 0.2-point difference. No variance, confidence interval, or repeated-run comparison is reported, and the comparison involves different base models, training sets, and data sizes. As stated, this headline comparison is not statistically supported.","section":"Table 3 and Abstract"}],"minor_comments":[{"comment":"The label smoothing loss is written as L_smooth(θ) = -E[(1-ε) log σ(...) - ε log σ(...)], where the two logarithmic terms are identical. As written, this reduces to the standard BT loss and provides no smoothing; presumably one term should be log σ(-(...)) or an equivalent modification. If the implementation matches the formula, the Label Smooth baseline in Table 1 is indistinguishable from the Baseline.","section":"Appendix G.1"},{"comment":"In the proof of Theorem 1, the sentence describing the bound on the orthogonal part ends with 'the square root of the smallest eigenvalue of the .' — an incomplete sentence that should reference Σ_S.","section":"Appendix C.2"},{"comment":"The line 'Assuming E[r] = E[w_S^T] = 0' is ill-typed: E[w_S^T] is a vector mean, not a scalar, and the intended statement is presumably E[r] = 0 and E[w_S^T f] = 0.","section":"Appendix C.1"},{"comment":"The proof states 'By assumption (3) we have 1^T α ≥ 0 and thus 1^T β > 0'; the latter should be ≥, since 1^T α = 0 is allowed. The subsequent assertion c > 0 should be c ≥ 0 unless strict positivity is separately assumed.","section":"Appendix C.2"},{"comment":"The affiliation for Xiaomin Li is listed as 'Harvard University'; this should be corrected to 'Harvard University'.","section":"Title page"},{"comment":"In Eq. (6), the equality defining r_m is followed by the inequality; writing 'r_m = ... ≥ ...' is confusing. The definition and the inequality should be presented separately.","section":"Theorem 1 statement"}],"recommendation":"major_revision","confidential_remarks":"The empirical recipe and the OOD reward-hacking study are potentially interesting, but the theoretical backbone of the paper is not sound as written. The key issue is the false equivalence between logistic and squared losses in the proof of Theorem 1; this is exactly the kind of load-bearing error that cannot be left in place. I would be willing to reconsider after a substantive revision in which the authors either (a) repair the theory with correct assumptions and a valid proof, or (b) substantially soften the theoretical claims and present the contributions as empirical findings with appropriate statistical support. The 0.2-point 7B-vs-70B headline should also be tempered or backed by repeated runs and confidence intervals."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the empirical core is genuinely new: nobody has trained a BT single-objective head and a multi-attribute regression head jointly on a shared embedding for reward models, and the OOD PPO/BoN evaluation for reward hacking fills a real gap. Second, the theory that supposedly explains the method is not sound. The proof of Theorem 1 replaces the logistic BT loss with squared loss and asserts the minimizers have the same direction. That is false in general—for skewed features the squared-loss minimizer is finite while the logistic minimizer can drift—and the derivation of w_S, the constants c and ε, and the claimed inequality r_m ≥ c r_s − ε collapses. The inequality is also largely a restatement of the assumed positive correlation 1^T α ≥ 0. Theorem 2 has the same disease: it smuggles in a per-attribute correlation assumption and asserts strict Fisher dominance without proving it. So the \"theoretically grounded\" claim in the abstract should go.\n\nWhere the paper earns its keep: the empirical work is extensive and the signal is consistent. SMORM-F and SMORM-L beat the baselines on RewardBench and RM-Bench across two base models and ID/OOD settings; the PPO/BoN curves show joint multi-attribute training helping where GRM and ODIN fail; and the hyperparameter analysis in Figure 6 suggests SMORM is stable where GRM is twitchy. The OOD framing is a real contribution, and the Appendix J analysis of why MORMs fail on RM-Bench is thoughtful.\n\nSoft spots, in proportion. The \"7B beats 70B\" headline rests on a 0.2-point RewardBench gap (89.0 vs 88.8), with no error bars. The RLHF figures are single-seed-looking; no variance is reported. The label-smoothing baseline in Appendix G.1 is written incorrectly—the formula as given is just (1−2ε) log σ, not label smoothing. And the Section 3 setup filters HelpSteer2 to pairs already in Skywork, which sits oddly with the later \"flexible training\" claim, though they do show flexibility in the 40K/400K configurations.\n\nMy take: the method probably works, but the paper over-claims theory and under-reports uncertainty. It deserves a serious referee and a request for major revision: cut or fix the theory, add error bars or multiple seeds, fix the appendix formula, and temper the scale-matching claim. If those land, this becomes a solid contribution that reward-modeling and alignment researchers will want to read and build on.","headline":"A genuinely useful empirical recipe—joint BT and multi-attribute regression on a shared embedding—supports a broken theoretical proof, and the headline 7B-vs-70B claim is thinner than it looks.","tokens_in":31116,"tokens_out":4434,"would_cite":true,"duration_ms":47710,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bradley–Terry preference training and multi-attribute regression training are complementary when done jointly on one shared backbone, and a quantitative lower bound captures that complementarity.","keywords":["reward hacking","RLHF","Bradley-Terry reward modeling","multi-objective reward model","out-of-distribution generalization","shared embedding space","reward overoptimization","LLM alignment"],"falsifier":"Train SMORM and its two isolated baselines on identical data, then on held-out pairs plot the average attribute score $r_m$ against the BT score $r_s$ and fit the line $c r_s - \\varepsilon$: if the empirical gap $r_m - (c r_s - \\varepsilon)$ goes negative for a non-negligible share of high-scoring responses, the claimed monotone coupling is absent. The proof's substitution step can be tested directly: on a linear head with skewed real features, compute the population minimizers under logistic and squared loss with unit target — if they disagree in direction, the derivation of $w_S = \\Sigma_S^{-1}\\mu_S$ and $w_M = \\Sigma_M^{-1} C_M$ does not follow.","tokens_in":29983,"feed_emoji":"🧠","tokens_out":14567,"duration_ms":128499,"temperature":0.7,"pith_summary":"This paper tries to establish that the two standard styles of reward modeling for RLHF — pairwise Bradley–Terry (BT) preference classification and multi-attribute score regression — are complementary, and that training them jointly on one shared embedding space yields both defense against reward hacking and accurate scoring in a single forward pass. Its central claim is that the regression task, even with only small amounts of fine-grained attribute data, hardens the BT head against reward hacking in out-of-distribution settings, while the BT task corrects the embedding so the multi-attribute head scores competitively without new annotations. The paper supports this with a theoretical coupling bound and with experiments in which joint training lets a 7B reward model outscore a 70B baseline on RewardBench while resisting overoptimization in in-distribution and out-of-distribution PPO and best-of-n sampling. If the claim is right, the data-scarcity bottleneck that has kept multi-objective reward models weak is removable without collecting more attribute labels.","feed_headline":"One backbone, two reward heads: a 7B model beats 70B","feed_subtitle":"Joint Bradley–Terry and multi-attribute training fends off reward hacking and sharpens scoring in one pass.","key_machinery":"The central object is SMORM (Single and Multi-Objective Reward Model): a decoder-only LLM feature extractor $f_\\theta$ whose final hidden state is shared by two linear heads — a single-objective head $w_S$ trained with the Bradley–Terry logistic loss on chosen/rejected pairs, and a multi-objective head $w_M$ trained with squared-error regression against attribute score vectors $r \\in \\mathbb{R}^K$. The argument is carried by the coupling inequality $r_m \\ge c r_s - \\varepsilon$, whose constants come from the closed-form population minimizers $w_S = \\Sigma_S^{-1}\\mu_S$ and $w_M = \\Sigma_M^{-1} C_M$ of squared-loss versions of the two objectives. The proof route is what does the work: it substitutes squared losses for the logistic loss (asserting minimizer directions are preserved), whitens the feature space, and projects each column of the multi-objective weight matrix onto the single-objective direction; the bounded-feature assumption bounds the residual as $\\varepsilon$. Theorem 2 supplies the opposite direction: the combined task set enlarges the Fisher information matrix, shrinking the MLE covariance and hence the asymptotic MSE of both heads, with Lemma 1 bridging reduced MSE to smaller pairwise preference error.","core_discovery":"The paper's central claim is that a reward model carrying two heads on one shared decoder — a single-objective head trained with the Bradley–Terry preference loss and a multi-objective head trained with squared-error regression on fine-grained attribute scores — produces complementary gains that neither training alone can match. The formal statement is Theorem 1, the 'Implicit Multi-Attribute Effect': once both heads converge to their population minimizers, the average multi-attribute score $r_m(x,y) = \\frac{1}{K}\\sum_{i=1}^K w_{M,i}^\\top f_\\theta(x,y)$ is bounded below by $c r_s(x,y) - \\varepsilon$, a positive affine function of the single-objective score $w_S^\\top f_\\theta(x,y)$, under a positive-correlation assumption between the aggregated attribute labels and the chosen/rejected preference direction. From this the paper draws two consequences: a high single-objective score alone guarantees a respectable level of fine-grained quality, which explains why policies trained with only the single-objective head perform comparably to those using both; and the BT signal positions responses correctly in the shared embedding space, so the multi-objective head reaches competitive scoring with far less multi-attribute data. A companion theorem argues that joint training strictly reduces the asymptotic mean squared error of both heads because the combined task set enlarges the shared Fisher information, and a lemma converts that reduction into a bound on pairwise preference error.","pith_inferences":["The projection machinery behind Theorem 1 is task-agnostic: any pair of objectives sharing an embedding space, one supplying a coarse global signal and the other fine-grained local labels, should exhibit the same type of coupling — a testable prediction for settings like step-level process reward modeling where attribute labels are even scarcer.","The coupling constant $c$ from Theorem 1 could serve as a training-time diagnostic the paper does not develop: monitoring $c$ during joint training would reveal when the multi-attribute dataset is misaligned with the preference direction and joint training should be expected to fail.","The 7B-beats-70B result suggests that model size and joint-task training are partly substitutable inputs to scoring quality; mapping that trade-off across base-model sizes and attribute taxonomies would show how far the substitution extends."],"forward_implications":["A single forward pass through one backbone yields both a hack-resistant BT score and an accurate multi-attribute score, removing the two-pass inference cost of separately trained single- and multi-objective ensembles.","Multi-objective reward models can reach competitive scoring with only a small human-annotated attribute dataset (the paper uses 20K HelpSteer2 samples), because the BT head corrects embedding-space positioning; the paper's 7B SMORM outscores a 70B reward model on RewardBench.","Policies optimized against the jointly trained single-objective head resist reward hacking in OOD PPO and best-of-n settings, improving all five measured attribute dimensions — helpfulness, correctness, coherence, complexity, verbosity — rather than gaming only style dimensions.","If the variance argument in Theorem 2 holds, joint training improves both heads over their isolated counterparts at equal data, making the data-scarcity bottleneck of multi-objective reward models removable without new annotation."],"supporting_citations":[{"why":"Bradley and Terry's paired-comparison model; supplies the preference loss (Eq. 1) that the single-objective head optimizes.","marker":"[44]"},{"why":"The convex-risk result cited in Appendix C.2 for the claim that squared loss can replace logistic loss without changing minimizer directions — the step on which Theorem 1's closed-form weights depend.","marker":"[75]"},{"why":"HelpSteer2, the 20K human-annotated multi-attribute dataset used as DM; it is both the training signal for the multi-objective head and the source of the data-scarcity bottleneck the paper addresses.","marker":"[41]"},{"why":"GRM, the state-of-the-art regularization baseline whose OOD reward hacking motivates the paper; its PPO/BoN setup is reused for the RLHF experiments.","marker":"[19]"},{"why":"RewardBench, the benchmark on which SMORM's scoring gains, including the 7B-beats-70B result, are measured.","marker":"[42]"},{"why":"RM-Bench, the style-bias benchmark used in Appendix J to diagnose why baseline MORMs fail and how the BT head corrects them.","marker":"[43]"},{"why":"UltraFeedback, the 240K GPT-annotated dataset used as DM in the 400K experimental setting.","marker":"[37]"},{"why":"ArmoRM, the gating-network multi-objective model trained on 15.9× more data that SMORM's 8B model matches on RewardBench.","marker":"[36]"}],"fun_headline_variants":["Shared backbone, dual heads: 7B reward model tops 70B","Two heads beat one: BT and multi-objective rewards join forces","Complementary rewards: shared embedding yields 7B > 70B","Joint BT and regression heads improve scoring, beat 70B"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central inequality rests on the claim, made in the proof of Theorem 1, that replacing the pairwise preference (logistic) loss with a squared-error loss leaves the direction of the optimal head weights unchanged; that equivalence fails in general — with skewed features the squared-loss minimizer stays finite while the logistic minimizer can drift — so the central bound is not established by the supplied proof.","fun_headline_variants_meta":{"raw":{"variants":["Shared backbone, dual heads: 7B reward model tops 70B","Two heads beat one: BT and multi-objective rewards join forces","Complementary rewards: shared embedding yields 7B > 70B","Joint BT and regression heads improve scoring, beat 70B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000653,"raw_usage":{"total_tokens":3082,"prompt_tokens":1123,"completion_tokens":1959,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":739,"completion_tokens_details":{"reasoning_tokens":1882}},"tokens_in":739,"tokens_out":1959,"duration_ms":16409,"temperature":1.0,"reasoning_tokens":1882,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:43:20.450391+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train SMORM and its two isolated baselines on identical data, then on held-out pairs plot the average attribute score $r_m$ against the BT score $r_s$ and fit the line $c r_s - \\varepsilon$: if the empirical gap $r_m - (c r_s - \\varepsilon)$ goes negative for a non-negligible share of high-scoring responses, the claimed monotone coupling is absent. The proof's substitution step can be tested directly: on a linear head with skewed real features, compute the population minimizers under logistic and squared loss with unit target — if they disagree in direction, the derivation of $w_S = \\Sigma_S^{-1}\\mu_S$ and $w_M = \\Sigma_M^{-1} C_M$ does not follow.","supporting_citations":[{"cited_title":"Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev","cited_arxiv_id":null,"evidence_quote":"HelpSteer2, the 20K human-annotated multi-attribute dataset used as DM; it is both the training signal for the multi-objective head and the source of the data-scarcity bottleneck the paper addresses."},{"cited_title":"Regularizing hidden states enables learning generalizable reward model for LLMs","cited_arxiv_id":null,"evidence_quote":"GRM, the state-of-the-art regularization baseline whose OOD reward hacking motivates the paper; its PPO/BoN setup is reused for the RLHF experiments."},{"cited_title":"RM-bench: Benchmarking reward models of language models with subtlety and style","cited_arxiv_id":null,"evidence_quote":"RM-Bench, the style-bias benchmark used in Appendix J to diagnose why baseline MORMs fail and how the BT head corrects them."},{"cited_title":"Interpretable prefer- ences via multi-objective reward modeling and mixture-of-experts","cited_arxiv_id":null,"evidence_quote":"ArmoRM, the gating-network multi-objective model trained on 15.9× more data that SMORM's 8B model matches on RewardBench."}],"review_version":1}