{"id":"5390ddb2-0ebd-480d-b428-199122233a6a","arxiv_id":"2505.10775","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Reward model quality changes by up to 14% depending on the base language model chosen, and a combination of five standard benchmarks can partially guide that choice.","lead":"This paper trains reward models on top of 40 different open-source language models and compares their scores on RewardBench. It finds that the choice of base model changes reward model accuracy by up to 14%, and that a few existing benchmarks can roughly predict the best starting models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"In-sample evaluation inflates the headline +18% model-selection gain: §4.2 fits Elastic Net to RewardBench, then reports coverage from the same full-data fit, so the selection benefit is not shown to generalize to new base models.","rationale":"The reader's verdict of CONDITIONAL is appropriate, but the most load-bearing concern is not the lack of repeated seeds; it is the in-sample evaluation of the model-selection regression. The paper's own setup in §4.2 uses 10-fold CV only for hyperparameter selection, then fits on all 40 models and reports coverage from that same fit. Since the regression target is RewardBench itself, high top-k coverage on the training models is unsurprising and does not establish that benchmark results can select good reward models a priori for new base models. The reader's weakest_assumption about single-run variance is real, but the headline 3-14% base-model effect is supported by consistent cross-family patterns across both Bradley-Terry and regression objectives, and by a per-model learning-rate search with validation checkpointing; a single noisy run would not easily explain the repeated advantage of Gemma-2 and Qwen2.5 models over Llama-3.x counterparts. The in-sample regression concern is decisive for the paper's predictive contribution, so a held-out coverage test is the right next step. If such a test preserves the +18% improvement, the central selection claim would be substantially strengthened; if not, the paper's empirical measurements remain useful but its model-selection method should be reframed as exploratory rather than predictive.","tokens_in":20782,"tokens_out":6827,"duration_ms":71563,"concrete_test":"Re-run the §4.2 pipeline with leave-one-model-out evaluation: for each held-out model, fit Elastic Net (including feature selection) on the remaining 39 models, predict the held-out RewardBench score, form out-of-fold predicted rankings, and compute C(β,ρ,L,k) for k=5 and k=10 against the ANLI-only baseline. If out-of-fold coverage no longer exceeds the single best benchmark, the +18% selection gain is an in-sample artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 states that a 10-fold cross-validation is run only to search Elastic Net hyperparameters, after which \"we fit a model over all samples using the best hyperparameters.\" Figure 6 then compares the coverage of this full-data model against single benchmarks. Because the regression target is the RewardBench score of the same 40 models and the features (MBPP+, HumanEval+, ToxiGen, IFEval, #Params) were selected after inspecting the full data, the reported +18% top-5/10 coverage is an in-sample description of fit, not a predictive result. A regression fit to the evaluation target will naturally rank the training models well; the open question is whether benchmark scores predict RewardBench for a new base model. The paper provides no held-out coverage statistic, so the model-selection contribution is currently unverified. This is more load-bearing than the single-run variance issue: even with perfect training reproducibility, the +18% claim would still be unestablished. The 3-14% base-model gains are better supported because family-level advantages (Gemma-2, Qwen2.5 over Llama-3.x) appear consistently across both training objectives and across multiple models, although no confidence intervals are reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates how the choice of base model affects downstream reward modeling performance. The authors train 40 chat models (0.49B to 10.3B parameters) from multiple families using two objectives — Bradley-Terry with binary preferences on HelpSteer2-Preference and multi-attribute regression on HelpSteer2 — and evaluate all resulting reward models on RewardBench. They report relative gains of up to 14% over the commonly used Llama-3.x baselines in matched parameter-size groups, analyze correlations between 33 standard benchmarks and RewardBench scores, and propose an Elastic Net regression that combines a small set of benchmark features for model selection, claiming an average +18% improvement in top-5/10 coverage over single benchmarks. The paper also studies post-training stages using publicly available Llama-3.1-Tulu-3 checkpoints, finding that SFT helps (+15.5%) while subsequent DPO/RLVR steps hurt (3-5%), and estimates pre-training data distributions of several models using length-normalized sequence probabilities, reporting a +1.5% improvement in reward-model prediction MAE when these distribution features are added.","tokens_in":20992,"tokens_out":3672,"duration_ms":40177,"significance":"If the results hold, the paper makes a useful practical contribution by identifying base model choice as a high-leverage hyperparameter for reward modeling. The study is broad in scope: 40 models, two training objectives, four RewardBench subcategories, and multiple training-stage comparisons. The authors also provide a helpful independent verification of RewardBench labels (Appendix A) and make concrete experimental details available, including full per-model results in the appendices. The main risk is that the model-selection claim is evaluated in-sample, so the headline +18% coverage improvement is not yet shown to transfer to new base models; additionally, the absence of repeated-seed variance reporting makes the size of the base-model effect less certain. These issues are fixable within the scope of the manuscript, so the work is promising but needs revision.","major_comments":[{"comment":"The central model-selection claim, an average +18% improvement in top-5/10 coverage, is evaluated in-sample. Section 4.2 states that a 10-fold cross-validation is run only to search Elastic Net hyperparameters, after which \"we fit a model over all samples using the best hyperparameters.\" The coverage plotted in Figure 6 is therefore computed for the same 40 models used to fit the final regression, so the comparison measures how well the fitted model ranks its own training data, not how well benchmark features predict RewardBench for a new base model. Because this claim appears in the abstract and in contribution 3, the authors should provide a held-out model-level evaluation, for example by leaving out whole models or model families in an outer cross-validation loop and reporting coverage on the held-out folds, or by using an explicit train/validation split. The feature-selection process should also be nested inside the validation procedure, since the five selected features were identified after inspecting the full dataset.","section":"Section 4.2 and Figure 6"},{"comment":"The headline result of 3-14% performance gains from base model choice rests on a single training run per base model. Section 3.1 describes one learning-rate grid search per model and checkpoint selection on the HelpSteer2 validation set, with no repeated seeds or confidence intervals reported in Section 3.3 or in Tables 4-5. If run-to-run variance from random seeds, learning-rate sensitivity, or checkpoint selection is comparable to the observed gaps, the gains cannot be cleanly attributed to base model choice. I recommend repeating training for at least the headline comparison pairs and a random subset of other models with multiple seeds, and reporting the mean and standard deviation (or confidence intervals) so the reader can assess whether the observed gaps exceed optimization noise. The consistent family-level patterns (Gemma-2 and Qwen2.5 outperforming Llama-3.x across both objectives) partially mitigate this concern, but the size of the claimed gains needs variance quantification.","section":"Section 3.1 and Section 3.3"},{"comment":"The reported +1.5% MAE improvement from adding estimated pre-training distribution features (from 3.2% to 1.7%) appears to use the same in-sample regression setup as Section 4.2: the regression is fit on all models and the MAE is computed on the same models. The paragraph also does not specify how many models are included in this analysis, and the pre-training distribution estimates are only available for a subset of the 40 models. As written, this improvement may reflect overfitting rather than predictive benefit. The authors should state whether the reported MAE is in-sample or cross-validated, and if the latter, describe the exact folds and model subset; if the former, they should add a held-out evaluation before claiming that pre-training distributions reduce prediction error.","section":"Section 5.2, last paragraph"}],"minor_comments":[{"comment":"The coverage filtering thresholds are inconsistent across figures: Figure 4 retains benchmarks with at least 0.4 and 0.7 coverage at k=5 and k=10, while Figures 9 and 10 in Appendix B use 0.4 and 0.6. Please reconcile the thresholds and state the final criterion clearly.","section":"Figure captions"},{"comment":"The significance test in Eq. (8) is applied to many pairwise correlations, but no multiple-testing correction is mentioned. The statement that \"most of the benchmarks' correlations have statistical significance\" should be qualified in light of the number of tests performed, or the analysis should include a correction such as Benjamini-Hochberg.","section":"Section 4.1, Significance Test"},{"comment":"There are several typos and minor wording issues, including \"RewarcBench\" in Section 4.2, \"LLama\" in the introduction, and the phrase \"finding their similar performances\" in Appendix E. A careful proofread would improve presentation.","section":"Throughout"},{"comment":"The text says results show \"potentially high variances in performance within groups of models with similar sizes,\" but no variance estimates are provided anywhere in the paper. Please either report such estimates or rephrase to refer to observed spread across models rather than statistical variance.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.CL and the empirical breadth is a strength. The main issue is that the model-selection contribution is currently an in-sample description of fit rather than a validated predictive result; this can be addressed with nested model-level cross-validation. I would also encourage the authors to add repeated-seed results for at least a subset of models, since the paper's central 'base model matters' claim depends on the observed gaps being larger than run-to-run noise."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing: this paper has a genuinely useful dataset—40 base models, two training objectives, RewardBench scores plus a broad benchmark battery—and the main qualitative finding holds up: base model family matters, with Qwen2.5 and Gemma-2 consistently beating Llama-3.x at similar sizes by 3-14%. That part is credible and worth taking seriously.\n\nThe soft spot is the regression-based selection claim. Section 4.2 does 10-fold CV only to pick hyperparameters, then refits on all 40 models and reports coverage on the same 40. The +18% top-5/10 coverage is thus a description of in-sample fit, not a predictive result. The stress-test note is right that this is load-bearing: even with perfect training reproducibility, the +18% claim would still be unverified for a new base model. The pre-training feature improvement (+1.5% MAE) appears to use the same fitted pipeline, so that number has the same problem. I'd also note the lack of repeated seeds or confidence intervals on the 3-14% gains; single runs make the magnitude uncertain, although the family-level pattern is robust enough across many models that I don't doubt the direction.\n\nWhat the paper does well: the benchmark correlation analysis is careful, with significance and coverage tests; the post-training stage analysis is small but informative; the limitations section is honest about compute constraints and single-dataset scope. The appendix verification of a RewardBench sample (98% agreement) is a nice touch. Citation practices look fine.\n\nWho this is for: RLHF practitioners who want a quick sense of which base model families are worth trying, and researchers working on model selection heuristics. The direct comparison table is the most valuable artifact. The predictive regression needs out-of-sample validation—cross-validated coverage or a held-out set of newer base models—before it can be used as a selection tool.\n\nMy recommendation: send it to peer review. The empirical core is solid enough to deserve referee time, and the in-sample issue is fixable with a proper held-out evaluation. A skeptical reviewer should focus on Section 4.2, but the paper's main contribution does not collapse without it.","headline":"A useful empirical sweep of 40 base models for reward modeling, but the headline model-selection claim is in-sample and needs re-validation before being cited as predictive.","tokens_in":21572,"tokens_out":1684,"would_cite":true,"duration_ms":15253,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims the base model is a critical, overlooked hyperparameter for reward modeling: swapping Llama-3.x for the best similar-size model improves RewardBench by up to 14%, and a five-benchmark regression predicts most of that…","keywords":["reward modeling","base model selection","RewardBench","HelpSteer2","model selection","benchmark correlation","post-training","pre-training data distribution"],"falsifier":"Retrain a subset of the 40 models across several random seeds and on at least one additional preference dataset, such as UltraFeedback. If the same-size ranking changes so that a different family leads, or the gap between Llama-3.x and the best alternative falls below the seed-to-seed spread, the headline 3–14% gain cannot be attributed cleanly to base-model choice.","tokens_in":20552,"feed_emoji":"🤖","tokens_out":14541,"duration_ms":122101,"temperature":0.7,"pith_summary":"Reward models are the scoring components used in reinforcement learning from human feedback; they rate which of two responses a user would prefer. This paper argues that the base model those scores are built on is a decisive hyperparameter, not a fixed default. Training the same reward-modeling objective on 40 chat models grouped by parameter count, it reports that replacing the common Llama-3.x base with the best model of similar size improves RewardBench by 3 to 14 percent, with Qwen2.5 and Gemma-2 models consistently ahead. The paper also claims that reward-model quality can be predicted before training: a simple linear combination of five reported benchmark scores beats any single benchmark at picking the top models, raising average top-5/10 overlap by about 18 percent. Its post-training analysis finds supervised fine-tuning is the stage that matters most (+15.5%), while later alignment steps cost 3 to 5 percent; the upshot is a practical recipe — treat the base model as a search axis, and use cheap benchmark proxies to choose it.","feed_headline":"Same-size base-model swap lifts reward scores by up to 14%","feed_subtitle":"Picking the right same-size base model buys 3–14% on RewardBench without changing data or training objective.","key_machinery":"The load-bearing setup is a controlled model-selection protocol. Forty chat models are partitioned into three parameter buckets ($<3$B, $3$–$6$B, $\\geq6$B); each is fine-tuned on the same HelpSteer2(-Preference) data under both a Bradley-Terry pairwise loss and a multi-attribute regression loss, and the best checkpoint per model is scored on RewardBench. Selection quality is quantified by top-$k$ coverage, $C(\\beta,\\rho,L,k)=|T_\\beta(L,k)\\cap T_\\rho(L,k)|/k$, the fraction of the true top-$k$ reward models that a benchmark also ranks in its top-$k$. Prediction is carried by a 10-fold Elastic Net over benchmark scores and parameter count. The pre-training probe is the length-normalized token log-probability $S_\\phi(D,N)=\\frac{1}{N}\\sum_{i=1}^{N}\\log p_\\phi(t_i|t_{1:i-1})$ computed on a 1M-document SlimPajama sample. These pieces let the paper turn a costly search over base models into a cheap regression problem while keeping training conditions fixed.","core_discovery":"The central discovery is an empirical regularity: holding the reward-modeling dataset, objective, and training budget fixed, the choice of base chat model changes RewardBench accuracy by up to 14 percent relative to the Llama-3.x default, within each of three parameter buckets (<3B, 3–6B, ≥6B). The paper further establishes that strong benchmark correlations (Pearson and Spearman, many at 0.8 or above) are misleading for model selection because top-rank coverage is low; a 10-fold Elastic Net trained on a small set of reported scores — two coding benchmarks, one safety benchmark, one instruction-following benchmark, and parameter count — recovers much of the ranking and improves top-5/10 overlap by about 18 percent on average. In post-training, supervised fine-tuning contributes the largest share of reward-modeling ability, and DPO/RLVR alignment stages reverse part of that gain. Finally, estimated pre-training data composition differs measurably across model families and, added to the same regression, lowers mean absolute prediction error from 3.2% to 1.7%.","pith_inferences":["A natural next test is to run the same selection recipe on a different reward training set, such as UltraFeedback; if the five-feature regression still ranks winners, the result transfers beyond HelpSteer2.","The paper's data-composition probe could be turned into a family-level correction term for reward-model scaling laws, something the authors do not attempt.","The predictor's reliance on coding, safety, instruction following, and parameter count suggests a cheap pre-screen for new base models: compute those five reported numbers before committing GPU-hours to reward-model training."],"forward_implications":["Practitioners can gain 3–14% RewardBench score by switching the base model within the same size class, with no changes to training data or objective.","Qwen2.5 and Gemma-2 families are reliable alternatives to Llama-3.x as reward-model bases at the sizes tested, so defaulting to one family is a measurable opportunity cost.","A linear model over five reported numbers — two coding benchmarks, a safety benchmark, an instruction-following benchmark, and parameter count — can serve as a cheaper model-selection screen than training every candidate.","Supervised fine-tuning is the load-bearing post-training stage for reward modeling, and running later alignment stages such as DPO or RLVR can reduce the final RewardBench score by 3–5%.","Estimated pre-training data composition carries usable signal for predicting reward-model quality, reducing mean absolute prediction error from 3.2% to 1.7% when added to the benchmark regression."],"supporting_citations":[{"why":"RewardBench is the evaluation benchmark whose overall accuracy defines the reward-modeling performance gains throughout the paper.","marker":"Lambert et al., 2024b"},{"why":"HelpSteer2 is the multi-attribute training dataset for the regression reward models and supplies the validation set for checkpoint selection.","marker":"Wang et al., 2024d"},{"why":"HelpSteer2-Preference is the binary-preference dataset on which every Bradley-Terry reward model is trained.","marker":"Wang et al., 2024c"},{"why":"The Llama-3.x family is the default baseline and the source of the 33-benchmark capability list used in the correlation and regression analyses.","marker":"Dubey et al., 2024"},{"why":"The Qwen2.5 family is one of the two families shown to consistently beat Llama-3.x at matched size, and it contributes benchmark scores.","marker":"Yang et al., 2024"},{"why":"The Gemma-2 family is the other consistently better alternative and contributes benchmark scores to the model pool.","marker":"Team et al., 2024"},{"why":"The Tulu 3 release provides the SFT, DPO, and RLVR checkpoints used to isolate the effect of each post-training stage.","marker":"Lambert et al., 2024a"},{"why":"SlimPajama supplies the 1M-document probe set from which estimated pre-training data distributions are computed.","marker":"Soboleva et al., 2023"},{"why":"Crystal, trained on all of SlimPajama, serves as the ground-truth reference for the membership-score distributions.","marker":"Liu et al., 2024d"}],"fun_headline_variants":["Same-size base model swap: up to 14% reward boost","Base model choice gives 14% reward modeling gain","Pick a better base model, gain 14% on reward benchmarks","Reward modeling improved 14% by smarter base model pick","Base model pick: up to 14% better reward scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper treats one training run per base model, with a small learning-rate grid and best-checkpoint selection on the HelpSteer2 validation set, as a stable measure of each base model's reward-modeling quality; if rerunning the same model with different seeds or hyperparameters changes the score by as much as the reported gaps, the ranking is not reliable.","fun_headline_variants_meta":{"raw":{"variants":["Same-size base model swap: up to 14% reward boost","Base model choice gives 14% reward modeling gain","Pick a better base model, gain 14% on reward benchmarks","Reward modeling improved 14% by smarter base model pick","Base model pick: up to 14% better reward scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001087,"raw_usage":{"total_tokens":4542,"prompt_tokens":940,"completion_tokens":3602,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":3516}},"tokens_in":556,"tokens_out":3602,"duration_ms":26449,"temperature":1.0,"reasoning_tokens":3516,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:03:37.538781+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain a subset of the 40 models across several random seeds and on at least one additional preference dataset, such as UltraFeedback. If the same-size ranking changes so that a different family leads, or the gap between Llama-3.x and the best alternative falls below the seed-to-seed spread, the headline 3–14% gain cannot be attributed cleanly to base-model choice.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SlimPajama supplies the 1M-document probe set from which estimated pre-training data distributions are computed."}],"review_version":1}