{"id":"584aca0e-fbfc-44ee-b07b-997a1614b07b","arxiv_id":"2506.03637","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RewardAnything follows natural-language reward principles at inference time and, with the new RABench benchmark, demonstrates that principle-conditioned listwise training beats fixed-preference reward models on held-out principles.","lead":"RewardAnything is a reward model that scores and ranks AI responses according to a natural-language principle supplied at inference time, instead of being locked to one implicit preference. The paper also introduces RABench, a benchmark for testing whether reward models can follow novel principles, and reports strong results plus an alignment case study.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RABench evidence for principle generalization is undermined by judge-derived labels: both training and evaluation use the same four-LLM consensus pipeline, and human QA only checks plausibility, so RewardAnything may be imitating the judges rather than following principles generally.","rationale":"The reader's weakest assumption and the most load-bearing concern are the same: both the training signal and the RABench evaluation signal are generated by the same four-LLM consensus pipeline, with human verification limited to a plausibility check rather than independent preference labels. If that assumption fails, the paper's headline generalization claim loses its main quantitative support. I considered the alternative concern that the 50 RABench principles are drawn from the same 200-principle curated pool used for training, which would limit claims about truly arbitrary principles; this is real but secondary, because the case study with an out-of-distribution principle provides some limited evidence of broader generalization, and the judge-loop issue contaminates even the within-distribution results. The paper has genuine strengths: the model and a clean Python API are released, the ablations in Table 4 are informative, and the RM-Bench result is a useful demonstration of inference-time principle conditioning. Those strengths are not enough to make the RABench-based generalization claim secure without independent human labels, because the benchmark and the model were both fit to the same judge distribution. A direct human-label comparison is the cheapest test that would settle the concern: if RewardAnything agrees with human preferences about as well as it agrees with the consensus labels, the central claim survives; if not, the verdict should shift from a conditional acceptance to a request for re-evaluation on independently labeled data.","tokens_in":25148,"tokens_out":5150,"duration_ms":55151,"concrete_test":"Take 300 RABench items stratified by principle category and domain and obtain direct pairwise human preference labels, not the binary validity check of Section 4.2, under each stated principle. Compute RewardAnything's pairwise accuracy and Kendall's tau against these human labels and against the original LLM-consensus labels. If agreement with humans is more than 5-10 points lower than agreement with consensus labels, or if human-consensus agreement is no better than RewardAnything-consensus agreement, the RABench-based generalization claim is unsupported. This single comparison isolates whether the four-judge loop, rather than true principle-following, explains Table 3.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims that a single 8B model can follow novel natural-language principles without retraining, and Table 3 is the main quantitative support. But the RABench ground truth (Section 4.1) and the 4,000 synthetic training examples (Section 5.3) are produced by the same consensus procedure over four LLM judges (Claude-3.7 Sonnet, GPT-4.1, DeepSeek-V3, Gemini 2.5 Pro). The human QA in Section 4.2 only asks whether each consensus ranking is a valid interpretation of the principle (89% agreement, kappa 0.57); it does not collect independent human rankings, so it cannot detect systematic biases shared by all four labelers. Algorithm 1 further weakens the word 'consensus': the final ranking is the source judge's full list with agreed-on responses kept in their original positions, and the tail order plus all scores are taken from that single judge. Thus high RABench accuracy may reflect skill at predicting these four judges' written judgments, not general principle adherence. The RM-Bench result in Table 2 is less vulnerable because the principle is supplied at inference time, but it does not establish the stronger claim of adapting to novel principles; that claim rests on RABench, where GPT-4.1 also helped create the labels it is compared against. If the four judges share systematic stylistic preferences, RewardAnything inherits them, and the reported state of the art may be inflated relative to true human principle-following.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes principle-following reward models (RMs) that condition on a natural-language principle at inference time. It introduces RABench, a benchmark of 1,002 listwise rankings built from 50 principles, prompts from RewardBench, candidate responses from 10 LLMs, and ground-truth rankings synthesized from four LLM judges (Claude-3.7 Sonnet, GPT-4.1, DeepSeek-V3, Gemini 2.5 Pro) via a dynamic-programming consensus algorithm. The authors then train RewardAnything-8B, a generative RM based on Qwen3-8B, using GRPO with a combined format and accuracy reward (Eqs. 2-4) on about 4,000 synthetic training examples labeled by the same consensus pipeline. Experiments report state-of-the-art accuracy on RM-Bench (Table 2, 86.4% overall) when a well-specified principle is supplied, and accuracy close to GPT-4.1 on RABench (Table 3, 81.9% vs. 82.5%). Ablations isolate the contribution of principle conditioning, listwise training, GRPO over SFT, relative-preference rewards, and inference-time reasoning. A case study aligns Qwen3-8B to nuanced safety behavior using only a natural-language principle and 2,000 prompts.","tokens_in":25471,"tokens_out":7418,"duration_ms":79806,"significance":"If the benchmark is trustworthy, the paper makes a practical contribution: it demonstrates that an 8B generative RM can rank a list of responses in a single call and adapt its criteria from a natural-language principle, reducing the need for task-specific RM retraining. This is a timely and potentially useful reframing, and the ablations are informative. The authors also release a Python package and detailed training hyperparameters, which aids reproducibility. However, the main generalization claim rests entirely on RABench, and RABench's ground truth is generated by the same four LLM judges used to create the training labels, with human validation limited to a coarse validity check (kappa 0.57). Until independent human rankings or an external benchmark show that RewardAnything follows principles rather than the judges' style, the headline claim should be treated as not fully established. The RM-Bench result is less affected by this circularity because that benchmark's labels are human-constructed, but it demonstrates principle-conditioned control rather than novel-principle generalization.","major_comments":[{"comment":"Both the RABench evaluation labels and the 4,000 training examples are produced by the same consensus procedure over the same four LLM judges (Claude-3.7 Sonnet, GPT-4.1, DeepSeek-V3, Gemini 2.5 Pro). The human QA in Section 4.2 only asks annotators whether a consensus ranking is a valid interpretation (89% agreement, Cohen's kappa 0.57); it does not collect independent human rankings. Consequently, high RABench accuracy may reflect the model's ability to imitate these four judges' shared judgments rather than to follow principles generally. I would ask the authors to add a held-out set of independent human rankings on a random subset of RABench and report RewardAnything's agreement with those human rankings, and to compare agreement against each individual judge to quantify judge-specific imitation.","section":"Sections 4.1, 5.3, and Algorithm 1 (Appendix F)"},{"comment":"The consensus algorithm produces only a partial consensus. The final ranking is the source judge's full list with the agreed subsequence fixed in place, and the scores in the ground truth are entirely the source judge's scores. Thus, for pairs outside the longest agreed subsequence, the ground truth is a single judge's opinion, yet Table 3 reports pairwise accuracy over all pairs. The paper should report the fraction of pairs actually supported by K=3 or more judges, and either re-aggregate the full ranking across judges or restrict evaluation to agreed pairs. This issue affects the benchmark's validity independently of the judge-label circularity concern.","section":"Algorithm 1 (Appendix F)"},{"comment":"The 50 RABench principles are sampled from the same manually curated pool of 200 principles used to create the 150 training principles. Although the sets are non-overlapping, they come from the same author-defined distribution and the same five categories (Content, Structure, Tone, Logic, Style). This supports within-distribution generalization, but not the stronger claim of adapting to arbitrary or truly novel principles. The authors should either soften the claim or extend the benchmark with principles written by external annotators or drawn from a different source, including multi-objective prioritized principles like the one used in the case study.","section":"Sections 3.1, 4.1, and 5.3"},{"comment":"The headline RM-Bench result is obtained by supplying a principle that explicitly tells the model to prioritize accuracy over length, which is exactly the objective that RM-Bench's 'hard' split is designed to test. This is a legitimate demonstration of controllability and bias mitigation, but it is not evidence of generalization to novel principles. Moreover, the same principle is passed to the baselines, and discriminative RMs cannot consume natural-language principles, so the comparison is not fully apples-to-apples as a reward-model quality benchmark. I recommend presenting this experiment as a controlled demonstration of principle-conditioned bias mitigation and relying on RABench (with the fixes above) for the generalization claim.","section":"Section 6.1 and Table 2"}],"minor_comments":[{"comment":"The description of Claude 3.5 Haiku cites reference [100], which is 'On the biology of a large language model' and is unrelated; this should be replaced with an appropriate Anthropic documentation or model-card reference.","section":"Appendix D"},{"comment":"The 'Accuracy' metric is described as pairwise ranking accuracy, but it is not defined in the text; please state how pairs are formed from the listwise rankings and how ties are counted.","section":"Table 3"},{"comment":"The 'Priority' and 'Clarity' experiments appear to use one principle per condition; please state the number of prompts used and report variance or confidence intervals, since Figure 3 is used to support a general recommendation about principle design.","section":"Figure 3"},{"comment":"The FindValidSubsequence function uses S(X_l) >= S(X_i) when counting supporting judges; if equal scores count as 'agreement,' the K threshold is not a strict preference threshold. Please clarify how ties are handled.","section":"Algorithm 1 (Appendix F)"},{"comment":"The training prompts are described as 'decontaminated' from Skywork-Reward, but no decontamination procedure is described; please specify how overlap with RABench prompts and principles was prevented.","section":"Section 5.3"},{"comment":"The case-study results are only shown in a single figure without quantitative tables; for the claim of 'significant improvements,' please provide the actual XSTest and MT-Bench scores and, where possible, significance tests.","section":"Section 6.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the reward-modeling community, and the proposed paradigm is timely. The main risk is that the benchmark is self-referential: both training and evaluation labels come from the same four-LLM consensus pipeline, and human verification does not provide independent rankings. I would encourage the editor to treat the 'SotA on RM-Bench' claim with care because the supplied principle explicitly encodes the benchmark's target objective; the more defensible claim is controllability. The revision should include human-ranked validation or an external evaluation before the generalization claim is accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a real contribution with a load-bearing but fixable evaluation flaw. RewardAnything is an open 8B generative reward model that takes a natural-language principle and a list of candidates and outputs scores and a ranking in one call. That artifact is new and useful, and the authors back it with a new benchmark (RABench), careful ablations, and a case study. The soft spot: RABench ground truth and the training labels come from the same four-LLM consensus pipeline, so part of the reported generalization could be learning to imitate those judges rather than following principles in general.\n\nWhat is genuinely new: the conceptual move is not new—SALMON proposed instructable reward models and concurrent work [24] did inference-time generalist RMs—but those didn't ship weights or a systematic benchmark. This paper ships a model (WisdomShell/RewardAnything-8B-v1), a Python package, a clean listwise GRPO recipe, and ablations showing GRPO beats SFT, reasoning matters, and listwise beats pairwise. The RM-Bench result is external and human-labeled, and the case study aligning Qwen3-8B on XSTest/MT-Bench is independent of the judge-loop. That is real evidence the principle-following paradigm can work.\n\nWhere it gets soft: the stress-test note is on target. Section 4.1 generates ground truth by having four LLM judges score and rank responses, then Algorithm 1 takes the longest subsequence of one judge's ranking supported by at least K judges; the final consensus is that source judge's full list with agreed responses kept in place, and the tail order plus all scores come from the single source judge. So 'consensus' is weaker than the word suggests. Section 5.3 uses the same style of consensus for 4,000 training examples. The human QA (89% agreement, kappa 0.57) verifies whether each element is a valid interpretation of the principle, not whether the ranking is independently correct. That means RABench accuracy partly measures agreement with the four judges. It doesn't sink the paper—the RM-Bench and case study results stand—but it does mean the headline 'adapts to novel principles without retraining' is overstated for the RABench evidence alone.\n\nMinor: the benchmark and training data are not clearly released; the model is, and that's good, but for a benchmark paper the missing data is a gap. Appendix I honestly flags sensitivity to principle phrasing as open, which I appreciate.\n\nWho it's for: anyone working on reward modeling, RLHF, or LLM evaluation. A serious referee should engage; the evaluation section needs an independent human-labeled test set or a held-out judge to kill the circularity. I'd take it, with major revision requested.","headline":"Useful open model and benchmark for principle-following reward models, but RABench's LLM-judge ground truth makes the headline generalization claim partly circular.","tokens_in":26008,"tokens_out":2693,"would_cite":true,"duration_ms":29924,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reward models can be made to follow natural-language principles at inference time, so a single 8B generative model judges any preference expressed in words without retraining, matching or surpassing task-specific models on standard…","keywords":["principle-following reward models","reward modeling","RLHF","GRPO","listwise preference learning","RABench","natural language principles","generative reward model"],"falsifier":"Construct a principle-prompt set in which the correct ranking deliberately contradicts the known preferences of the four judge models—for instance a principle that rewards charming but factually wrong answers, or the longest-winded response—collect human rankings for the same set, and check whether RewardAnything follows the stated principle or the judges' consensus whenever the two diverge; the generalization claim fails if its rankings track judge opinion rather than the principle on such cases.","tokens_in":24974,"feed_emoji":"🏆","tokens_out":12277,"duration_ms":119332,"temperature":0.7,"pith_summary":"Reward models that steer RLHF are normally trained on fixed preference datasets, so adapting to a new preference—brevity in one product, warmth in another—means collecting new annotations and retraining a model. This paper argues that reward modeling should work like instruction following: a reward model should take a natural-language principle as input and judge candidate responses against it, generalizing to novel principles without retraining. To make the capability measurable, they build RABench, a benchmark of 1002 human-validated rankings built from 50 principles in five quality dimensions, and show that existing discriminative and generative reward models generalize poorly to explicit principles. They then train RewardAnything-8B, a generative reward model that reasons about a principle and scores and ranks any list of responses in a single call; given a well-specified principle it reports state-of-the-art accuracy on the standard RM-Bench, and on RABench it matches GPT-4.1 while exceeding all reward models tested.","feed_headline":"One 8B model scores by any written principle, no retraining","feed_subtitle":"Ranks responses by whatever rule you state, and beats task-specific models on standard benchmarks.","key_machinery":"The central object is the listwise, principle-conditioned scoring task $S(P, Q, X_i) \\to \\mathbb{R}$, in which a reward model receives a natural-language principle $P$, a prompt $Q$, and $k$ candidate responses, and must return a score for each response and an overall ranking that reflect adherence to $P$. The training machinery is Group Relative Preference Learning, an application of GRPO in which the reward model, starting from an 8B instruction-tuned backbone, learns to generate structured JSON evaluations; the learning signal is a weighted reward $r = \\lambda_f r_f + \\lambda_a r_a$ combining a format reward over five structural criteria with an accuracy reward over four sub-metrics (a score-difference-weighted reversed-pair penalty, score distribution matching, partial credit for near-correct scores, and Kendall's τ with top-$k$ agreement). The measurement instrument is RABench, whose ground-truth rankings are produced by a dynamic-programming consensus algorithm that extracts the longest subsequence of one judge's ranking that at least $K$ of four LLM judges support.","core_discovery":"The paper claims that generalizable principle-following is achievable with a single 8B reward model. RewardAnything is trained to map a principle, a prompt, and a list of responses to a structured evaluation—chain-of-thought reasoning, per-response scores, and a best-to-worst ranking—on synthetic listwise data whose labels are consensus rankings of four LLM judges. Evaluated on RM-Bench with a clear principle supplied, the model reaches 86.4% overall and 84.4% on the hard style-bias split, versus 83.9% and 76.7% for the strongest reasoning-based generative reward model; evaluated on RABench, it posts 81.9% pairwise accuracy, roughly matching GPT-4.1 (82.5%) while ranking first on Kendall's τ and NDCG, and far above specialized discriminative reward models (63.0–71.4%). Ablations attribute the behavior to explicit principle conditioning, listwise training, GRPO rather than supervised fine-tuning, and inference-time reasoning: removing any of them drops accuracy by roughly 4 to 20 points. A closing case study claims that one out-of-distribution principle and 2000 prompts are enough to align Qwen3-8B toward nuanced safety behavior, improving both refusal helpfulness and MT-Bench text quality over a policy aligned with a conventional reward model.","pith_inferences":["If the central claim holds, the practical bottleneck of reward modeling shifts from collecting preference data to articulating principles; the appendix's finding that vague or unprioritized principles hurt performance suggests principle-writing skill will matter as much as model capability.","Because the same four LLM judges generate both training and evaluation labels, a bias common to those judges would be inherited by the model and enshrined by the benchmark; a held-out set with human-only labels on adversarial principles would settle whether the model follows principles or judges.","The paper flags adversarial manipulation of principles as future work; the concrete consequence is that reward signals, and anything trained on them, inherit both the strengths and the weaknesses of the supplied principle text, turning principle validation into a safety-critical step.","The same listwise machinery could extend beyond single principles to composition with explicit priorities, which RABench's five categories and the case study's prioritized multi-objective principle already point toward."],"forward_implications":["Preferences become input text instead of training data: adapting a deployed reward model to a new product or value reduces to writing a principle, so one model serves many preference scenarios.","Known reward biases can be stipulated away: on RM-Bench's hard split, supplying a principle that prioritizes factual accuracy over length takes RewardAnything to 84.4%, where conventional reward models hover near or below chance.","Ranking $k$ responses costs $\\Theta(1)$ inference calls and $\\Theta(k)$ tokens instead of $\\Theta(k^2)$ pairwise comparisons, which is what makes principle-conditioned scoring practical inside PPO and GRPO training loops.","RLHF can run on principles alone: the case study aligns a policy with one out-of-distribution principle and 2000 prompts, with no preference annotation and no reward-model training.","RABench makes principle-following a measurable capability, so progress on generalization to novel principles can be tracked separately from traditional reward accuracy."],"supporting_citations":[{"why":"the standard benchmark whose 'hard' style-bias setting grounds the bias-mitigation result; RewardAnything's state-of-the-art numbers are reported against it.","marker":"[7]"},{"why":"supplies the prompts from which RABench evaluation cases are built, so the benchmark's diversity rests on it.","marker":"[20]"},{"why":"introduces GRPO, the reinforcement learning algorithm that Group Relative Preference Learning is built on.","marker":"[28]"},{"why":"provides the decontaminated preference data whose prompts seed the synthetic training set for RewardAnything.","marker":"[9]"},{"why":"the prior 'instructable reward model' whose unavailability of weights and lack of benchmarking defines the gap this paper fills.","marker":"[23]"},{"why":"the strongest concurrent reasoning-based generative reward model, the baseline RewardAnything must beat on RM-Bench.","marker":"[25]"},{"why":"supplies the 2000 safety prompts used in the principle-only alignment case study.","marker":"[32]"},{"why":"the XSTest suite used to measure refusal and over-refusal behavior in the aligned-policy case study.","marker":"[33]"},{"why":"MT-Bench, used with GPT-4.1 judging to compare the aligned policy's text quality.","marker":"[10]"}],"fun_headline_variants":["One 8B reward model obeys any written principle, no retraining","State-of-the-art reward model follows any principle without retraining","Single 8B model adapts to new reward principles instantly, beats task-specific","No retraining needed: one reward model follows any rule you write","Generalizable reward model: write a rule, it obeys, no retraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the consensus rankings produced by four large language model judges, with human annotators checking only whether each ranking is valid (89% agreement, Cohen's $\\kappa = 0.57$), correctly measure adherence to arbitrary natural-language principles; if those judges share a systematic bias, RewardAnything may be learning to imitate the judges rather than to follow the principles.","fun_headline_variants_meta":{"raw":{"variants":["One 8B reward model obeys any written principle, no retraining","State-of-the-art reward model follows any principle without retraining","Single 8B model adapts to new reward principles instantly, beats task-specific","No retraining needed: one reward model follows any rule you write","Generalizable reward model: write a rule, it obeys, no retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000915,"raw_usage":{"total_tokens":3980,"prompt_tokens":1047,"completion_tokens":2933,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":2835}},"tokens_in":663,"tokens_out":2933,"duration_ms":19658,"temperature":1.0,"reasoning_tokens":2835,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:58:12.123505+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a principle-prompt set in which the correct ranking deliberately contradicts the known preferences of the four judge models—for instance a principle that rewards charming but factually wrong answers, or the longest-winded response—collect human rankings for the same set, and check whether RewardAnything follows the stated principle or the judges' consensus whenever the two diverge; the generalization claim fails if its rankings track judge opinion rather than the principle on such cases.","supporting_citations":[],"review_version":1}