{"id":"f39e0ada-4deb-4696-9696-effee4645467","arxiv_id":"2504.14838","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RETA measures reward model reliability as the average oracle quality of the top η fraction of ranked responses, divided by the average response quality.","lead":"The paper introduces RETA, a metric that measures how good the highest-scoring responses of a reward model actually are, using a second model as judge. It gives practitioners a practical way to compare reward models and to detect when their rankings stop matching true quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RETA's validity depends on GPT-4 being a faithful human-preference oracle and on Llama2-7B-Chat being the right reference policy; validation against another GPT-4-based metric is circular, so RETA may measure agreement with GPT-4 rather than true reliability.","rationale":"After checking Theorem 1 and the estimator in Sec. 3.3, I find no fatal internal error: the limit in Eq. 4 follows from standard quantile convergence. The proof in Appendix B.1 contains a typo—p=1/n should be 1/(n+1)—but the limit is unaffected, and Eq. 5 is asymptotically consistent when both N and n grow. The empirical stability in Fig. 2 is plausible. However, the semantic claim that RETA measures reliability relative to 'actual human preferences' depends on the oracle and reference policy assumptions. Sec. 3.1 chooses GPT-4 as oracle and Llama2-7B-Chat as reference policy; Sec. 4.4 validates RETA only against 'Win Rate against GPT4', which uses the same judge and is therefore not an independent check of human alignment. Several tested RMs are trained from Llama2 or on Anthropic-Helpful, so the reference policy may favor them. Thus the central claim is conditional: if an independent human-labeled subset and an alternative reference policy preserve the RM ranking, the claim holds; otherwise RETA is a measure of agreement with GPT-4 on a fixed policy's outputs. This matches the reader's weakest assumption, so no verdict change is needed.","tokens_in":17862,"tokens_out":11403,"duration_ms":110279,"concrete_test":"Compute RETA for all 10 RMs on a random subset of 200 prompts from the released benchmark, using human preference scores (e.g., averaged over at least 3 annotators) as the oracle instead of GPT-4. Compare the resulting RM ranking to the GPT-4-oracle RETA ranking using Kendall's tau. If tau is below a pre-specified threshold such as 0.6, the oracle assumption fails and RETA measures agreement with GPT-4 rather than human preferences; if tau is high, the oracle concern is substantially settled, though reference-policy dependence should still be checked by regenerating responses with a different policy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RETA measures RM reliability—agreement with actual human preferences—rests on two premises introduced in Sec. 3.1: (i) GPT-4 scores are a faithful oracle for human judgment, and (ii) Llama2-7B-Chat at T=1 is a representative reference policy for the deployment distribution. The paper provides no human validation: the only external check in Sec. 4.4 and Appendix A.1 is 'Win Rate against GPT4', which uses the same GPT-4 judge as the oracle, so the validation is circular. If GPT-4 has systematic biases (e.g., length, style, or safety preferences) or if the reference policy is close to the training distribution of several tested RMs (Starling, RMv1-3 are Llama2-based), then RETA ranks models by agreement with GPT-4 on Llama2-generated outputs, not by human-aligned reliability. The theorem in Eq. 4 is mathematically correct and the estimator in Eq. 5 is asymptotically consistent, so this is an external-validity concern rather than an internal inconsistency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RETA (Reliable at η), a scalar metric for quantifying the reliability of reward models (RMs). RETA is defined as the limit, as the number of sampled responses per prompt grows, of the average oracle quality of the top η-quantile of responses ranked by the RM, normalized by the average oracle quality. The authors prove a limiting characterization (Theorem 1), propose a resampling-based estimator (Eq. 5), and build an end-to-end benchmarking pipeline that uses DPP-based prompt selection, Llama2-7B-Chat as the reference policy, and GPT-4 as the oracle. They benchmark 10 RMs on two datasets (Anthropic-Helpful and MT-Bench based) and show RETA curves, comparisons against BON and other metrics, and ablation of the normalization term. The paper also releases code and the benchmark.","tokens_in":18100,"tokens_out":8760,"duration_ms":79999,"significance":"If the proposed estimator and validation are sound, RETA addresses a genuinely important gap: a direct, budget-friendly reliability metric for RMs that does not require running RLHF. The quantile-average formulation is a sensible and principled improvement over best-of-n curves, which are unstable because they depend on a single selected response. The theoretical limit in Theorem 1 gives the metric a clear statistical interpretation, and the release of code and benchmark data supports reproducibility and community uptake. However, the paper's empirical validation is weakened by the use of GPT-4 both as the oracle and as the judge in the 'Win Rate' validation, and by an estimator whose unbiasedness is not rigorously established. These issues, while fixable, currently limit the confidence one can place in the reported rankings and in the claim that RETA captures human-aligned reliability.","major_comments":[{"comment":"The claim that the estimator in Eq. (5) is 'asymptotically unbiased' is not justified. The estimator is a ratio of two random quantities: the numerator is the resampled average of oracle scores over a random subset A of size n, and the denominator is the full-sample average over A_q. Linearity of expectation does not apply to ratios, and E[Σ_{a∈A} J_q(a) / Σ_{a∈A_q} J_q(a)] ≠ E[Σ_{a∈A} J_q(a)] / E[Σ_{a∈A_q} J_q(a)]. Moreover, the numerator and denominator are correlated because the subset is drawn without replacement from the same labeled set. The paper provides no bias or variance analysis; the statement in Appendix B.2 that the two adjacent order statistics 'will be arbitrarily close as n approaches infinity' is not a proof of unbiasedness. This is a load-bearing issue because the experimental RETA values in Figures 2–3 and Table 2 are computed with this estimator, and the pipeline's claimed validity depends on it. The authors should either provide a rigorous consistency and finite-sample bias analysis or replace the estimator with a provably unbiased one (e.g., a combinatorial estimator analogous to the BON sampler in Appendix B.3).","section":"Sec. 3.3, Eq. (5)"},{"comment":"The proof of Theorem 1 contains an equality that is not correct as stated: p(X_(j−1:n) ≥ Y_q(a) ≥ X_(j+1:n)) = 1/n. The event involves the random variable a with density p(a) and the sample X_1,...,X_n; unless a is explicitly one of the n sample points and the probability is taken over the full n-sample including a, the probability that Y_q(a) falls between two adjacent order statistics is not 1/n. The notation conflates 'a' as an index of a sampled response with 'a' as an integration variable. A rigorous proof should use exchangeability or a conditioning argument on the empirical distribution of the pairs (Y_q(a), J_q(a)) and the standard consistency of sample quantiles. The limiting statement itself is plausible and likely true, but the proof as written does not establish it. The authors should rewrite the proof with clear probabilistic formalism.","section":"Appendix B.1, proof of Theorem 1"},{"comment":"The validation of RETA against 'Win Rate against GPT4' is circular. As the authors note in Appendix A.1, the Win Rate metric is 'a variant of the RETA metric using this oracle labeling scheme': it selects the top 64 of 256 responses by the RM and asks GPT-4 to judge their quality against GPT-4's own answer. Since the same GPT-4 oracle is used both to compute RETA and to produce the Win Rate labels, the agreement observed in Table 2 does not provide evidence that RETA reflects human preferences independent of GPT-4's biases. To substantiate the claim that RETA measures reliability with respect to actual human judgment, the authors should provide external validation with human annotations (or at least an independent oracle) on a subset of the data.","section":"Sec. 4.4 and Appendix A.1"},{"comment":"The benchmark's validity as a measure of RM reliability rests on two assumptions that are not empirically checked: (i) GPT-4 scores are a faithful proxy for human preferences, and (ii) Llama2-7B-Chat at temperature T=1 is a representative reference policy for the deployment distribution. The latter is especially concerning because several of the tested RMs (Starling-7B, RMv1-3, RM5H-7B, RMEns-3x7B) are trained from Llama2-7B, so a reference policy from the same model family may bias the comparisons. At minimum, the authors should report a sensitivity analysis with a different reference policy and/or a small human agreement study to show that the RETA rankings are robust to these choices.","section":"Sec. 3.1 and Appendix B.4"}],"minor_comments":[{"comment":"The notation E_A in Eq. (5) is undefined; please state that A is a uniformly random subset of size n drawn without replacement from A_q, and clarify that the expectation is with respect to this resampling distribution.","section":"Sec. 3.3"},{"comment":"The choice of the range [3N^(2/3), 5N^(2/3)] is said to be 'empirically selected', but no details are provided. Please describe the selection procedure and report sensitivity of the final RETA values to the constants 3 and 5.","section":"Sec. 3.3"},{"comment":"The quantile function Θ(η) is defined as inf(x : F(x) ≥ 1−η), so 'continuity points of Θ' should be stated precisely with respect to the parameter η; also clarify that the theorem requires F to have no atom at the quantile or that the conditional expectation is defined appropriately in the presence of ties.","section":"Sec. 3.2, Theorem 1"},{"comment":"The statement that a_(⌊ηn⌋:n) and a_(⌊ηn⌋+1:n) 'will be arbitrarily close as n approaches infinity' is imprecise; the gap between adjacent order statistics is O_p(1/n) under a positive density, but the argument should be stated rigorously if it is used to justify asymptotic unbiasedness.","section":"Appendix B.2"},{"comment":"In the 'Oracle' row, the Win Rate against GPT4 is reported as 52.93%, which seems surprising because the oracle's own selections are compared against GPT-4's answers. Please explain why the score is not near 100% and what this row represents.","section":"Table 2"},{"comment":"The DPP sampling cost is listed as O(|Q0|k), but the MCMC algorithm cited in the text has complexity O(|Q0|k log(1/ε)); the table omits the approximation factor ε.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper introduces a useful metric and a concrete benchmark, but the two main technical pillars—the unbiasedness of the estimator and the proof of Theorem 1—need to be put on solid ground before publication. The circular validation is also a significant weakness; a human-label subset would substantially strengthen the paper. The reviewer believes the issues are addressable within the scope of a major revision, so reject is not recommended. Also note that the reference policy choice may create a distributional bias favoring Llama-2-based RMs; this should be discussed or controlled for."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Chen et al. propose RETA, a normalized top-η quantile reliability metric for reward models, plus a pipeline to compute it cheaply. The core idea is solid: instead of the noisy best-of-n curve, average the oracle quality of the top η fraction of RM-ranked responses, normalized by mean oracle score. Theorem 1 is standard conditional-expectation/quantile theory, but the packaging and the resampling-based estimation make it practical. The benchmarking pipeline (DPP prompt sampling, fixed response pool, no extra oracle calls per RM evaluation) is a real engineering contribution, and the experiments are informative — e.g., RAFT-3B tops test accuracy yet collapses on RETA, and ensembling improves reliability.\n\nThe soft spots are real but not fatal. First, Eq. (5) is called an 'asymptotic unbiased estimator' by 'linearity of expectation,' but that does not apply to a ratio of a resampled numerator and a fixed denominator sum. The B.1 proof establishes the limit of the unnormalized expectation, not unbiasedness of the ratio estimator. This is an overclaim; the estimator is probably consistent by Slutsky, but the unbiasedness claim needs proof or correction.\n\nSecond, the validation is largely internal. The 'Win Rate against GPT4' used as ground truth is itself a GPT-4-based comparison, and the paper admits it is 'a variant of the RETA metric.' So agreement between RETA and Win Rate shows robustness to the oracle scoring scheme, but it does not establish that GPT-4 approximates human preferences. No human-labeled validation is provided. Combined with Llama2-7B-Chat as the reference policy — close to the training distribution of several tested RMs — the external validity is unproven. The paper is honest about the reference policy being a simplification, but the GPT-4-as-human-proxy premise is load-bearing.\n\nThird, minor: the n-range constants and DPP λ are empirically chosen, the multi-turn benchmark uses only 20 prompts, and the arXiv text lacks a code/data link despite claiming release.\n\nOverall, this is a valuable paper for anyone auditing reward models. The metric is well-defined, the experiments are extensive, and the estimator issue is fixable. I'd send it to review, but require the authors to correct or clarify the unbiasedness claim and add at least a small human-labeled sanity check.","headline":"A useful, well-defined reliability metric with a clean pipeline, but the estimator's claimed unbiasedness is unjustified and the validation never leaves the GPT-4 oracle.","tokens_in":18650,"tokens_out":3947,"would_cite":true,"duration_ms":34769,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes RETA, a normalized metric that measures reward-model reliability through the average oracle quality of the top η quantile of ranked responses.","keywords":["reward model reliability","RETA metric","best-of-n sampling","reward overoptimization","LLM alignment evaluation","quantile-based metric","oracle scoring","reward benchmarking"],"falsifier":"Take the released benchmark response sets, re-score them with human annotators instead of GPT-4, recompute RETA for the ten tested reward models, and check whether the model rankings and RETA curves stay the same; if they change materially, RETA's validity rests on the GPT-4 oracle assumption.","tokens_in":17659,"feed_emoji":"📊","tokens_out":9167,"duration_ms":76107,"temperature":0.7,"pith_summary":"The paper proposes a single number, Reliability at η (RETA), that measures how trustworthy a reward model is: take the top η fraction of responses the model ranks highest for a prompt, average the oracle-judged quality of those responses, and divide by the average quality of all responses. The central claim is that this ratio has a well-defined limiting value, converges quickly with finite response pools, and, once normalized, is largely insensitive to which prompts are included. RETA is meant to replace accuracy and best-of-n evaluations, which the paper argues either miss reliability or wobble with the ranking of a single response. Because the oracle labels a fixed response set once, every new reward model can be scored on the same pre-labeled pool without additional labeling cost, which makes reliability benchmarking affordable.","feed_headline":"New metric RETA reveals when AI reward models stop being reliable","feed_subtitle":"It ranks reward models from one pre-labeled response set, with no extra oracle calls per model.","key_machinery":"The load-bearing object is the BETA subset — the top $\\eta$ fraction of responses selected by the reward model $Y$ — together with the normalized average oracle score over that subset. Theorem 1 shows that as the response pool grows, this average converges to $\\mathbb{E}_a[J_q(a)\\mid Y_q(a)\\ge \\Theta(\\eta)]/\\mathbb{E}_a[J_q(a)]$, where $\\Theta(\\eta)$ is the upper-$\\eta$ quantile of reward scores. The denominator $\\mathbb{E}_a[J_q(a)]$ is what makes RETA interpretable: it puts the random baseline at 1 and removes much of the prompt-to-prompt variation in oracle scores. The estimation scheme chooses resampling sizes proportional to $N^{2/3}$, averaging over $n\\in[3N^{2/3},5N^{2/3}]$, to balance bias from replacing the true quantile with the sample quantile against variance from Monte Carlo estimation.","core_discovery":"The paper's central discovery is Theorem 1: as the candidate pool grows, RETA converges to the conditional expectation of oracle quality among responses whose reward score is at or above the upper-$\\eta$ quantile, divided by the unconditional mean oracle quality, $$\\mathrm{RETA}_Y(\\eta)=\\frac{1}{|\\mathcal{Q}|}\\sum_{q\\in\\mathcal{Q}}\\frac{\\mathbb{E}_a[J_q(a)\\mid Y_q(a)\\ge \\Theta(\\eta)]}{\\mathbb{E}_a[J_q(a)]}.$$ The value 1 is the random baseline, so values above 1 mean the reward model's top quantile beats average response quality, and a RETA curve across $\\eta$ shows where that advantage decays. The paper further claims that the empirical estimator of Eq. (5), averaging over resampled subsets of size $n$ in $[3N^{2/3},5N^{2/3}]$, is asymptotically unbiased and that RETA(1/4) curves converge visibly to their limits; on the two constructed benchmarks it uses the curves to rank publicly available and in-house reward models and to identify which quantile is safest for selection.","pith_inferences":["Editorial inference: RETA is best read as reliability on a fixed reference-policy distribution, not as a universal property of the reward model; a model that looks reliable under Llama2-7B-Chat responses could fail on a different response distribution.","Editorial inference: the identical recipe — oracle-scored candidates, selector scoring, average quality of the top $\\eta$ — applies to any selection system beyond reward models, such as retrieval rankers or LLM-as-a-judge pipelines, provided an oracle score is available for the candidate pool.","Editorial inference: the paper's future-work proposal implies a sharper test of the metric's usefulness: if training a reward model to maximize RETA does not improve downstream RLHF policy quality, then RETA would be measuring oracle agreement rather than alignment.","Editorial inference: because the oracle is GPT-4, RETA as implemented measures agreement with GPT-4's notion of quality; recomputing the same benchmark with human labels or multiple oracle models would reveal how much of the ranking is oracle-dependent."],"forward_implications":["RETA lets a lab rank any number of reward models using one pre-labeled response set: each new RM only scores existing responses, so there is no incremental oracle cost.","The RETA curve gives an operational stopping rule for rejection sampling: the $\\eta$ at which the curve stops rising is the quantile below which selecting even fewer responses no longer helps.","Accuracy alone can be misleading as a reliability signal: RAFT-3B, the highest test-accuracy RM in the study, is among the weakest under RETA, so RETA captures behavior that accuracy benchmarks do not.","The normalized metric is the right one for comparing across prompts: without the denominator, RETA drifts with prompt perplexity; with it, the relation flattens.","In the paper's experiments, ensembling several reward models keeps RETA close to the best member, pointing to ensembling as a practical mitigation for reward hacking."],"supporting_citations":[{"why":"Defines the best-of-n (BON) curve and its unbiased estimator; RETA is designed to improve on BON's variance and single-response dependence.","marker":"[26]"},{"why":"Establishes best-of-n/rejection sampling as an RM evaluation approach and motivates reliability assessment through overoptimization scaling laws.","marker":"[18]"},{"why":"Supplies the RewardBench accuracy benchmark from which the tested reward models are drawn and against which RETA is compared.","marker":"[23]"},{"why":"Provides the Anthropic-Helpful prompt set that serves as the main Reliability-on-Helpfulness benchmark.","marker":"[7]"},{"why":"Defines the reference policy, Llama2-7B-Chat, used to generate candidate responses for RETA evaluation.","marker":"[37]"},{"why":"Supplies the quantile-convergence lemma (Proposition 5) used in the proof of Theorem 1.","marker":"[17]"},{"why":"Justifies the resampling-size range n proportional to N^{2/3} that the RETA estimator averages over.","marker":"[11]"}],"fun_headline_variants":["RETA curve pinpoints the safe quantile to sample from LLMs","RETA metric measures when reward models lose reliability","New metric RETA ranks reward models with no extra oracle costs","RETA: know when your reward model stops being trustworthy","RETA reliability metric needs only one labeled response set"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The validity of RETA depends on GPT-4's scores serving as a faithful oracle for genuine human preferences, with responses generated by Llama2-7B-Chat at temperature 1 representing the deployment distribution; if either premise fails, RETA measures agreement with GPT-4's judgment on that particular response distribution rather than true reliability.","fun_headline_variants_meta":{"raw":{"variants":["RETA curve pinpoints the safe quantile to sample from LLMs","RETA metric measures when reward models lose reliability","New metric RETA ranks reward models with no extra oracle costs","RETA: know when your reward model stops being trustworthy","RETA reliability metric needs only one labeled response set"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000597,"raw_usage":{"total_tokens":2813,"prompt_tokens":985,"completion_tokens":1828,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":1746}},"tokens_in":601,"tokens_out":1828,"duration_ms":13281,"temperature":1.0,"reasoning_tokens":1746,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:39:04.373125+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the released benchmark response sets, re-score them with human annotators instead of GPT-4, recompute RETA for the ten tested reward models, and check whether the model rankings and RETA curves stay the same; if they change materially, RETA's validity rests on the GPT-4 oracle assumption.","supporting_citations":[{"cited_title":"A modern approach to probability theory","cited_arxiv_id":null,"evidence_quote":"Supplies the quantile-convergence lemma (Proposition 5) used in the proof of Theorem 1."},{"cited_title":"Variance estimation for sample quantiles using the m out of n bootstrap","cited_arxiv_id":null,"evidence_quote":"Justifies the resampling-size range n proportional to N^{2/3} that the RETA estimator averages over."}],"review_version":1}