{"id":"ac74ffeb-a809-49e2-9078-2c0e46d4e165","arxiv_id":"2505.20556","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Pessimistic fine-tuning of reward models against rejection-sampling policies lets RLHF agents optimize greedily without KL regularization and still avoid reward hacking.","lead":"This paper proposes PET, a method that fine-tunes a reward model to be pessimistic about the exact policies it will be used to evaluate, so that RLHF agents can be optimized greedily without KL regularization. On summarization and sentiment tasks, the resulting policies beat several baselines and even performed well at high KL divergence.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.3's proof in Appendix A.3 applies reward optimality to an expression that is not the PET objective; the central no-regularization guarantee is unsupported as written.","rationale":"The paper's central claim is that a PET-fine-tuned pessimistic reward makes regularization-free policy optimization safe from reward hacking. The key support for that claim is Theorem 3.3, and the proof of that theorem has a load-bearing gap: optimality of r_hat for h(r) does not justify the inequality used to drop the reward-model discrepancy term. Because the rejection-sampling policy is itself a function of the reward being minimized, the envelope-style step in Appendix A.3 cannot be applied as written. This is an internal mathematical gap, not a disagreement with the field's consensus. The empirical direction is still meaningful: the adversarial rejection-sampling objective is plausible, and Table 2's 40.8% win rate at KL 114.0 is a useful data point. However, with the proof gap and no convergence guarantee for Algorithm 1, the theoretical basis for the abstract's 'without relying on any regularization' claim is not established, and the evaluation remains thin (two seeds, no code release). The reader's REJECT verdict is consistent with this assessment, so I recommend leaving it unchanged.","tokens_in":17485,"tokens_out":20459,"duration_ms":200360,"concrete_test":"Re-derive the second displayed inequality in Appendix A.3 using only h(r_hat) <= h(r*) and pi_hat = pi_RS(r_hat); check algebraically whether f(r_hat, pi_hat) <= f(r*, pi_hat) follows. If it does not, search small finite MDPs (for example |A|=3, |R|<=3, n=2, varying dataset proportions and beta) for an instance where the theorem's claimed bound is violated. Report either a concrete counterexample or a corrected proof of Theorem 3.3.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The proof of Theorem 3.3 needs the inequality f(r_hat, pi_hat) <= f(r*, pi_hat), where f(r, pi) = V_r(pi) - V_r(pi_ref) + beta * L_D(r). The paper derives this from optimality of r_hat for Eq. 2. But Eq. 2 minimizes h(r) = f(r, pi_RS(r)), and pi_hat = pi_RS(r_hat), so optimality gives only h(r_hat) <= h(r*), i.e. f(r_hat, pi_hat) <= f(r*, pi_RS(r*)). This does not imply the needed inequality: because pi_RS(r*) maximizes V_{r*}, h(r*) >= f(r*, pi_hat), so the optimality condition can hold while f(r_hat, pi_hat) > f(r*, pi_hat). The dropped term is positive exactly when the pessimistic reward's RS policy is worse on the true reward, the situation the theorem must control. Consequently the bound in Theorem 3.3 does not follow from the stated assumptions as written. The same gap severs the link to the practical claim: the theorem concerns an exact minimizer and rejection-sampling policies, whereas Algorithm 1 has no convergence guarantee and Table 2 uses PPO, so the 'without any regularization' statement is not backed by the theory.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PET, a pessimistic reward fine-tuning method for offline RLHF. PET is formulated as a minimax problem in which a reward model is trained to assign low relative scores to rejection-sampling policies induced by that same reward model, while keeping low prediction loss on the preference dataset. The authors state Theorem 3.3, which claims that rejection sampling on the PET reward attains performance comparable to any dataset-covered rejection-sampling policy without KL regularization. They then test PET with rejection sampling and with PPO on TL;DR summarization and IMDB, reporting win rates against human summaries and ground-truth sentiment likelihoods, and conclude that a greedy policy on the pessimistic reward avoids reward hacking without any regularization.","tokens_in":17686,"tokens_out":9510,"duration_ms":90711,"significance":"If Theorem 3.3 were sound, the paper would make a substantive contribution: it would show that pessimism can be injected at the reward-model stage rather than through KL regularization, with a finite-sample gap bound governed by dataset coverage. The paper also deserves credit for including a ground-truth evaluation on IMDB and direct pairwise comparisons against baselines, which provide a non-circular check on the central claim. However, the main proof contains a real derivation gap, the algorithm is not shown to solve the theoretical objective, and the headline empirical claim rests on a policy class and an experimental regime not covered by the theory and on very limited seed information.","major_comments":[{"comment":"The proof requires the inequality V_{\\hat r}(\\hat\\pi)-V_{\\hat r}(\\pi_\\mathrm{ref})+\\beta L_D(\\hat r) \\le V_{r^*}(\\hat\\pi)-V_{\\hat r}(\\pi_\\mathrm{ref})+\\beta L_D(r^*), which is then used to drop a bracket; equivalently the proof needs V_{\\hat r}(\\hat\\pi)+\\beta L_D(\\hat r) \\le V_{r^*}(\\hat\\pi)+\\beta L_D(r^*). The paper justifies this by saying that \\hat r minimizes Eq. (2). But Eq. (2) minimizes h(r)=V_r(\\pi_{\\mathrm{RS}}(r))-V_r(\\pi_\\mathrm{ref})+\\beta L_D(r), and since \\pi_{\\mathrm{RS}}(r^*) maximizes V_{r^*}, optimality of \\hat r gives only h(\\hat r)\\le V_{r^*}(\\pi_{\\mathrm{RS}}(r^*))-V_{r^*}(\\pi_\\mathrm{ref})+\\beta L_D(r^*), whose right-hand side is at least V_{r^*}(\\hat\\pi)-V_{r^*}(\\pi_\\mathrm{ref})+\\beta L_D(r^*). The needed inequality has the smaller right-hand side V_{r^*}(\\hat\\pi)-V_{\\hat r}(\\pi_\\mathrm{ref})+\\beta L_D(r^*), so it does not follow. The dropped term can be positive precisely when \\pi_{\\mathrm{RS}}(\\hat r) is worse under the true reward than \\pi_{\\mathrm{RS}}(r^*), which is the situation the theorem is supposed to control. As written, the proof does not establish Theorem 3.3.","section":"Appendix A.3 (proof of Theorem 3.3)"},{"comment":"The derivation that PET is stochastic gradient descent on h(r) uses the claim that \\nabla_\\pi f(r,\\pi)=0 at \\pi=\\pi_{\\mathrm{RS}}(r) because \\pi_{\\mathrm{RS}} maximizes f over \\Pi_{\\mathrm{RS}}. Rejection sampling selects the argmax among n discrete samples, so the induced policy is not differentiable in the reward parameters and is not an unconstrained interior maximizer; the stationarity condition \\nabla_\\pi f=0 is therefore not justified. Additionally, Algorithm 1 provides only stochastic gradient updates with no convergence guarantee, so the returned reward model is not shown to be the exact minimizer of Eq. (2) used in Theorem 3.3. This severs the link between the practical algorithm and the theoretical guarantee.","section":"Appendix A.1 (PET as SGD on Eq. (2))"},{"comment":"The headline empirical claim is that an unregularized PPO policy trained on the pessimistic reward achieves 40.8% win rate at KL 114.0, in contrast to 7.2% for unregularized PPO on the proxy reward. Theorem 3.3 applies only to rejection-sampling policies in \\Pi_{\\mathrm{RS}} that are covered by the dataset; PPO policies are not in that class, so the theory does not license the no-regularization claim for PPO. Moreover, Table 2 reports no standard errors or seed count; the two-seed analysis in Table 6 covers PET-PPO versus baselines but not the regularized-versus-unregularized comparison in Table 2. The central empirical contrast is thus supported by at most two seeds and is presented without any uncertainty quantification.","section":"Section 4.3 (Table 2) and Theorem 3.3"}],"minor_comments":[{"comment":"The caption says the results are on the IMDB dataset, but the evaluation metric is win rate against human responses and the surrounding text in Section 4.2 describes the TL;DR summarization setting; please reconcile the caption with the dataset actually used.","section":"Table 1 caption"},{"comment":"The single-run numbers in Table 3 differ from the two-seed means in Table 6 in the same metrics (for example DPO summarization 38.8 in Table 3 versus 37.5 +/- 1.3 in Table 6, and PET-PPO IMDB 98.3 versus 96.1 +/- 2.2). The paper should state explicitly which table reports which seeds and provide the per-seed values for all reported results.","section":"Tables 3 and 6"},{"comment":"The text defines sigmoid(y):=1/(1-exp(-y)); the standard definition is 1/(1+exp(-y)), which is needed for the Bradley-Terry likelihood to be a valid probability.","section":"Section 2.1 (sigmoid definition)"},{"comment":"The loss expression r_t(x_i, \\pi_t(x_i, a_i)) mixes response variables with policy notation; it should be written in terms of the sampled responses a_i and a_ref,i, for example r_t(x_i,a_i) and r_t(x_i,a_ref,i).","section":"Algorithm 1, line 7"},{"comment":"The opening of the proof is incomplete: 'Let \\hat r be the reward solution to \\hat r \\in \\arg\\min_{r\\in R} V...' cuts off and should be rewritten as a complete statement of the minimizer and the objective.","section":"Appendix A.3, first sentence"}],"recommendation":"reject","confidential_remarks":"The proof gap in Appendix A.3 is not a local typo; the asserted optimality step does not follow because the rejection-sampling policy depends on the reward being optimized. Combined with the lack of a convergence guarantee for Algorithm 1 and the theory-practice mismatch for unregularized PPO, the paper's central claims are not established as written. A substantially different proof or additional assumptions would be needed, which in my view is outside the scope of a minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read this one. The mechanism is genuinely new and worth thinking about: PET adversarially fine-tunes a reward model against the rejection-sampling policy induced by the current reward, with no KL term, and the empirical result in Table 2 is a real data point — unregularized PPO on the pessimistic reward gets 40.8% win rate at KL 114, while the proxy reward collapses to 7.2% at KL 192. On IMDB, where sentiment is ground truth, PET-PPO beats the baselines clearly (98.3 vs 95.0 for DPO), and that evaluation is non-circular. Credit where it's due: Proposition 2.1 is trivial but useful, and the reduction from the minimax objective to the minimization in Eq. 2 is a clean trick that makes the adversarially-trained reward easy to implement.\n\nThe soft spot is load-bearing and it's in the theory. The proof of Theorem 3.3 in Appendix A.3 needs the inequality f(hat r, hat pi) <= f(r*, hat pi). The paper derives it from optimality of hat r for Eq. 2, but Eq. 2 minimizes h(r) = f(r, pi_RS(r)), and pi_RS(r*) is the RS policy for r*, not for hat r. Because pi_RS(r*) maximizes V_{r*}, h(r*) >= f(r*, hat pi), so the optimality of hat r does not give the needed inequality. The dropped term is positive exactly when the pessimistic reward's RS policy is worse on the true reward — the situation the theorem is supposed to control. So Theorem 3.3 is not proven as written. Also, Appendix A.1 claims the gradient of f with respect to pi is zero at the RS policy because it is the maximizer; that is also not justified, since the RS policies form a discrete set and the maximizer is at a boundary. None of this kills the empirical direction, but it does mean the abstract's 'without relying on any regularization' claim has no theoretical support in this draft.\n\nEmpirically it is thin: two seeds, no code, a caption error in Table 1 (it says IMDB but the metric is win rate against human summaries), and Table 3's IMDB PET-PPO is 98.3 while Table 6 reports 96.1 ± 2.2 for the same method. They skip the InferenceTimePessimism baseline for resource reasons, which is acceptable but weakens the comparison. To their credit, they state the limitations up front: offline only, small models, subset of tasks.\n\nBottom line: the idea is worth a serious look, and the empirical finding on IMDB is interesting enough to warrant referee time, but the theory needs real repair. Send it to peer review with the expectation that the proof of Theorem 3.3 must be fixed or the claims substantially softened; otherwise it's a reject.","headline":"Novel and empirically interesting pessimistic-reward fine-tuning idea, but the central theorem's proof has a load-bearing gap and the no-regularization claim is not supported as written.","tokens_in":18309,"tokens_out":3315,"would_cite":false,"duration_ms":31869,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a reward model made pessimistic by adversarial fine-tuning against rejection sampling lets offline RLHF greedily optimize policies without KL regularization.","keywords":["pessimistic reward model","reward hacking","RLHF","rejection sampling","KL regularization","offline reinforcement learning","adversarial training","TL;DR summarization"],"falsifier":"Run unregularized PPO on a PET-trained reward on a held-out task, measure the true reward (or human preference) of the final policy as a function of KL from the reference policy; if win rate drops as KL grows, as it does with a proxy reward, the claim that pessimistic fine-tuning alone prevents reward hacking is false.","tokens_in":17222,"feed_emoji":"🎯","tokens_out":5142,"duration_ms":48599,"temperature":0.7,"pith_summary":"This paper tries to show that reward hacking in offline RLHF can be prevented by making the reward model itself pessimistic, rather than by regularizing the policy toward the dataset. Its method, PET, fine-tunes a proxy reward so that it gives low relative scores to the policy produced by rejection sampling on that reward, while still fitting the preference data. The central claim is that once such a reward model is learned, a greedy policy optimizer (PPO with no KL term) is safe and can even find high-performing policies far from the dataset. On the TL;DR summarization task, unregularized PPO on the PET reward reaches a 40.8% win rate against human summaries at KL divergence 114.0, whereas the same greedy run on the proxy reward collapses to a 7.2% win rate. This matters because KL regularization is widely treated as necessary in RLHF, and this work argues it is a crude substitute for pessimism that excludes good out-of-distribution policies.","feed_headline":"No KL needed: pessimistic reward stops reward hacking in RLHF","feed_subtitle":"Fine-tuning the reward against a rejection-sampling policy lets greedy PPO keep a 40.8% win rate at KL 114.","key_machinery":"The central object is the PET objective, Equation 2: minimize over reward models $r$ the relative score $V_r(\\mathrm{RS}(r)) - V_r(\\pi_{\\mathrm{ref}})$ plus $\\beta$ times the preference prediction loss $L_D(r)$. The rejection-sampling policy $\\mathrm{RS}(r)$ is the actor; Proposition 2.1 guarantees that this particular policy maximizes $V_r$ within the RS policy class, so the original minimax formulation becomes a single minimization. The dataset-loss term $\\beta L_D(r)$ is what keeps pessimism from becoming over-pessimism, and the coverage coefficient defined from the dataset distribution controls how far outside the data the guarantee extends.","core_discovery":"PET solves a minimax problem in which the critic (reward model) tries to minimize the gap between the rejection-sampling policy and a reference policy, subject to keeping low prediction loss on the preference dataset. Proposition 2.1 collapses the game to a pure minimization over rewards: because for any reward the rejection-sampling policy is already optimal among rejection-sampling policies on that reward, the actor step can be folded into the reward update. The paper's Theorem 3.3 states that the resulting pessimistic reward, when used for rejection sampling, matches any dataset-covered rejection-sampling policy up to a gap that shrinks as $N^{-1/2}$ in the number of preference tuples. Empirically, the paper finds that greedy PPO on the PET reward outperforms the same on the proxy reward and matches or beats DPO, RPO, and chi-PO baselines, while running at a much higher KL divergence.","pith_inferences":["If PET rewards are trustworthy at high KL, the same fine-tuning could replace KL constraints in other alignment loops, such as iterative best-of-N or online preference collection, where exploration is currently capped.","Because the adversary is a rejection-sampling policy, the method's guarantees depend on that specific actor; extending the idea to arbitrary policy classes would need a different tractable 'best response'.","A direct test of pessimism is to measure the PET reward's ranking accuracy on deliberately adversarially sampled responses; if greedy search can still find overestimated responses, the method fails by its own criterion."],"forward_implications":["Greedy policy optimization on a PET reward needs no KL term, so high-performing policies far from the dataset are no longer excluded.","Rejection sampling (best-of-N) on a PET reward beats the same procedure on the proxy reward, e.g., 39.2% versus 32.0% win rate on TL;DR.","PPO on the PET reward is competitive with or better than KL-regularized PPO, DPO, RPO, and chi-PO in the reported evaluations.","The theoretical bound gives a sample-efficiency guarantee: with dataset coverage, performance gap shrinks like $N^{-1/2}$."],"supporting_citations":[{"why":"Supplies the adversarial-training formulation and proof technique that Theorem 3.3 extends, and the RPO baseline.","marker":"[Liu et al., 2024b]"},{"why":"Gives the coverage coefficient used in Definition 3.2 and the offline preference-learning framework.","marker":"[Zhan et al., 2023]"},{"why":"Source of the adversarially trained actor-critic idea and the covering-number tool used in the bound.","marker":"[Cheng et al., 2022]"},{"why":"Provides theoretical guarantees on best-of-N alignment that motivate the rejection-sampling actor.","marker":"[Beirami et al., 2024]"},{"why":"Documents reward overoptimization that the paper aims to prevent, and its scaling-law evaluation setup.","marker":"[Gao et al., 2023]"},{"why":"Introduces the practical rejection sampling / best-of-N procedure used as the actor and a baseline.","marker":"[Touvron et al., 2023]"},{"why":"The DPO baseline and the implicit-reward framework the paper contrasts against.","marker":"[Rafailov et al., 2023]"},{"why":"The chi-PO baseline, a KL-free direct alignment method whose guarantee the paper compares with its own.","marker":"[Huang et al., 2024a]"}],"fun_headline_variants":["PET: pessimistic reward model kills reward hacking without KL","No KL needed: PET learns a pessimistic reward that beats reward hacking","Pessimistic reward model nixes KL regularization in RLHF","Reward hacking? PET's pessimistic reward makes it moot, no KL needed","High KL, high performance: PET pessimistic reward skips RLHF regularization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof assumes PET returns a reward model that exactly minimizes the pessimistic objective (or otherwise satisfies the optimality condition used in the proof), but Algorithm 1 is only a stochastic gradient update with no convergence guarantee.","fun_headline_variants_meta":{"raw":{"variants":["PET: pessimistic reward model kills reward hacking without KL","No KL needed: PET learns a pessimistic reward that beats reward hacking","Pessimistic reward model nixes KL regularization in RLHF","Reward hacking? PET's pessimistic reward makes it moot, no KL needed","High KL, high performance: PET pessimistic reward skips RLHF regularization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000783,"raw_usage":{"total_tokens":3441,"prompt_tokens":916,"completion_tokens":2525,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":2434}},"tokens_in":532,"tokens_out":2525,"duration_ms":16966,"temperature":1.0,"reasoning_tokens":2434,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:52:34.382598+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run unregularized PPO on a PET-trained reward on a held-out task, measure the true reward (or human preference) of the final policy as a function of KL from the reference policy; if win rate drops as KL grows, as it does with a proxy reward, the claim that pessimistic fine-tuning alone prevents reward hacking is false.","supporting_citations":[],"review_version":1}