Pith.

Machine Learning · arXiv:2611.03244 v1 · submitted 2 Nov 2026 · 53 references

Forecasting How RLHF Responds to Reward-Model Edits Before Training

Zixiao Wang, Duzhen Zhang, Xingbo Du, Dongxia Wu, Le Song

The paper says

“Responses with headers, lists and greater length receive high scores from reward models (RMs), and RLHF policies learn to produce these features even when true quality does not improve.”

more of the abstract

Practitioners therefore edit their RMs, but they learn whether an edit works only after an RL run. We show that the response of a policy to an RM edit can be forecast before training, from samples of the current policy and the scores of the old and new RM. If we read the RM as a set of prices, the first-order theory of KL-regularized RL implies that each feature moves in proportion to its within-prompt covariance with the change in reward. We implant controlled price changes into retrained RMs under a 2^(4-1) fractional factorial design with a self-distilled placebo, run 54 RL runs from six parent policies, and fix every forecast before the runs. With the covariance predictor, we predict the direction of all 20 changes in supply whose forecast has the same sign at every parent and explain 82% of the standardized change. With counterfactual prices, the classical alternative, we explain 50%. The forecast also holds under GRPO and for real RM upgrades, where we explain 66% out of sample with a coefficient frozen from the factorial runs. For rent, the gap between the proxy reward and a gold reward from stronger judges, we rank RM edits correctly with the same covariance over the first 30 steps (Spearman 0.94). This measures the horizon of the forecast: by step 200, the ranking of the factorial edits reverses under both RLOO and GRPO, while the real upgrades keep their order. When we vary the KL coefficient beta, the initial response does not depend on beta, and the response saturates below the KL equilibrium. In one experiment, the long-run response to a price rise is consistent with an additional implicit regularizer beta_0 of about 0.003 to 0.007.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 3 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A pre-registered, controlled test of a classical covariance law for forecasting RLHF supply; direction and early-window claims hold, but the 200-step magnitude rests on an unverified stability assumption the authors themselves measure. This line is wrong

2 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 53 references say.

The paper says, at §5, Eq. (3), Table 12; §6; Appendix M

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.82):

“We can measure this change at a checkpoint but cannot yet forecast it before RL.”

Pith says: Objection 1 of 2 · major

The 200-step supply validation assumes that the branch-point covariance remains the governing incentive through steps 160-200, but the paper's own data show that the analogous incentive is unstable. Rent rankings reverse by step 200 under both RLOO and GRPO (Section 6), and Appendix M reports that the formatting covariance at step 400 grows to 2.08 times its branch-point value at β=0.02 and changes sign (-2.06) at β=0.005 after a price cut.

more of objection 1

In the factorial runs, the two price-cut arms overshoot the forecast by factors of 2.71 and 2.11 (Table 12), in the direction a drifting incentive would produce, while in the symmetric-shock batch the same β=0.005 cut arm reaches only about 0.53 of the forecast (Table 21); the authors acknowledge the batch difference but do not explain it. The 20/20 sign agreement does not settle the point, because early movement can determine the endpoint even if the covariance no longer describes steps 160-200. The paper concedes the crux ([b211], [b212]): the incentive change can be measured at a checkpoint but cannot yet be forecast before training, which is exactly the premise on which the 200-step size claim rests. I am not asking for the covariance to be constant over the window: the direction result can survive drift, and the out-of-sample upgrade evidence with frozen κ is solid. As written, however, the abstract's '82% of the standardized change after 200 steps' overstates what Eq. (3) mechanistically supports. Either recompute E_x[Cov_x(f_k, Δr)] at checkpoints inside the factorial runs and quantify its drift and its effect on the explained share, or reframe the headline so that the before-training forecast covers the early response, with the 200-step supply claim presented as direction-robust while its sizes depend on an unforecastable incentive drift.

The paper says, at Abstract; Table 2

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The headline 82% share is an in-sample calibration: one coefficient κ per feature is fitted to the same six-arm factorial data on which the share is computed (Section 4 and the Figure 1 caption), whereas only the 66% frozen-coefficient result on the three real upgrades is a true out-of-sample prediction.

more of objection 2

The body discloses this, and the upgrade test is the right evidence for the 'before training' claim, but the abstract presents the 82% without the qualifier. Please mark the 82% as a fitted, in-sample share at its first occurrence, or lead the abstract with the frozen-coefficient number.

6 minor objections

The paper says, at Appendix M, Table 22

Pith says: minor objection 1 of 6

The pre-specified hypothesis S6, which fixes the ratio of early slopes after a price cut and a price rise to within [0.67, 1.5] of the forecast, failed at all three values of β (observed ratios 0.44-0.61), and this first-order-theory miss is reported only in an appendix; the main-text scope discussion should state that the early response to a price cut was slower than forecast.

The paper says, at §7 and Abstract

Pith says: minor objection 2 of 6

The range β0 ≈ 0.003-0.007 in the abstract should be read with the acknowledged uncertainty of the fits: the joint estimate is 0.0065 with a parent range of 0.0034-0.0186, the prospective rise-arm value is 0.0029, the cut arm yields β0 = 0, and the pre-specified acceptance interval is admittedly weak ([b346]); please carry at least one of these numbers into the abstract.

show 4 more

The paper says, at Appendix D and Appendix Q

Pith says: minor objection 3 of 6

Because the predictor is estimated from only 16 responses per prompt, a sentence on the prompt-level sampling error of Cov_x(f, Δr) and on the number of prompts needed for a reliable sign would make the recipe in Appendix Q actionable for practitioners.

The paper says, at §4

Pith says: minor objection 4 of 6

The factorial notation '24−1' should read '2^(4−1)' to avoid being misread as a 23-run design.

The paper says, at §5 and Table 11

Pith says: minor objection 5 of 6

The covariance-versus-price comparison is partly an artifact of the feature set, since length is a feature but, by the authors' own definition, not a priced good; on the two goods that both predictors can price, the shares are much closer (0.71 and 0.87 for headers, 0.91 and 0.66 for list markers), and this nuance deserves a sentence in the main text rather than only in the appendix table.

The paper says, at §5 and Appendix H

Pith says: minor objection 6 of 6

The claim of 'all 20 stable changes' applies to 6 of the 8 factorial arms, because the two arms that hit the summary dose are analyzed descriptively under the pre-specified runaway rule; the abstract states the stable-sign condition but not the arm exclusion, so please make the 6-of-8 scope visible at the claim's first occurrence.

What would settle it

Recompute E_x[Cov_x(f_k, Δr)] on samples of each of the six factorial arms and the placebo at steps 50, 100, and 200, using the same 500 held-out prompts, 16 samples per prompt, and the same standardized scores used for the branch-point forecasts. Compare signs and normalized magnitudes to the branch-point values used in Table 12. If the covariance sign for F1a, F1b, or length reverses before step 200, or if the magnitude drifts by more than a factor of two in any of the six arms, then the end-window supply change is not governed by the branch-point covariance and Eq. 3's 200-step validation cannot be attributed to the stated mechanism. If all signs persist and magnitudes stay close to the branch-point values, the stability premise holds and this concern is resolved.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 3 Nov 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 59 s · 30,173 tokens read, 7,603 written
  3. Editor wrote the verdict above 40 s · 31,135 tokens read, 6,214 written
  4. Referee major revision: 2 major objections, 6 minor 2 min 2 s · 30,457 tokens read, 20,539 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Two things to know. This is the most carefully controlled empirical paper on RLHF reward-model editing I have seen: analysis plan frozen before runs, a decision log disclosing four post-freeze design changes and a cancelled batch, a self-distilled placebo, a fractional factorial with validation thresholds, six parents, and an out-of-sample upgrade test with a frozen coefficient. The central claim—feature supply reacts to an RM edit like Δs ≈ κ·Ex[Cov_x(f, Δr)], computable before training—holds for direction and for the early window. The 200-step magnitude claim is weaker than the abstract suggests, because the mechanism is unverified at that horizon.

What is new: the covariance identity is classical (Hotelling, Kubo, Price; the paper credits all three), but reducing RLHF response to a pre-training forecast from branch-point samples plus two score sets, validated with controlled soft-label price implants, is genuinely new. The sharpest result is length: the price predictor explains 7% of the length response, the covariance predictor 84%, exactly because Δr carries changes outside the priced feature span.

What is good: the design is unusually honest. Sign forecasts were fixed before the runs; later analyses are marked exploratory; failed hypotheses (R2, R3, S2, the Llama overshoot) are reported plainly. The rent forecast with a measured horizon—Spearman 0.94 at steps 20–30, reversal by step 200 under RLOO and GRPO—is a real result, not a masked failure. The direction forecast is parameter-free and not circular: it uses only policy samples and the two RMs' scores, no outcome data.

Soft spots, in proportion. The 82% share fits one κ per feature to the same factorial data: calibration, not construction. The frozen-κ transfer to real upgrades (66%) is genuine out-of-sample evidence and mitigates this. The load-bearing caveat is the horizon. Eq. 3 is a local first-order statement, yet the headline reports the steps 160–200 window. The paper's own Appendix M shows the analogous covariance for formatting doubles (2.08×) after a price cut at β=0.02 and changes sign at β=0.005; Table 12 shows the price-cut arms overshooting by 2.7× and 2.1×. The authors say they can measure this at a checkpoint but cannot forecast it before RL ([b211]–[b212])—honest, but it means the supply forecast surviving to step 200 has no mechanism; early movement may determine the endpoint. That does not break the paper: direction, early window, and transfer stand. But the abstract's "all 20 changes ... 82%" overstates what is established at the end window.

Minor: one 1.7B testbed; the foundational testbed is an unpublished same-group preprint; no released code. The β₀ "implicit regularizer" is a fitted effective coefficient with a weak criterion, fails on price cuts, and is labeled a conjecture—acceptable, but the name overpromises.

Who this is for: practitioners iterating reward models (Appendix Q is directly usable) and anyone wanting a model of pre-registered empirical RLHF. It deserves a serious referee. I would push on the end-window mechanism and ask that the horizon caveat sit in the abstract; with that, I would take it.

The strongest objection

Eq. 3 is derived in Section 3 from two local statements: the equilibrium tilt of KL-regularized RL (Eq. 2) and the one-step Price selection term (Appendix C). Neither justifies using a branch-point covariance to determine the step 160-200 window unless Cov_x(f_k, Δr) remains proportional to the actual training incentive throughout the run. The paper's own data show the analogous quantity for rent is not stable: the ranking of factorial arms by rent reverses under both RLOO and GRPO (Section 6), and Appendix M reports that the covariance of formatting with the reward difference at step 400 changes sign after a price cut at β=0.005 and doubles at β=0.02. Table 12 shows the two price-cut arms overshooting the covariance forecast in headers by factors of about 2.7 and 2.1, which is the direction an incentive drift would produce. The 20/20 sign agreement does not settle this, because early movement can determine the endpoint supply even if the branch-point covariance no longer describes the incentive at steps 160-200. The paper states that it can measure this change at a checkpoint but cannot yet forecast it before RL ([b211], [b212]); that is precisely the unverified premise behind the 200-step supply claim.

I read the central claim as: the supply response to an RM edit is forecastable before training from κ E_x[Cov_x(f_k, Δr)] evaluated at the branch point. The pre-specified factorial protocol, the self-distilled placebo, the fixed-before-runs analysis plan, the parameter-free sign result, and the frozen-κ transfer to real upgrades are genuine independent support, and I do not see an internal inconsistency in the derivation of Eq. 3. The single most load-bearing step is temporal: Eq. 3 is derived as an instantaneous first-order relation, but the headline validation is at steps 160-200, so the branch-point covariance must remain proportional to the actual incentive over the whole window. The reader's weakest assumption identifies exactly this premise. My concern is a sharper version: the paper never measures the trajectory of the supply covariances in the factorial arms, and the one place where it does recompute a covariance along the path (Appendix M, formatting after symmetric shocks) shows sign change or growth. The 20/20 direction result is consistent with early movement dominating the endpoint and does not establish that the branch-point covariance is the operative incentive at the end window. A checkpoint covariance measurement at steps 50, 100, and 200 for the factorial arms would settle whether the stability premise holds. Because the paper already carries a CONDITIONAL verdict due to this and related horizon issues, my read does not move the verdict; it specifies the test that would confirm or dissolve the condition.

The referee's report

The practical payoff, if the results hold, is large: one scoring pass over roughly 16 responses per prompt would let practitioners screen RM edits before spending GPU-hours on RL. The methodology is a real strength: the analysis plan was frozen before the runs with a decision log; the self-distilled placebo isolates the planted price change from retraining noise; the forecasts are compared with a classical alternative; the upgrade test with frozen κ (66%, 7/7 directions) is genuinely out of sample; and the GRPO and replication batches test robustness. The paper also reports its failures candidly: pre-specified hypotheses S1-S6 are only partly supported, the Llama upgrade exceeds the header forecast by a factor of 3.49, and the formatting covariance at step 400 reverses after a price cut at β=0.005. This honesty, together with falsifiable pre-registered predictions, makes the empirical core unusually trustworthy. The main gap is that the headline 200-step supply claim is not backed by a direct stability check of the covariance over the window, and the abstract's 82% is an in-sample calibration share.

Pith's reading of the paper (a machine's summary, not the authors')

This paper claims that the effect of editing a reward model on an RLHF policy can be computed before any RL training: sample the current policy, score the responses with the old and new reward models, and take the within-prompt covariance between each behavior and the score change. The first-order theory of KL-regularized RL says each behavior's supply moves as $\Delta s_k \approx \kappa\,\mathbb{E}_x[\mathrm{Cov}_x(f_k,\Delta r)]$, and the paper verifies this on 54 pre-registered RL runs with price changes deliberately implanted into retrained reward models under a fractional factorial design with a self-distilled placebo. The covariance predictor gets all 20 stable directions right and explains 82% of the standardized supply change at step 200, against 50% for the classical counterfactual-price predictor, transfers to real reward-model upgrades (66% out of sample), and holds under GRPO. The same covariance also ranks edits by their effect on rent for the first 30 steps (Spearman 0.94), which is enough to screen edits, but the ranking reverses by step 200, so the paper measures the forecast's horizon rather than claiming it lasts. If correct, the result lets practitioners evaluate a reward-model edit at the cost of one scoring pass instead of hundreds of GPU-hours.

What carries the argument. The covariance predictor, Eq. (3): $\Delta s_k \approx \kappa\,\mathbb{E}_x[\mathrm{Cov}_x(f_k,\Delta r)]$, the within-prompt covariance between a behavior and the change in reward, averaged over prompts and scaled by one response coefficient $\kappa$. It is the supply-side fluctuation–response relation: in the KL-regularized optimum the supply matrix is the within-prompt covariance matrix divided by $\beta$, and in early training the Price selection term gives the same form with $\kappa$ acting as an effective learning rate. The paper contrasts it with the price predictor $\kappa\Sigma\,\Delta p$, which multiplies counterfactual feature-price changes by the covariance of the features; the two agree only for additive reward models, and the measured gap between them is concentrated in length. The experimental instrument is a $2^{4-1}$ fractional factorial design with a self-distilled placebo: reward models are retrained on the same pairs with soft labels shifted by known doses, so each treated model differs from the placebo only in the implanted prices of chosen features.

Core claim. Reading a reward model as a set of prices, the paper derives and tests a single forecasting identity: after an edit $\Delta r$, the change in the supply of any behavior $f_k$ is $\Delta s_k \approx \kappa\,\mathbb{E}_x[\mathrm{Cov}_x(f_k,\Delta r)]$, where the covariance is taken within prompt across responses of the current policy and averaged over prompts, and $\kappa$ is a response coefficient fitted once per feature. The identity follows two ways — from the exponential-tilt optimum of KL-regularized RL and from the one-step selection term of policy updates — and it needs no list of priced features, which is why it captures effects, such as length, that the classical counterfactual-price predictor $\kappa\Sigma\,\Delta p$ misses when the reward model values features non-additively. The paper's controlled experiment implants price changes into retrained reward models through soft labels, with a self-distilled placebo, across six parent policies, and fixes every forecast before the runs; with this design it predicts the direction of all 20 stable supply changes and explains 82% of their standardized size, versus 50% for the price predictor. With rent in place of a feature, the same covariance ranks edits correctly over the first 30 steps (Spearman 0.94), but that ranking reverses by step 200, so the paper states the forecast's horizon explicitly: it holds while the policy stays near the branch point, and how long that is depends on how far the policy travels. The paper also reports that the initial response to a price change is independent of the KL coefficient $\beta$ and saturates below the equilibrium level $1/\beta$, consistent with an implicit regularizer $\beta_0 \approx 0.003$–$0.007$ added to the nominal KL penalty.

The premise it rests on. The load-bearing premise is that a behavior's covariance with the reward change, measured once at the start of the run on about 16 responses per prompt, keeps its sign and relative size for the whole 200-step window; the paper's own experiments show that for rent this same covariance can grow or reverse as the policy moves away from the branch point.

If the paper is right (Pith's inferences, not the authors')

One scoring pass at a branch point — about 16 responses per prompt, scored by both reward models — predicts the direction of every stable supply change in the factorial experiment (20 of 20) and 82% of its standardized size at step 200, which is what a team needs to reject a bad edit before spending GPU time.

The classical alternative, counterfactual feature prices, explains only 50% of the same responses, so price-based analysis measures what a reward model values rather than the incentive a policy faces.

With rent in place of a feature, the same covariance ranks edits correctly over the first 30 steps (Spearman 0.94 for rent, 0.89 for gold reward), but the ranking reverses by step 200 under both RLOO and GRPO; real upgrades keep their order, and the paper's recipe tracks KL distance and recomputes the covariance at checkpoints.

The forecast transfers out of the box: on three real reward-model upgrades it explains 66% of the standardized response with coefficients frozen from the factorial runs (7 of 7 stable directions), and under GRPO it explains 75% at step 200.

RL stops short of the KL equilibrium: the initial response does not depend on the KL coefficient $\beta$, and supply saturates below the $1/\beta$ level, consistent with an implicit regularizer $\beta_0 \approx 0.003$–$0.007$ in addition to the nominal KL penalty.

My extension: because the recipe needs no feature list, it should transfer to behaviors the paper never priced — refusals, sycophancy, tone, code comments — through the same two-model scoring pass; a cheap test is to run the recipe on such behaviors in the same testbed.

My extension: the measured horizon suggests an iterated-forecast protocol that the paper only sketches — re-estimate the covariance at checkpoints and treat the forecast as valid while its sign persists — which would turn a one-shot prediction into a schedule of expiry dates for each edit.

My extension: if the implicit regularizer $\beta_0$ is real, the effective KL strength in deployed RLHF is larger than the nominal $\beta$, which matters for anyone tuning $\beta$ or comparing KL budgets across runs; the saturation curve itself provides a per-pipeline estimate of $\beta_0$.

My extension: the length gap between the two predictors implies that efforts to repair reward hacking by decomposing rewards into priced features will keep missing the largest channel, because any edit that changes style prices carries a length component that is invisible to price-based analysis but visible to the covariance.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Responses with headers, lists and greater length receive high scores from reward models (RMs), and RLHF policies learn to produce these features even when true quality does not improve. Practitioners therefore edit their RMs, but they learn whether an edit works only after an RL run. We show that the response of a policy to an RM edit can be forecast before training, from samples of the current policy and the scores of the old and new RM. If we read the RM as a set of prices, the first-order theory of KL-regularized RL implies that each feature moves in proportion to its within-prompt covariance with the change in reward. We implant controlled price changes into retrained RMs under a 2^(4-1) fractional factorial design with a self-distilled placebo, run 54 RL runs from six parent policies, and fix every forecast before the runs. With the covariance predictor, we predict the direction of all 20 changes in supply whose forecast has the same sign at every parent and explain 82% of the standardized change. With counterfactual prices, the classical alternative, we explain 50%. The forecast also holds under GRPO and for real RM upgrades, where we explain 66% out of sample with a coefficient frozen from the factorial runs. For rent, the gap between the proxy reward and a gold reward from stronger judges, we rank RM edits correctly with the same covariance over the first 30 steps (Spearman 0.94). This measures the horizon of the forecast: by step 200, the ranking of the factorial edits reverses under both RLOO and GRPO, while the real upgrades keep their order. When we vary the KL coefficient beta, the initial response does not depend on beta, and the response saturates below the KL equilibrium. In one experiment, the long-run response to a price rise is consistent with an additional implicit regularizer beta_0 of about 0.003 to 0.007.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-04

The authors' main claim, as the classifier picked it out 1.00

“We show that the response of a policy to an RM edit can be forecast before training, from samples of the current policy and the scores of the old and new RM.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.89 · human participants 0.05 · controlled comparison 0.82 · randomized assignment 0.08 · states quantitative result 0.99 · compares with prior results 0.94 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Responses with headers, lists and greater length receive high scores from reward models (RMs), and RLHF policies learn to produce these features even when true quality does not improve. background 1.00 0.17 0.04 asserted 1.00
Practitioners therefore edit their RMs, but they learn whether an edit works only after an RL run. gap 0.68 0.09 0.02 asserted 1.00
We show that the response of a policy to an RM edit can be forecast before training, from samples of the current policy and the scores of the old and new RM. contribution 0.97 0.94 0.87 asserted 1.00
If we read the RM as a set of prices, the first-order theory of KL-regularized RL implies that each feature moves in proportion to its within-prompt covariance with the change in reward. interpretation 0.60 0.31 0.22 asserted 0.99
We implant controlled price changes into retrained RMs under a 2^(4-1) fractional factorial design with a self-distilled placebo, run 54 RL runs from six parent policies, and fix every forecast before the runs. method 0.98 0.97 0.06 asserted 0.95
With the covariance predictor, we predict the direction of all 20 changes in supply whose forecast has the same sign at every parent and explain 82% of the standardized change. result 1.00 0.96 0.27 asserted 1.00
With counterfactual prices, the classical alternative, we explain 50%. result 1.00 0.88 0.05 asserted 1.00
The forecast also holds under GRPO and for real RM upgrades, where we explain 66% out of sample with a coefficient frozen from the factorial runs. result 1.00 0.96 0.17 asserted 1.00
For rent, the gap between the proxy reward and a gold reward from stronger judges, we rank RM edits correctly with the same covariance over the first 30 steps (Spearman 0.94). result 1.00 0.94 0.07 asserted 1.00
This measures the horizon of the forecast: by step 200, the ranking of the factorial edits reverses under both RLOO and GRPO, while the real upgrades keep their order. result 0.94 0.95 0.07 asserted 1.00
When we vary the KL coefficient beta, the initial response does not depend on beta, and the response saturates below the KL equilibrium. result 1.00 0.92 0.05 asserted 1.00
In one experiment, the long-run response to a price rise is consistent with an additional implicit regularizer beta_0 of about 0.003 to 0.007. result 0.84 0.93 0.04 hedged possibility 0.98

What kind of paper the classifier reports this is: empirical study 0.46 · evidence from observation or experiment 0.94

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-04: 1 request, 9,087 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (7)
Figure 1 of the paper
Figure 1. Predicted against observed change in sup
Figure 2 of the paper
Figure 2. Left: frozen forecast of the change in rent
Figure 3 of the paper
Figure 3. Prospective test with symmetric shocks: change in formatting after a price rise and a price cut of equal size, divided by the forecast C/(β + β0), mean over three parents. At β = 0.00125, the policy under the cut reaches the length limit. whereas the incentive after the rise has fallen to 0.68 of its value, as on the placebo policy. The forecast ranks earlier patches. Wang et al. (2026) trained the same policy family from SFT un￾der 13 distinct patches. For each patch, we compute ∆r from the reward rule of the patch on 256 SFT samples per prompt and evaluate Ex[Covx(ρ, ∆r)]. We kne
Figure 4 of the paper
Figure 4. Change in formatting supply relative to the
Figure 5 of the paper
Figure 5. Change in supply relative to the placebo of the same parent, mean over six parents, first batch. Solid lines:
Figure 6 of the paper
Figure 6. Change in rent relative to the placebo of the same parent, in the first batch (six parents), the replication
Figure 7 of the paper
Figure 7. KL divergence accumulated beyond the placebo in steps 160–200 against the change in rent at step 200,

Figures are the authors', reproduced from the paper.

References (53)
  1. [1] Arash Ahmadian, Chris Cremer, Matthias Gall \'e , Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet \"U st \"u n, and Sara Hooker. 2024. https://doi.org/10.18653/v1/2024.acl-long.662 Back to basics: Revisiting REINFORCE -style optimization for learning from human feedbac... on Pithdoi:10.18653/v1/2024.acl-long.662
  2. [2] Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. 2025. https://arxiv.org/abs/2503.11926 Monitoring reasoning models for misbehavior and the risks of promoting obfuscation . Preprint, arXiv:2503.11926 on PithPith's reviewarXiv
  3. [3] Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D'Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. 2025. https://arxiv.org/abs/2401.01879 Theoretical guarantees on the best-of-n alignment policy . In Proceedings of the 42nd International Conference ... on PithPith's reviewarXiv
  4. [4] George E. P. Box and J. Stuart Hunter. 1961. The 2^ k-p fractional factorial designs, Part I . Technometrics, 3(3):311--351 on Pith
  5. [5] Yuyan Bu, Liangyu Huo, Yi Jing, and Qing Yang. 2025. https://doi.org/10.18653/v1/2025.findings-naacl.169 Beyond excess and deficiency: Adaptive length bias mitigation in reward models for RLHF . In Findings of the Association for Computational Linguistics: NAACL 2025, pages 30... on Pithdoi:10.18653/v1/2025.findings-naacl.169
  6. [6] Stephen Casper, Xander Davies, et al. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research on Pith
  7. [7] Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. https://proceedings.mlr.press/v235/chen24bn.html ODIN : Disentangled reward mitigates hacking in RLHF . In Proceedings of the 41st Internati... on Pith
  8. [8] Brian Christian, Jessica A. F. Thompson, Elle Michelle Yang, Vincent Adam, Hannah Rose Kirk, Christopher Summerfield, and Tsvetomira Dumbalska. 2026. https://arxiv.org/abs/2601.20838 Reward models inherit value biases from pretraining . In The Fourteenth International Conferen... on PitharXiv
  9. [9] Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. https://arxiv.org/abs/1706.03741 Deep reinforcement learning from human preferences . In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc on PithPith's reviewarXiv
  10. [10] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. 2024. https://arxiv.org/abs/2310.02743 Reward model ensembles help mitigate overoptimization . In The Twelfth International Conference on Learning Representations (ICLR) on PithPith's reviewarXiv
  11. [11] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. https://arxiv.org/abs/2310.01377 UltraFeedback : Boosting language models with scaled AI feedback . In Proceedings of the 4... on PithPith's reviewarXiv
  12. [12] Yann Dubois, Bal \'a zs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. https://arxiv.org/abs/2404.04475 Length-controlled AlpacaEval : A simple way to debias automatic evaluators . Preprint, arXiv:2404.04475 on PithPith's reviewarXiv
  13. [13] Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D'Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. 2024. https://arxiv.org/abs/2312.09244 Helping or herding? Reward model ensembles mitig... on PithPith's reviewarXiv
  14. [14] Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N. Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2025. How to evaluate reward models for RLHF . In The Thirteenth International Conference on Learning Representations on Pith
  15. [15] Leo Gao, John Schulman, and Jacob Hilton. 2023. https://arxiv.org/abs/2210.10760 Scaling laws for reward model overoptimization . In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10835--10866. PMLR on PithPith's reviewarXiv
  16. [16] Dylan Hadfield-Menell and Gillian K. Hadfield. 2019. https://doi.org/10.1145/3306618.3314250 Incomplete contracting and AI alignment . In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (AIES), pages 417--422. ACM on Pithdoi:10.1145/3306618.3314250
  17. [17] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. https://arxiv.org/abs/1503.02531 Distilling the knowledge in a neural network . Preprint, arXiv:1503.02531 on PithPith's reviewarXiv
  18. [18] Harold Hotelling. 1932. Edgeworth's taxation paradox and the nature of demand and supply functions. Journal of Political Economy, 40(5):577--616 on Pith
  19. [19] Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. 2022. Datamodels: Understanding predictions with data and data with predictions. In Proceedings of the 39th International Conference on Machine Learning on Pith
  20. [20] Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1885--1894 on Pith
show 33 more
  1. [21] Tomasz Korbak, Ethan Perez, and Christopher L. Buckley. 2022. RL with KL penalties is better viewed as B ayesian inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1083--1091 on Pith
  2. [22] Anne O. Krueger. 1974. The political economy of the rent-seeking society. The American Economic Review, 64(3):291--303 on Pith
  3. [23] Ryogo Kubo. 1966. The fluctuation-dissipation theorem. Reports on Progress in Physics, 29(1):255--284 on Pith
  4. [24] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luc... on PithPith's reviewarXiv
  5. [25] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2025 b . RewardBench : Evaluating reward models for language modeling. In Findings of the As... on Pith
  6. [26] Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu, and Yahui Zhou. 2026. https://arxiv.org/abs/2507.01352 Skywork-Reward-V2 : Scaling preference data curation via human- AI synergy . In The Fou... on PithPith's reviewarXiv
  7. [27] Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, et al. 2025 a . RRM : Robust reward model training mitigates reward hacking. In International Conference on Learning Representations on Pith
  8. [28] Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. 2025 b . RM-Bench : Benchmarking reward models of language models with subtlety and style. In The Thirteenth International Conference on Learning Representations on Pith
  9. [29] David Manheim and Scott Garrabrant. 2018. https://arxiv.org/abs/1803.04585 Categorizing variants of Goodhart's law . Preprint, arXiv:1803.04585 on PithPith's reviewarXiv
  10. [30] Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. 2018. The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11):2600--2606 on Pith
  11. [31] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, ... on Pithdoi:10.52202/068431-2011
  12. [32] Alexander Pan, Kush Bhatia, and Jacob Steinhardt. 2022. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations on Pith
  13. [33] George R. Price. 1970. Selection and covariance. Nature, 227:520--521 on Pith
  14. [34] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems on Pith
  15. [35] Alexandre Ram \'e , Nino Vieillard, L \'e onard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. 2024. WARM : On the benefits of weight averaged reward models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of P... on Pith
  16. [36] Javier Rando and Florian Tram \`e r. 2024. Universal jailbreak backdoors from poisoned human feedback. In International Conference on Learning Representations on Pith
  17. [37] Yi Ren and Danica J. Sutherland. 2025. Learning dynamics of LLM finetuning. In International Conference on Learning Representations on Pith
  18. [38] Sherwin Rosen. 1974. Hedonic prices and implicit markets: Product differentiation in pure competition. Journal of Political Economy, 82(1):34--55 on Pith
  19. [39] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. https://arxiv.org/abs/2402.03300 DeepSeekMath : Pushing the limits of mathematical reasoning in open language models . Preprint, arXiv:2402.03300 on PithPith's reviewarXiv
  20. [40] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. https://doi.org/10.1145/3689031.3696075 HybridFlow : A flexible and efficient RLHF framework . In Proceedings of the Twentieth European Conference on Comput... on Pithdoi:10.1145/3689031.3696075
  21. [41] Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2024. https://arxiv.org/abs/2310.03716 A long way to go: Investigating length correlations in RLHF . In First Conference on Language Modeling (COLM) on PithPith's reviewarXiv
  22. [42] Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. https://doi.org/10.52202/068431-0687 Defining and characterizing reward hacking . In Advances in Neural Information Processing Systems, volume 35, pages 9460--9471. Curran Associates, Inc on Pithdoi:10.52202/068431-0687
  23. [43] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. https://arxiv.org/abs/2009.01325 Learning to summarize with human feedback . In Advances in Neural Information Processing Systems, vol... on PithPith's reviewarXiv
  24. [44] Gordon Tullock. 1967. https://doi.org/10.1111/j.1465-7295.1967.tb01923.x The welfare costs of tariffs, monopolies, and theft . Western Economic Journal, 5(3):224--232 on Pithdoi:10.1111/j.1465-7295.1967.tb01923.x
  25. [45] Chaoqi Wang, Zhuokai Zhao, Yibo Jiang, Zhaorun Chen, Chen Zhu, Yuxin Chen, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Hao Ma, and Sinong Wang. 2025. https://arxiv.org/abs/2501.09620 Beyond reward hacking: Causal rewards for large language model alignment . Preprint, arXiv:2501.09620 on PithPith's reviewarXiv
  26. [46] Jiacheng Wang and Jinbin Huang. 2026. https://arxiv.org/abs/2603.28063 Reward hacking as equilibrium under finite evaluation . Preprint, arXiv:2603.28063 on PitharXiv
  27. [47] Jiongxiao Wang, Junlin Wu, Muhao Chen, Yevgeniy Vorobeychik, and Chaowei Xiao. 2024. RLHFPoison : Reward poisoning attack for reinforcement learning with human feedback in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin... on Pith
  28. [48] Zixiao Wang, Xingbo Du, Duzhen Zhang, and Le Song. 2026. https://arxiv.org/abs/2610.19832 Rent-seeking language models: Reward models set the prices, frequent audits curb the rent . Preprint, arXiv:2610.19832 on PithPith's reviewarXiv
  29. [49] Lorenz Wolf, Robert Kirk, and Mirco Musolesi. 2025. https://arxiv.org/abs/2505.18126 Reward model overoptimisation in iterated RLHF . Preprint, arXiv:2505.18126 on PitharXiv
  30. [50] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, et al. 2025. https://arxiv.org/abs/2505.09388 Qwen3 technical report . Preprint, arXiv:2505.09388 on PithPith's reviewarXiv
  31. [51] Xiaoying Zhang, Jean-Fran c ois Ton, Wei Shen, Hongning Wang, and Yang Liu. 2024. https://doi.org/10.52202/079017-2596 Mitigating reward overoptimization via lightweight uncertainty estimation . In Advances in Neural Information Processing Systems, volume 37, pages 81717--8174... on Pithdoi:10.52202/079017-2596
  32. [52] Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, and Tong Zhang. 2025. https://doi.org/10.18653/v1/2025.acl-long.1308 From lists to emojis: How format bias affects model alignment . In Proceedings of the 63rd Annual Meeting of the Association for Computationa... on Pithdoi:10.18653/v1/2025.acl-long.1308
  33. [53] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. https://arxiv.org/abs/1909.08593 Fine-tuning language models from human preferences . Preprint, arXiv:1909.08593 on PithPith's reviewarXiv
Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment