REVIEW 3 major objections 6 minor 17 references
The Strong, Weak and Benign Goodhart's law. An independence-free and paradigm-agnostic formalisation
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Proxy-goal dependence, not tail weight alone, settles which Goodhart regime applies.
desk verdict Useful core theorem on heavy-tailed goals without independence, but the abstract's light-tailed claim is contradicted by the paper's own Gaussian formulas. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the conditional expectation $\mathbb{E}[G\mid M>m]$ together with the conditional correlation $\mathrm{Corr}(G,M\mid M>m)$, evaluated as $m$ approaches the upper support of $M$. Optimisation is modelled mechanism-agnostically as conditioning on the extreme tail event $M>m$. For the heavy-tailed-goal theorem, the proof machinery is a lemma showing that for a two-sided light-tailed variable $\xi$, the conditional tail $\mathbb{P}(\xi>t\mid G)$ decays almost surely faster than any $e^{-ct}$; this lets the proof cut the integral defining $\mathbb{E}[G\mid M>m]$ and show that the slow tail of $G$ dominates. In the Gaussian case, exact Gaussian-tail expansions and conditional moment formulas give explicit asymptotics for the conditional mean, variance, covariance, and hence correlation. In the exponential-goal case, an integration-by-parts expansion for integrals of the form $P(g)\exp(-g((m-g)/\eta)^{b-1})$ yields the leading terms.
What would settle it
Simulate a heavy-tailed $G$ (for instance Pareto with tail index $1/\gamma$) and a two-sided light-tailed $\xi$ with an adversarially chosen dependence (for example $\xi$ almost surely negative when $G$ is large), set $M=G+\xi$, and compute $\mathbb{E}[G\mid M>m]$ for increasing $m$; Theorem 4.1 predicts this conditional mean is at least $m+o(m)$ for every coupling, so a bounded or decreasing value at large $m$ would refute it. For the Gaussian claim, fit a bivariate normal with $\mathrm{Var}(G)>\mathrm{Var}(\xi)$ and check whether $\log\mathrm{Corr}(G,M\mid M>m)$ versus $\log m$ has slope $-1$; any different slope would refute the claimed benign decorrelation.
Extended reading notes
Core claim
The central claim is a tail-thickness trichotomy that survives dependence, plus two dependent examples showing that coupling controls the outcome. Formally, Theorem 4.1 says that whenever the goal's survival function is regularly varying of index $-1/\gamma$ and the discrepancy is exponentially light on both sides, conditioning on $M>m$ forces $\mathbb{E}[G\mid M>m]$ to grow at least as $m+o(m)$, regardless of how $G$ and $\xi$ are coupled. Theorem 4.2 and Lemma 4.1 show that for Gaussian $(G,\xi)$ with $\mathrm{Var}(G)>\mathrm{Var}(\xi)$, $\mathbb{E}[G\mid M>m]\sim \frac{a+c}{a+b+2c}m$ while $\mathrm{Corr}(G,M\mid M>m)\sim \frac{(a+c)\sqrt{a+b+2c}}{m\sqrt{ab-c^2}}\to 0$: the benign case where the proxy becomes uninformative before the goal stops improving. Lemma 4.3 shows that when $G\sim\mathrm{Exp}(1)$ and $\xi\mid G$ has conditional density $G\exp(-G((x/\eta)^{b-1}-1))x^{b-2}(b-1)/\eta^{b-1}$ for $x>\eta$, the same conditioning sends $\mathbb{E}[G\mid M>m]$ to zero at rate $m^{1-b}$, with larger $b$ (lighter discrepancy tail) meaning faster collapse. Together these results motivate the paper's formal four-way classification: no, benign, weak, and strong Goodhart.
Load-bearing premise
The paper treats optimising a proxy as conditioning on the proxy exceeding a very high threshold $m$; if real optimisation procedures do not drive the proxy into its extreme upper tail this way, the predicted Goodhart regimes need not transfer to practice.
Editorial extensions
If this is right
- If the goal is heavy-tailed with regularly varying survival and the discrepancy is light-tailed on both sides, no dependence structure can create weak or strong Goodhart: the expected goal must grow at least linearly with the optimisation threshold.
- In the Gaussian light-tailed regime with $\mathrm{Var}(G)>\mathrm{Var}(\xi)$, a monitor that tracks only the proxy-goal correlation will see it decay toward zero while the goal continues to grow, so correlation collapse is not by itself evidence of goal failure.
- In the exponential-goal/heavy-tailed-discrepancy regime, a coupling that associates large discrepancies with small goal values turns an independent-case weak Goodhart situation into strong Goodhart, and a lighter discrepancy tail makes the expected goal fall to zero faster.
- The paper's formal classification (no, benign, weak, strong) gives empirical auditors a concrete checklist: estimate the tail classes of goal and discrepancy, estimate their copula, and read off the predicted regime from the asymptotics.
Reading between the lines
- In deployed systems where the discrepancy between a learned reward and the true reward is conditionally heavier when the true reward is small, independence-based Goodhart analyses may be systematically optimistic; measuring the conditional law of $\xi\mid G$, not just its marginal tail, would indicate whether collapse is faster than the independent-case formulas predict.
- The benign Gaussian case carries a practical warning: using the correlation between proxy and goal as an early-stopping or 'proxy still valid' signal can advise stopping just as the goal is still improving, since the correlation decays like $1/m$.
- A natural next test is to take real proxy-goal pairs (for example a learned reward model and a gold-standard reward), estimate the tail index of $G$ and the conditional tail of $\xi\mid G$, and compare the measured $\mathbb{E}[G\mid M>m]$ along an actual optimisation trajectory to the three asymptotic regimes.
- The dependence-free lower bound suggests a design heuristic the paper does not itself propose: making the discrepancy two-sided light-tailed, or ensuring the goal's tail dominates the discrepancy's, would rule out the worst Goodhart outcomes no matter how the two are coupled.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper formalises Goodhart's law in the paradigm-agnostic framework G = M + ξ, where optimisation of the proxy metric is modelled as conditioning on M > m with m → S(M). It defines four outcomes (no, benign, weak, strong Goodhart) and studies how the dependence between the goal G and the discrepancy ξ changes the outcome. The main results are: (i) Theorem 4.1, asserting that when G is heavy-tailed and ξ is two-sided light-tailed, E[G|M>m] ≥ m + o(m) regardless of dependence; (ii) a Gaussian analysis (Lemma 4.1, Theorem 4.2) giving the conditional expectation and correlation asymptotics and identifying a 'benign Goodhart' regime under Var(G) > Var(ξ); and (iii) an exponential-goal / conditionally heavy-tailed discrepancy example (Lemmas 4.2 and 4.3) producing strong Goodhart with rate η^{b-1}/m^{b-1}. The appendix contains detailed computations, several of which are verified with Sympy.
Significance. If the claims are appropriately qualified, the paper makes a useful contribution: it extends the independence-based analysis of El-Mhamdi and Hoang to dependent couplings, provides explicit and checkable Gaussian formulas, and exhibits a concrete heavy-tailed coupling that turns weak into strong Goodhart. The machine-checked symbolic computations are a genuine strength, as is the effort to state tail conditions explicitly. However, the headline claim that dependence does not change the nature of Goodhart's law in the light-tailed case is not supported by the paper's own Gaussian formulas, and one of the heavy-tailed lemmas is stated outside its domain of validity. With corrections and a more accurate abstract, the core material can be made publishable.
major comments (3)
- [Abstract and §4.3 (Lemma 4.1)] The abstract claims that in the light-tailed-goal/light-tailed-discrepancy case 'dependence does not change the nature of Goodhart's effect.' This is contradicted by Lemma 4.1, which gives E[G|M>m] ∼ (a+c)/(a+b+2c) m. For admissible parameters with Var(G)<Var(ξ), for example a=1, b=2, c=−1.4 (satisfying |c|<√(ab)=1.414), the coefficient a+c is negative, so E[G|M>m]→−∞, which is strong Goodhart by Table 1, whereas under independence (c=0) the same a,b give a positive coefficient and E[G|M>m]→+∞, i.e. benign Goodhart. Dependence therefore flips the nature of the effect within the paper's own definitions. Section 5's admission that dependence matters when 'relative tail thickness are not favorable' is inconsistent with the abstract. The abstract and Section 4.1 overview must be revised to state the condition a+c>0 (e.g. Var(G)>Var(ξ)) under which the benign conclusion holds, and Theorem 4.2's 'no matter the correlation' wording should be qualified because decorrelation alone does not imply benign Goodhart.
- [§4.4, Lemma 4.2 and §6.4] Lemma 4.2 states E[ξ|M>m] ∼ m(b−1)/(b−2) under the setup of Section 6.4, where b is only assumed to lie in (1,∞). This asymptotic is valid only for b>2. The proof in Section 6.7 contains the term (b−1)η^{b−1}∫_m^∞ x^{−(b−1)}dx, which diverges for 1<b<2, and Lemma 6.7 itself explicitly restricts to b>2. For 1<b<2 the printed leading coefficient (b−1)/(b−2) is negative and the conditional expectation of ξ is not covered by the stated expansion. The domain restriction must be imposed in Lemma 4.2 and in the Section 4.4 claims that the discrepancy is maximised; as written, the strong-Goodhart example is stated beyond its range of validity.
- [§4.2 / Appendix 6.8 (Theorem 4.1)] The proof of Theorem 4.1 is not complete as written. In bounding I1 and E1, the text invokes Slutsky's theorem to pass from e^{cm}P(ξ>m|G)→0 almost surely to e^{cm}E[|G|1{G<0}P(ξ>m|G)]→0, but the random variable |G| is unbounded and, for heavy-tailed G with γ≥1, may have infinite mean; no uniform integrability or dominating function is supplied, and convergence in probability does not imply convergence of the expectation. The earlier bound P(ξ>m−g)≤P(ξ>m) used for negative g is similarly insufficient when E|G| is infinite. The theorem may be true, but the stated lower bound E[G|M>m]≥m+o(m) is not established for the full class of heavy-tailed G admitted by the statement, and the theorem should either add conditions ensuring the conditional expectation is finite or provide a rigorous truncation argument.
minor comments (6)
- [§6.6] The heading 'Proof of lemma 3.3' should read 'Proof of Lemma 4.3'; the lemma is numbered 4.3 in the main text.
- [§6.7] In the proof of Lemma 4.2, the displayed asymptotic for αE[ξ|G+ξ>m] omits the leading term (b−1)η^{b−1}/((b−2)m^{b−2}); as printed, multiplying by Δ does not produce the stated equivalent and the proof appears internally inconsistent.
- [§6.5, Lemma 6.15] The error term o(1/m^{1−4b}) in Lemma 6.15 should presumably be o(1/m^{4b−1}); the current exponent changes sign for non-integer b and is inconsistent with the expansion it follows.
- [§4.3, Table 3] Table 3 labels the effect of c>0 on E[G|M>m] as negative, but under the stated condition Var(G)>Var(ξ) the conditional expectation still diverges to +∞; the sign refers to the coefficient in the asymptotic equivalent, not to the direction of divergence, and the table should say so explicitly.
- [§4.1] The overview of the exponential case writes the conditional density as exp(G((x/η)^{b−1}−1)x^{b−2}; the typesetting loses the minus sign in the exponential and the intended exponent, making the density ambiguous compared with the correct definition in Section 6.4.
- [§3] The identification of 'optimising the proxy' with conditioning on M>m for m→S(M) is a strong mechanism-agnostic modelling choice that is not empirically justified; the paper should state more prominently that its results describe the tail of M under this conditioning model and may not transfer to specific optimisation algorithms.
Circularity Check
No significant circularity; the derivations are explicit from stated assumptions, though the abstract overgeneralizes the light-tailed claim (a correctness issue, not circularity).
full rationale
All three central results are proven from the stated assumptions rather than imported. Theorem 4.1 is established by bounding E[G 1{M>m}] from below with the Fréchet lower copula bound and controlling the light-tailed discrepancy through Lemma 6.16, whose Borel-Cantelli proof is given in the appendix; the conclusion E[G|M>m] ≥ m+o(m) follows from these bounds, not from the definition of heavy/light tails. In the Gaussian case, Lemmas 4.1 and 6.5 and Theorem 4.2 are direct tail expansions of Gaussian integrals (Appendix 6.2-6.3) and do not assume any Goodhart outcome. The exponential-goal/heavy-tailed-discrepancy example is an explicit constructed conditional density; Lemmas 4.2 and 4.3 follow from the integration-by-parts Lemma 6.8, and no parameter is fitted to data and then renamed a prediction. The only self-citation, [EH24], supplies the conditioning-on-M>m modelling convention and baseline vocabulary (weak/strong Goodhart), but none of the new theorems is replaced by a citation: the paper derives the corollaries it attributes to [EH24] through its own computation. The paper does contain a serious internal-consistency/correctness flaw that is not a circularity: the abstract's claim that in the light-tailed goal/light-tailed discrepancy case 'dependence does not change the nature of Goodhart's effect' is contradicted by its own Lemma 4.1, since for (G,ξ)~N(0,Σ) with a=1,b=2,c=-1.4 the coefficient (a+c)/(a+b+2c) is negative, giving E[G|M>m]->-infinity (strong Goodhart), while c=0 gives a positive coefficient (benign). That is an overgeneralization of the stated Var(G)>Var(ξ) condition, but it does not make the derivation circular. The M>m conditioning is a modelling assumption inherited from prior work, not an input that is also the output, so no step reduces by construction.
Assumptions & free parameters
free parameters (2)
- b (tail shape of discrepancy ξ)
- η (scale of discrepancy ξ)
assumptions (6)
- domain assumption Optimisation is modelled by conditioning on M > m with m → ∞.
- domain assumption The proxy metric decomposes additively as M = G + ξ.
- domain assumption In Theorem 4.1, G is regularly varying with index 1/γ and ξ is left- and right-light-tailed.
- ad hoc to paper The conditional law of ξ given G in Section 4.4 is the specific Weibull-like density.
- domain assumption In the Gaussian case, (G, ξ) is jointly Gaussian with Var(G) > Var(ξ).
- ad hoc to paper The asymptotic expansions for E[ξ|M>m] in Lemma 4.2 require b > 2.
Cite this review
Pith. "Pith review of The Strong, Weak and Benign Goodhart's law. An independence-free and paradigm-agnostic formalisation." pith.science (2026). https://pith.science/paper/6SDQDYVJ
@misc{pith2026250523445,
author = {Pith},
title = {Pith review of: The Strong, Weak and Benign Goodhart's law. An independence-free and paradigm-agnostic formalisation},
year = {2026},
howpublished = {\url{https://pith.science/paper/6SDQDYVJ}},
note = {Machine review of arXiv:2505.23445}
}
read the original abstract
Goodhart's law is a famous adage in policy-making that states that ``When a measure becomes a target, it ceases to be a good measure''. As machine learning models and the optimisation capacity to train them grow, growing empirical evidence reinforced the belief in the validity of this law without however being formalised. Recently, a few attempts were made to formalise Goodhart's law, either by categorising variants of it, or by looking at how optimising a proxy metric affects the optimisation of an intended goal. In this work, we alleviate the simplifying independence assumption, made in previous works, and the assumption on the learning paradigm made in most of them, to study the effect of the coupling between the proxy metric and the intended goal on Goodhart's law. Our results show that in the case of light tailed goal and light tailed discrepancy, dependence does not change the nature of Goodhart's effect. However, in the light tailed goal and heavy tailed discrepancy case, we exhibit an example where over-optimisation occurs at a rate inversely proportional to the heavy tailedness of the discrepancy between the goal and the metric. %
Figures
Reference graph
Works this paper leans on
-
[1]
Concrete problems in ai safety
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man \'e . Concrete problems in ai safety. arXiv preprint arXiv:1606.06565 , 2016
arXiv 2016
-
[2]
Cohen, Nikolay Malkin, Matt MacDermott, Damiano Fornasiere, Pietro Greiner, and Younesse Kaddar
Yoshua Bengio, Michael K. Cohen, Nikolay Malkin, Matt MacDermott, Damiano Fornasiere, Pietro Greiner, and Younesse Kaddar. Can a bayesian oracle prevent harm from an agent?, 2024
work page 2024
-
[3]
Faulty reward functions in the wild
Jack Clark and Dario Amodei. Faulty reward functions in the wild. Open AI Blog , 2016
work page 2016
-
[4]
On goodhart's law, with an application to value alignment, 2024
El-Mahdi El-Mhamdi and Lê-Nguyên Hoang. On goodhart's law, with an application to value alignment, 2024
work page 2024
-
[5]
Monetary relationships : a view from threadneedle street
Charles Goodhart. Monetary relationships : a view from threadneedle street. Papers in monetary economics 1975 , 1, 1975
work page 1975
-
[6]
Scaling laws for reward model overoptimization, 2022
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, 2022
2022
-
[7]
Augustin Godinot, Gilles Tredan, Erwan Le Merrer, Camilla Penzo, and Francois Taiani. Under manipulations, are some AI models harder to audit? In 2nd IEEE Conference on Secure and Trustworthy Machine Learning , 2024
work page 2024
- [8]
Show all 17 references
-
[9]
awful idea of accountability
Keith Hoskin. The "awful idea of accountability" : inscribing people into the measurement of objects. 1996
1996
-
[10]
Catastrophic goodhart: regularizing rlhf with kl divergence does not mitigate heavy-tailed reward misspecification
Thomas Kwa, Drake Thomas, and Adri\` a Garriga-Alonso. Catastrophic goodhart: regularizing rlhf with kl divergence does not mitigate heavy-tailed reward misspecification. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in...
2024
-
[11]
Robert E. Lucas. Econometric policy evaluation: A critique. Carnegie-Rochester Conference Series on Public Policy , 1:19--46, January 1976
1976
-
[12]
Building less-flawed metrics: Understanding and creating better measurement and incentive systems
David Manheim. Building less-flawed metrics: Understanding and creating better measurement and incentive systems. Patterns , 4(10):100842, October 2023
2023
-
[13]
Categorizing variants of goodhart's law
David Manheim and Scott Garrabrant. Categorizing variants of goodhart's law. CoRR , abs/1803.04585, 2018
2018 arXiv
-
[14]
Adversarial machine learning: A taxonomy and terminology of attacks and mitigations
Alina Oprea and Apostol Vassilev. Adversarial machine learning: A taxonomy and terminology of attacks and mitigations. Technical report, National Institute of Standards and Technology, 2023
2023
-
[15]
Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. In Proceedings of the 36th International Conference on Neural Information Processing Systems , NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc
2022
-
[16]
‘improving ratings’: audit in the british university system
Marilyn Strathern. ‘improving ratings’: audit in the british university system. European Review , 5(3):305–321, 1997
1997
-
[17]
Consequences of misaligned ai
Simon Zhuang and Dylan Hadfield-Menell. Consequences of misaligned ai. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 15763--15773. Curran Associates, Inc., 2020
2020
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.