REVIEW 3 major objections 3 minor 2 cited by
Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correction
T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Causal Reward Adjustment treats reward hacking in PRM-based reasoning as a confounding problem and corrects it with backdoor adjustment over sparse-autoencoder features, improving math accuracy without retraining the reward model.
desk verdict A plausible new combination of SAE feature recovery and backdoor adjustment for reward hacking, but the abstract alone leaves the causal identification burden unmet. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing identity is the backdoor adjustment formula from causal inference. Let $Z$ be the set of semantic features recovered from the PRM's internal activations by the sparse autoencoder, and let $R$ be the reward assigned to a reasoning path. Instead of using $R$ directly, CRA estimates the adjusted reward $\sum_z \mathbb{E}[R \mid \text{path quality}, z]\,P(z)$, which strips out the spurious association between $Z$ and high scores. The sparse autoencoder is the enabler: it converts high-dimensional, opaque activations into a discrete set of interpretable features that can be conditioned on in the adjustment.
What would settle it
On a math benchmark with known ground-truth labels, take a set of reasoning paths and artificially vary a known spurious feature, such as verbosity or overconfident phrasing, while holding the actual solution fixed. If CRA's adjusted score still ranks the spuriously confident wrong path above the correct one as often as the raw PRM score does, then the backdoor adjustment is not removing the confounding it claims to remove.
Extended reading notes
Core claim
The central claim is that high-scoring but logically incorrect reasoning paths arise because the process reward model's score is confounded by semantic features that correlate with surface characteristics rather than with correctness. CRA recovers these features from the PRM's internal activations using a sparse autoencoder, then applies backdoor adjustment to estimate the reward a path would receive if those confounders were held fixed. Experiments on math solving datasets show that this adjusted reward selects more correct reasoning paths than the raw PRM score, mitigating reward hacking and improving final accuracy. The method works without changing the policy model or retraining the PRM,
Load-bearing premise
The whole method rests on the assumption that the features recovered by the sparse autoencoder are exactly the confounders, and that no important confounder is left out.
Editorial extensions
If this is right
- Raw PRM scores are a biased selection signal because of confounding semantic features; CRA's adjusted score is the better signal for choosing reasoning paths.
- CRA reduces reward hacking and improves final accuracy on math problem-solving datasets.
- The method requires no modification of the policy model and no retraining of the PRM, so it can be layered onto existing systems.
- Sparse autoencoder features learned from PRM activations can stand in for confounders in the backdoor adjustment.
Reading between the lines
- The same logic would apply beyond math: any verifier whose internal activations carry style or surface confounders could be corrected with the same recipe, as long as sparse autoencoders recover the confounders.
- A direct corollary the paper does not draw: the difference between raw and adjusted reward on a given path is a measurement of how 'hacked' that path is, which could be used as a diagnostic or a search heuristic.
- If the feature set is incomplete, the adjusted reward inherits residual confounding; this suggests the method's ceiling on a dataset is set by how faithfully the sparse autoencoder captures the PRM's actual shortcut features.
- A testable extension: use the same adjustment on a reward model trained for a different task, such as code generation or planning, and compare selection accuracy, which would separate the causal claim from the specifics of math datasets.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Causal Reward Adjustment (CRA), a method to mitigate reward hacking in PRM-based external reasoning systems. CRA trains sparse autoencoders on PRM internal activations to recover interpretable features, then uses backdoor adjustment to correct confounding semantic features, thereby estimating the true reward of a reasoning path. The abstract reports experiments on math solving datasets showing that CRA mitigates reward hacking and improves final accuracy without modifying the policy model or retraining the PRM.
Significance. If CRA works as claimed, it introduces a causal perspective on reward hacking and offers a practical, training-free correction that could improve the reliability of process reward models in reasoning tasks. The use of sparse autoencoder features as adjustment variables is an interesting bridge between interpretability and causal inference. However, the abstract alone provides no verifiable evidence: no causal graph, no identification assumptions, no quantitative results, and no validation that the SAE features are valid confounders. The significance therefore remains conditional on the full paper supplying these missing elements.
major comments (3)
- [Abstract] The central claim that CRA 'corrects confounding by using backdoor adjustment' is not verifiable from the abstract. Backdoor adjustment requires a well-defined treatment, outcome, and adjustment set that satisfies the backdoor criterion in an explicit causal graph. The abstract gives none of these: no DAG, no target estimand, no justification that the SAE-recovered features block all backdoor paths and do not open collider paths. If the full paper does not provide this, the 'true reward' is not identified and the method is not causal.
- [Abstract] There is a potential circularity: the SAE is trained on the same PRM activations that are suspected of encoding spurious correlations. If the recovered features merely re-encode the PRM's biases, then adjusting for them may remove legitimate signal or reinforce the reward hacking. The abstract provides no evidence of causal validity of these feature directions, such as causal discovery, intervention tests, or at least a clear argument that the features correspond to semantic confounders rather than arbitrary linear directions.
- [Abstract] The empirical claim ('improves final accuracy') is reported without any numbers, baselines, or error bars. No comparison to existing reward-hacking mitigation methods is given, nor is there any indication of statistical significance. Since the paper's support rests on these experiments, the abstract's bare assertion is insufficient for the reader to judge whether the improvement is real or a feature-selection artifact.
minor comments (3)
- [Abstract] The term 'reward hacking' should be defined explicitly; the abstract implies high-scoring but logically incorrect paths, but a formal definition and relationship to similar concepts in the literature would help.
- [Abstract] The phrase 'without modifying the policy model or retraining PRM' is clear at a high level, but it would help to specify what components are changed (e.g., only a post-hoc reward adjustment layer) and what overhead is introduced.
- [Abstract] Please provide a precise mathematical expression for the backdoor adjustment used, e.g., the corrected reward formula in terms of the SAE features and their conditional distributions.
Circularity Check
No significant circularity: CRA's adjusted reward is validated against external answer accuracy, not derived from it.
full rationale
The paper's central claim is that PRM reward hacking is caused by confounding semantic features and that CRA corrects this via sparse-autoencoder features and backdoor adjustment. The adjusted reward is then evaluated by final problem-solving accuracy on math datasets. This is an external benchmark, not a quantity that is definitionally equal to the SAE features or the raw PRM scores. The main vulnerability—that the SAE features may be incomplete or non-causal—is a validity threat to the causal identification, not a circular reduction: backdoor adjustment is an assumption about the causal graph, and the correctness of that assumption is an empirical identifiability matter, not a logical equivalence between input and output. No equation in the provided text defines the target reward in terms of the fitted features, and no fitted parameter is renamed as a prediction. The abstract alone provides no evidence of self-citation, ansatz smuggling, or renaming of a known result. Ascribing circularity here would require speculative claims about feature selection post hoc, which the hard rules prohibit without exhibitable reduction.
Assumptions & free parameters
free parameters (2)
- SAE dictionary size and sparsity coefficient =
not reported in abstract
- Confounder feature set used in the backdoor adjustment =
not reported in abstract
assumptions (3)
- domain assumption Reward hacking in PRM selection is primarily caused by confounding semantic features in the PRM's internal activations.
- domain assumption The sparse autoencoder features are the actual confounders, and the causal graph satisfies the backdoor criterion (no unmeasured confounding of score and correctness).
- domain assumption Math solving benchmarks used in experiments are representative enough to generalize the claim beyond the evaluated datasets.
Cite this review
Pith. "Pith review of Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correction." pith.science (2026). https://pith.science/paper/V3M4SFXS
@misc{pith2026250804216,
author = {Pith},
title = {Pith review of: Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correction},
year = {2026},
howpublished = {\url{https://pith.science/paper/V3M4SFXS}},
note = {Machine review of arXiv:2508.04216}
}
read the original abstract
External reasoning systems combine language models with process reward models (PRMs) to select high-quality reasoning paths for complex tasks such as mathematical problem solving. However, these systems are prone to reward hacking, where high-scoring but logically incorrect paths are assigned high scores by the PRMs, leading to incorrect answers. From a causal inference perspective, we attribute this phenomenon primarily to the presence of confounding semantic features. To address it, we propose Causal Reward Adjustment (CRA), a method that mitigates reward hacking by estimating the true reward of a reasoning path. CRA trains sparse autoencoders on the PRM's internal activations to recover interpretable features, then corrects confounding by using backdoor adjustment. Experiments on math solving datasets demonstrate that CRA mitigates reward hacking and improves final accuracy, without modifying the policy model or retraining PRM.
Forward citations
Cited by 2 Pith papers
-
Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure
Single-axis reward bias mitigations redirect optimization pressure to correlated proxies, and audit-distribution scoring produces identical observables for successful mitigation, bias substitution, and overcorrection.
-
Factored Causal Representation Learning for Robust Reward Modeling in RLHF
A factored causal representation learning method improves robustness of reward models in RLHF by isolating causal factors from biases like length and sycophancy using adversarial gradient reversal.
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Amodei, D.; Olah, C.; Steinhardt, J.; Christiano, P.; Schulman, J.; and Mané, D. 2016. Concrete Problems in AI Safety. arXiv:1606.06565
arXiv 2016
-
[4]
Besta, M.; Blach, N.; Kubicek, A.; Gerstenberger, R.; Podstawski, M.; Gianinazzi, L.; Gajda, J.; Lehmann, T.; Niewiadomski, H.; and Nyczyk, P. 2024. Graph of Thoughts: Solving Elaborate Problems with Large Language Models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, 17682--17690
work page 2024
-
[5]
E.; Hume, T.; Carter, S.; Henighan, T.; and Olah, C
Bricken, T.; Templeton, A.; Batson, J.; Chen, B.; Jermyn, A.; Conerly, T.; Turner, N.; Anil, C.; Denison, C.; Askell, A.; Lasenby, R.; Wu, Y.; Kravec, S.; Schiefer, N.; Maxwell, T.; Joseph, N.; Hatfield-Dodds, Z.; Tamkin, A.; Nguyen, K.; McLean, B.; Burke, J. E.; Hume, T.; Carter, S.; Henighan, T.; and Olah, C. 2023. Towards Monosemanticity: Decomposing L...
2023
-
[6]
V.; R \'e , C.; and Mirhoseini, A
Brown, B.; Juravsky, J.; Ehrlich, R.; Clark, R.; Le, Q. V.; R \'e , C.; and Mirhoseini, A. 2024. Large Language Monkeys : Scaling Inference Compute with Repeated Sampling . arXiv:2407.21787
arXiv 2024
-
[7]
Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168
arXiv 2021
-
[8]
Cunningham, H.; Ewart, A.; Riggs, L.; Huben, R.; and Sharkey, L. 2023. Sparse Autoencoders Find Highly Interpretable Features in Language Models. arXiv:2309.08600
arXiv 2023
Show all 43 references
-
[9]
DeepSeek-AI . 2025. DeepSeek-R1 : Incentivizing Reasoning Capability in LLMs via Reinforcement Learning . arXiv:2501.12948
2025 arXiv
-
[10]
Goodhart, C. 2015. Goodhart’s law
2015
-
[11]
Grattafiori, A.; Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783
2024 arXiv
-
[12]
Hadfield-Menell, D.; Milli, S.; Abbeel, P.; Russell, S.; and Dragan, A. 2020. Inverse Reward Design. arXiv:1711.02827
2020 arXiv
-
[13]
Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS
2021
-
[14]
Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2023. Let's Verify Step by Step . arXiv:2305.20050
2023 arXiv
-
[15]
Liu, R.; Gao, J.; Zhao, J.; Zhang, K.; Li, X.; Qi, B.; Ouyang, W.; and Zhou, B. 2025. Can 1b llm surpass 405b llm? rethinking compute-optimal test-time scaling. arXiv preprint arXiv:2502.06703
2025 arXiv
-
[16]
Liu, T.; Xiong, W.; Ren, J.; Chen, L.; Wu, J.; Joshi, R.; Gao, Y.; Shen, J.; Qin, Z.; Yu, T.; et al. 2024. Rrm: Robust reward model training mitigates reward hacking. arXiv preprint arXiv:2409.13156
2024 arXiv
-
[17]
P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P
Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; Gupta, S.; Majumder, B. P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P. 2023. Self- Refine : Iterative Refinement with Self-Feedback . In Thirt...
2023
-
[18]
Makelov, A.; Lange, G.; and Nanda, N. 2024. Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control. arXiv:2405.08366
2024 arXiv
-
[19]
Miao, Y.; Zhang, S.; Ding, L.; Bao, R.; Zhang, L.; and Tao, D. 2024. Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling. Advances in Neural Information Processing Systems, 37: 134387--134429
2024
-
[20]
Pan, A.; Jones, E.; Jagadeesan, M.; and Steinhardt, J. 2024. Feedback loops with language models drive in-context reward hacking. arXiv preprint arXiv:2402.06627
2024 arXiv
-
[21]
Pearl, J. 2009. Causality. Cambridge university press
2009
-
[22]
Pearl, J.; Glymour, M.; and Jewell, N. P. 2016. Causal inference in statistics: A primer. John Wiley & Sons
2016
-
[23]
Peters, J.; Janzing, D.; and Sch \"o lkopf, B. 2017. Elements of causal inference: foundations and learning algorithms. The MIT press
2017
-
[24]
Saunders, W.; Yeh, C.; Wu, J.; Bills, S.; Ouyang, L.; Ward, J.; and Leike, J. 2022. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802
2022 arXiv
-
[25]
Sel, B.; Tawaha, A.; Khattar, V.; Jia, R.; and Jin, M. 2024. Algorithm of Thoughts : Enhancing Exploration of Ideas in Large Language Models . In Forty-First International Conference on Machine Learning
2024
-
[26]
Shao, M.; Basit, A.; Karri, R.; and Shafique, M. 2024 a . Survey of Different Large Language Model Architectures: Trends, Benchmarks, and Challenges. IEEE Access, 12: 188664–188706
2024
-
[27]
K.; Wu, Y.; and Guo, D
Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024 b . DeepSeekMath : Pushing the Limits of Mathematical Reasoning in Open Language Models . arXiv:2402.03300
2024 arXiv
-
[28]
Skalse, J.; Howe, N.; Krasheninnikov, D.; and Krueger, D. 2022. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35: 9460--9471
2022
-
[29]
Snell, C.; Lee, J.; Xu, K.; and Kumar, A. 2024 a . Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters . arXiv:2408.03314
2024 arXiv
-
[30]
V.; Lee, J.; Xu, K.; and Kumar, A
Snell, C. V.; Lee, J.; Xu, K.; and Kumar, A. 2024 b . Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Parameters for Reasoning . In The Thirteenth International Conference on Learning Representations
2024
-
[31]
Spirtes, P.; and Zhang, K. 2016. Causal discovery and inference: concepts and recent methodological advances. In Applied informatics, volume 3, 3. Springer
2016
-
[32]
Team, Q. 2024. Qwen2.5: A Party of Foundation Models
2024
-
[33]
L.; McDougall, C.; MacDiarmid, M.; Freeman, C
Templeton, A.; Conerly, T.; Marcus, J.; Lindsey, J.; Bricken, T.; Chen, B.; Pearce, A.; Citro, C.; Ameisen, E.; Jones, A.; Cunningham, H.; Turner, N. L.; McDougall, C.; MacDiarmid, M.; Freeman, C. D.; Sumers, T. R.; Rees, E.; Batson, J.; Jermyn, A.; Carter, S.; Olah, C.; and H...
2024
-
[34]
Uesato, J.; Kushman, N.; Kumar, R.; Song, F.; Siegel, N.; Wang, L.; Creswell, A.; Irving, G.; and Higgins, I. 2022. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275
2022 arXiv
-
[35]
Wang, P.; Li, L.; Shao, Z.; Xu, R.; Dai, D.; Li, Y.; Chen, D.; Wu, Y.; and Sui, Z. 2023. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. arXiv preprint arXiv:2312.08935
2023 arXiv
-
[36]
V.; and Zhou, D
Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; and Zhou, D. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in neural information processing systems, 35: 24824--24837
2022
-
[37]
Weng, L. 2024. Reward Hacking in Reinforcement Learning. lilianweng.github.io
2024
-
[38]
Wu, Y.; Sun, Z.; Li, S.; Welleck, S.; and Yang, Y. 2024. Inference Scaling Laws : An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving . In The Thirteenth International Conference on Learning Representations
2024
-
[39]
Yang, A.; Zhang, B.; Hui, B.; Gao, B.; Yu, B.; Li, C.; Liu, D.; Tu, J.; Zhou, J.; Lin, J.; Lu, K.; Xue, M.; Lin, R.; Liu, T.; Ren, X.; and Zhang, Z. 2024. Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement. arXiv preprint arXiv:2409.12122
2024 arXiv
-
[40]
L.; Cao, Y.; and Narasimhan, K
Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T. L.; Cao, Y.; and Narasimhan, K. R. 2023. Tree of Thoughts : Deliberate Problem Solving with Large Language Models . In Thirty-Seventh Conference on Neural Information Processing Systems
2023
-
[41]
Zhang, Z.; Zhang, A.; Li, M.; and Smola, A. 2022. Automatic Chain of Thought Prompting in Large Language Models . arXiv:2210.03493
2022 arXiv
-
[43]
Zhang, Z.; Zheng, C.; Wu, Y.; Zhang, B.; Lin, R.; Yu, B.; Liu, D.; Zhou, J.; and Lin, J. 2025 b . The Lessons of Developing Process Reward Models in Mathematical Reasoning. arXiv preprint arXiv:2501.07301
2025 arXiv
-
[44]
X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al
Zhao, W. X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2)
2023 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.