Pith. sign in

REVIEW 3 major objections 3 minor 2 cited by

Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correction

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read Causal Reward Adjustment treats reward hacking in PRM-based reasoning as a confounding problem and corrects it with backdoor adjustment over sparse-autoencoder features, improving math accuracy without retraining the reward model.

desk verdict A plausible new combination of SAE feature recovery and backdoor adjustment for reward hacking, but the abstract alone leaves the causal identification burden unmet. read the letter →

arxiv 2508.04216 v1 pith:V3M4SFXS submitted 2025-08-06 cs.LG

classification cs.LG
keywords rewardhackingprocessmodelscausalinferencebackdooradjustmentsparseautoencodersmathematicalreasoningexternal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that reward hacking in PRM-based reasoning systems is not just a scoring flaw but a causal inference problem: the reward model's scores are distorted by confounding semantic features that are not actually tied to logical correctness. The authors propose Causal Reward Adjustment (CRA), which trains sparse autoencoders on the PRM's internal activations to isolate interpretable features, then applies backdoor adjustment to remove the confounding influence and estimate the true reward of a reasoning path. On mathematical problem-solving datasets, CRA reduces the number of high-scoring but wrong paths selected and improves final answer accuracy, without modifying the policy model or retraining the PRM. If the causal story is right, any PRM-based selection system can be made more trustworthy by correcting its scores rather than replacing the model.

What carries the argument

The load-bearing identity is the backdoor adjustment formula from causal inference. Let $Z$ be the set of semantic features recovered from the PRM's internal activations by the sparse autoencoder, and let $R$ be the reward assigned to a reasoning path. Instead of using $R$ directly, CRA estimates the adjusted reward $\sum_z \mathbb{E}[R \mid \text{path quality}, z]\,P(z)$, which strips out the spurious association between $Z$ and high scores. The sparse autoencoder is the enabler: it converts high-dimensional, opaque activations into a discrete set of interpretable features that can be conditioned on in the adjustment.

What would settle it

On a math benchmark with known ground-truth labels, take a set of reasoning paths and artificially vary a known spurious feature, such as verbosity or overconfident phrasing, while holding the actual solution fixed. If CRA's adjusted score still ranks the spuriously confident wrong path above the correct one as often as the raw PRM score does, then the backdoor adjustment is not removing the confounding it claims to remove.

Watch

Extended reading notes

Core claim

The central claim is that high-scoring but logically incorrect reasoning paths arise because the process reward model's score is confounded by semantic features that correlate with surface characteristics rather than with correctness. CRA recovers these features from the PRM's internal activations using a sparse autoencoder, then applies backdoor adjustment to estimate the reward a path would receive if those confounders were held fixed. Experiments on math solving datasets show that this adjusted reward selects more correct reasoning paths than the raw PRM score, mitigating reward hacking and improving final accuracy. The method works without changing the policy model or retraining the PRM,

Load-bearing premise

The whole method rests on the assumption that the features recovered by the sparse autoencoder are exactly the confounders, and that no important confounder is left out.

Editorial extensions

If this is right

  • Raw PRM scores are a biased selection signal because of confounding semantic features; CRA's adjusted score is the better signal for choosing reasoning paths.
  • CRA reduces reward hacking and improves final accuracy on math problem-solving datasets.
  • The method requires no modification of the policy model and no retraining of the PRM, so it can be layered onto existing systems.
  • Sparse autoencoder features learned from PRM activations can stand in for confounders in the backdoor adjustment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same logic would apply beyond math: any verifier whose internal activations carry style or surface confounders could be corrected with the same recipe, as long as sparse autoencoders recover the confounders.
  • A direct corollary the paper does not draw: the difference between raw and adjusted reward on a given path is a measurement of how 'hacked' that path is, which could be used as a diagnostic or a search heuristic.
  • If the feature set is incomplete, the adjusted reward inherits residual confounding; this suggests the method's ceiling on a dataset is set by how faithfully the sparse autoencoder captures the PRM's actual shortcut features.
  • A testable extension: use the same adjustment on a reward model trained for a different task, such as code generation or planning, and compare selection accuracy, which would separate the causal claim from the specifics of math datasets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes Causal Reward Adjustment (CRA), a method to mitigate reward hacking in PRM-based external reasoning systems. CRA trains sparse autoencoders on PRM internal activations to recover interpretable features, then uses backdoor adjustment to correct confounding semantic features, thereby estimating the true reward of a reasoning path. The abstract reports experiments on math solving datasets showing that CRA mitigates reward hacking and improves final accuracy without modifying the policy model or retraining the PRM.

Significance. If CRA works as claimed, it introduces a causal perspective on reward hacking and offers a practical, training-free correction that could improve the reliability of process reward models in reasoning tasks. The use of sparse autoencoder features as adjustment variables is an interesting bridge between interpretability and causal inference. However, the abstract alone provides no verifiable evidence: no causal graph, no identification assumptions, no quantitative results, and no validation that the SAE features are valid confounders. The significance therefore remains conditional on the full paper supplying these missing elements.

major comments (3)
  1. [Abstract] The central claim that CRA 'corrects confounding by using backdoor adjustment' is not verifiable from the abstract. Backdoor adjustment requires a well-defined treatment, outcome, and adjustment set that satisfies the backdoor criterion in an explicit causal graph. The abstract gives none of these: no DAG, no target estimand, no justification that the SAE-recovered features block all backdoor paths and do not open collider paths. If the full paper does not provide this, the 'true reward' is not identified and the method is not causal.
  2. [Abstract] There is a potential circularity: the SAE is trained on the same PRM activations that are suspected of encoding spurious correlations. If the recovered features merely re-encode the PRM's biases, then adjusting for them may remove legitimate signal or reinforce the reward hacking. The abstract provides no evidence of causal validity of these feature directions, such as causal discovery, intervention tests, or at least a clear argument that the features correspond to semantic confounders rather than arbitrary linear directions.
  3. [Abstract] The empirical claim ('improves final accuracy') is reported without any numbers, baselines, or error bars. No comparison to existing reward-hacking mitigation methods is given, nor is there any indication of statistical significance. Since the paper's support rests on these experiments, the abstract's bare assertion is insufficient for the reader to judge whether the improvement is real or a feature-selection artifact.
minor comments (3)
  1. [Abstract] The term 'reward hacking' should be defined explicitly; the abstract implies high-scoring but logically incorrect paths, but a formal definition and relationship to similar concepts in the literature would help.
  2. [Abstract] The phrase 'without modifying the policy model or retraining PRM' is clear at a high level, but it would help to specify what components are changed (e.g., only a post-hoc reward adjustment layer) and what overhead is introduced.
  3. [Abstract] Please provide a precise mathematical expression for the backdoor adjustment used, e.g., the corrected reward formula in terms of the SAE features and their conditional distributions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CRA's adjusted reward is validated against external answer accuracy, not derived from it.

full rationale

The paper's central claim is that PRM reward hacking is caused by confounding semantic features and that CRA corrects this via sparse-autoencoder features and backdoor adjustment. The adjusted reward is then evaluated by final problem-solving accuracy on math datasets. This is an external benchmark, not a quantity that is definitionally equal to the SAE features or the raw PRM scores. The main vulnerability—that the SAE features may be incomplete or non-causal—is a validity threat to the causal identification, not a circular reduction: backdoor adjustment is an assumption about the causal graph, and the correctness of that assumption is an empirical identifiability matter, not a logical equivalence between input and output. No equation in the provided text defines the target reward in terms of the fitted features, and no fitted parameter is renamed as a prediction. The abstract alone provides no evidence of self-citation, ansatz smuggling, or renaming of a known result. Ascribing circularity here would require speculative claims about feature selection post hoc, which the hard rules prohibit without exhibitable reduction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The abstract does not provide enough detail to derive the method from first principles; it relies on the existence of confounding semantic features, the correctness of the causal graph used for backdoor adjustment, and the representativeness of the math benchmarks. The sparse autoencoder hyperparameters and the choice of confounder features are the main unstated degrees of freedom.

free parameters (2)
  • SAE dictionary size and sparsity coefficient = not reported in abstract
    Choice of sparse autoencoder capacity and sparsity penalty changes which features are recovered; these are hand-tuned hyperparameters in practice.
  • Confounder feature set used in the backdoor adjustment = not reported in abstract
    The specific features treated as confounders are a modeling choice; if selected after inspecting validation performance, they function as fitted parameters.
assumptions (3)
  • domain assumption Reward hacking in PRM selection is primarily caused by confounding semantic features in the PRM's internal activations.
    Abstract's causal attribution; if false, there is nothing to adjust and the method lacks a target.
  • domain assumption The sparse autoencoder features are the actual confounders, and the causal graph satisfies the backdoor criterion (no unmeasured confounding of score and correctness).
    Backdoor adjustment is only valid under this identification condition, which the abstract does not demonstrate.
  • domain assumption Math solving benchmarks used in experiments are representative enough to generalize the claim beyond the evaluated datasets.
    Abstract reports improvements on math solving datasets; external validity is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correction." pith.science (2026). https://pith.science/paper/V3M4SFXS

@misc{pith2026250804216,
  author       = {Pith},
  title        = {Pith review of: Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V3M4SFXS}},
  note         = {Machine review of arXiv:2508.04216}
}
read the original abstract

External reasoning systems combine language models with process reward models (PRMs) to select high-quality reasoning paths for complex tasks such as mathematical problem solving. However, these systems are prone to reward hacking, where high-scoring but logically incorrect paths are assigned high scores by the PRMs, leading to incorrect answers. From a causal inference perspective, we attribute this phenomenon primarily to the presence of confounding semantic features. To address it, we propose Causal Reward Adjustment (CRA), a method that mitigates reward hacking by estimating the true reward of a reasoning path. CRA trains sparse autoencoders on the PRM's internal activations to recover interpretable features, then corrects confounding by using backdoor adjustment. Experiments on math solving datasets demonstrate that CRA mitigates reward hacking and improves final accuracy, without modifying the policy model or retraining PRM.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure

    cs.AI 2026-05 accept novelty 7.0 of 10

    Single-axis reward bias mitigations redirect optimization pressure to correlated proxies, and audit-distribution scoring produces identical observables for successful mitigation, bias substitution, and overcorrection.

  2. Factored Causal Representation Learning for Robust Reward Modeling in RLHF

    cs.LG 2026-01 unverdicted novelty 6.0 of 10

    A factored causal representation learning method improves robustness of reward models in RLHF by isolating causal factors from biases like length and sycophancy using adversarial gradient reversal.

Reference graph

Works this paper leans on

43 extracted references · 15 canonical work pages · cited by 2 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Amodei, D.; Olah, C.; Steinhardt, J.; Christiano, P.; Schulman, J.; and Mané, D. 2016. Concrete Problems in AI Safety. arXiv:1606.06565

  4. [4]

    Besta, M.; Blach, N.; Kubicek, A.; Gerstenberger, R.; Podstawski, M.; Gianinazzi, L.; Gajda, J.; Lehmann, T.; Niewiadomski, H.; and Nyczyk, P. 2024. Graph of Thoughts: Solving Elaborate Problems with Large Language Models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, 17682--17690

  5. [5]

    E.; Hume, T.; Carter, S.; Henighan, T.; and Olah, C

    Bricken, T.; Templeton, A.; Batson, J.; Chen, B.; Jermyn, A.; Conerly, T.; Turner, N.; Anil, C.; Denison, C.; Askell, A.; Lasenby, R.; Wu, Y.; Kravec, S.; Schiefer, N.; Maxwell, T.; Joseph, N.; Hatfield-Dodds, Z.; Tamkin, A.; Nguyen, K.; McLean, B.; Burke, J. E.; Hume, T.; Carter, S.; Henighan, T.; and Olah, C. 2023. Towards Monosemanticity: Decomposing L...

  6. [6]

    V.; R \'e , C.; and Mirhoseini, A

    Brown, B.; Juravsky, J.; Ehrlich, R.; Clark, R.; Le, Q. V.; R \'e , C.; and Mirhoseini, A. 2024. Large Language Monkeys : Scaling Inference Compute with Repeated Sampling . arXiv:2407.21787

  7. [7]

    Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168

  8. [8]

    Cunningham, H.; Ewart, A.; Riggs, L.; Huben, R.; and Sharkey, L. 2023. Sparse Autoencoders Find Highly Interpretable Features in Language Models. arXiv:2309.08600

Show all 43 references
  1. [9]

    DeepSeek-AI . 2025. DeepSeek-R1 : Incentivizing Reasoning Capability in LLMs via Reinforcement Learning . arXiv:2501.12948

  2. [10]

    Goodhart, C. 2015. Goodhart’s law

  3. [11]

    Grattafiori, A.; Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783

  4. [12]

    Hadfield-Menell, D.; Milli, S.; Abbeel, P.; Russell, S.; and Dragan, A. 2020. Inverse Reward Design. arXiv:1711.02827

  5. [13]

    Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS

  6. [14]

    Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2023. Let's Verify Step by Step . arXiv:2305.20050

  7. [15]

    Liu, R.; Gao, J.; Zhao, J.; Zhang, K.; Li, X.; Qi, B.; Ouyang, W.; and Zhou, B. 2025. Can 1b llm surpass 405b llm? rethinking compute-optimal test-time scaling. arXiv preprint arXiv:2502.06703

  8. [16]

    Liu, T.; Xiong, W.; Ren, J.; Chen, L.; Wu, J.; Joshi, R.; Gao, Y.; Shen, J.; Qin, Z.; Yu, T.; et al. 2024. Rrm: Robust reward model training mitigates reward hacking. arXiv preprint arXiv:2409.13156

  9. [17]

    P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P

    Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; Gupta, S.; Majumder, B. P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P. 2023. Self- Refine : Iterative Refinement with Self-Feedback . In Thirt...

  10. [18]

    Makelov, A.; Lange, G.; and Nanda, N. 2024. Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control. arXiv:2405.08366

  11. [19]

    Miao, Y.; Zhang, S.; Ding, L.; Bao, R.; Zhang, L.; and Tao, D. 2024. Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling. Advances in Neural Information Processing Systems, 37: 134387--134429

  12. [20]

    Pan, A.; Jones, E.; Jagadeesan, M.; and Steinhardt, J. 2024. Feedback loops with language models drive in-context reward hacking. arXiv preprint arXiv:2402.06627

  13. [21]

    Pearl, J. 2009. Causality. Cambridge university press

  14. [22]

    Pearl, J.; Glymour, M.; and Jewell, N. P. 2016. Causal inference in statistics: A primer. John Wiley & Sons

  15. [23]

    Peters, J.; Janzing, D.; and Sch \"o lkopf, B. 2017. Elements of causal inference: foundations and learning algorithms. The MIT press

  16. [24]

    Saunders, W.; Yeh, C.; Wu, J.; Bills, S.; Ouyang, L.; Ward, J.; and Leike, J. 2022. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802

  17. [25]

    Sel, B.; Tawaha, A.; Khattar, V.; Jia, R.; and Jin, M. 2024. Algorithm of Thoughts : Enhancing Exploration of Ideas in Large Language Models . In Forty-First International Conference on Machine Learning

  18. [26]

    Shao, M.; Basit, A.; Karri, R.; and Shafique, M. 2024 a . Survey of Different Large Language Model Architectures: Trends, Benchmarks, and Challenges. IEEE Access, 12: 188664–188706

  19. [27]

    K.; Wu, Y.; and Guo, D

    Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024 b . DeepSeekMath : Pushing the Limits of Mathematical Reasoning in Open Language Models . arXiv:2402.03300

  20. [28]

    Skalse, J.; Howe, N.; Krasheninnikov, D.; and Krueger, D. 2022. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35: 9460--9471

  21. [29]

    Snell, C.; Lee, J.; Xu, K.; and Kumar, A. 2024 a . Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters . arXiv:2408.03314

  22. [30]

    V.; Lee, J.; Xu, K.; and Kumar, A

    Snell, C. V.; Lee, J.; Xu, K.; and Kumar, A. 2024 b . Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Parameters for Reasoning . In The Thirteenth International Conference on Learning Representations

  23. [31]

    Spirtes, P.; and Zhang, K. 2016. Causal discovery and inference: concepts and recent methodological advances. In Applied informatics, volume 3, 3. Springer

  24. [32]

    Team, Q. 2024. Qwen2.5: A Party of Foundation Models

  25. [33]

    L.; McDougall, C.; MacDiarmid, M.; Freeman, C

    Templeton, A.; Conerly, T.; Marcus, J.; Lindsey, J.; Bricken, T.; Chen, B.; Pearce, A.; Citro, C.; Ameisen, E.; Jones, A.; Cunningham, H.; Turner, N. L.; McDougall, C.; MacDiarmid, M.; Freeman, C. D.; Sumers, T. R.; Rees, E.; Batson, J.; Jermyn, A.; Carter, S.; Olah, C.; and H...

  26. [34]

    Uesato, J.; Kushman, N.; Kumar, R.; Song, F.; Siegel, N.; Wang, L.; Creswell, A.; Irving, G.; and Higgins, I. 2022. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275

  27. [35]

    Wang, P.; Li, L.; Shao, Z.; Xu, R.; Dai, D.; Li, Y.; Chen, D.; Wu, Y.; and Sui, Z. 2023. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. arXiv preprint arXiv:2312.08935

  28. [36]

    V.; and Zhou, D

    Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; and Zhou, D. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in neural information processing systems, 35: 24824--24837

  29. [37]

    Weng, L. 2024. Reward Hacking in Reinforcement Learning. lilianweng.github.io

  30. [38]

    Wu, Y.; Sun, Z.; Li, S.; Welleck, S.; and Yang, Y. 2024. Inference Scaling Laws : An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving . In The Thirteenth International Conference on Learning Representations

  31. [39]

    Yang, A.; Zhang, B.; Hui, B.; Gao, B.; Yu, B.; Li, C.; Liu, D.; Tu, J.; Zhou, J.; Lin, J.; Lu, K.; Xue, M.; Lin, R.; Liu, T.; Ren, X.; and Zhang, Z. 2024. Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement. arXiv preprint arXiv:2409.12122

  32. [40]

    L.; Cao, Y.; and Narasimhan, K

    Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T. L.; Cao, Y.; and Narasimhan, K. R. 2023. Tree of Thoughts : Deliberate Problem Solving with Large Language Models . In Thirty-Seventh Conference on Neural Information Processing Systems

  33. [41]

    Zhang, Z.; Zhang, A.; Li, M.; and Smola, A. 2022. Automatic Chain of Thought Prompting in Large Language Models . arXiv:2210.03493

  34. [43]

    Zhang, Z.; Zheng, C.; Wu, Y.; Zhang, B.; Lin, R.; Yu, B.; Liu, D.; Zhou, J.; and Lin, J. 2025 b . The Lessons of Developing Process Reward Models in Mathematical Reasoning. arXiv preprint arXiv:2501.07301

  35. [44]

    X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al

    Zhao, W. X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2)

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.