{"id":"a2ea2f24-9b69-4b3f-8a83-18c9130a8778","arxiv_id":"2508.04216","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"CRA trains sparse autoencoders on PRM activations and applies backdoor adjustment to estimate true rewards, reducing reward hacking in math reasoning.","lead":"This paper proposes Causal Reward Adjustment (CRA), which uses sparse autoencoders and backdoor adjustment to remove confusing internal features from process reward model scores. If it works, AI math reasoning becomes more reliable without retraining the policy or reward model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CRA's backdoor adjustment is only valid if SAE-recovered features are true confounders; the abstract offers no causal graph or validation, leaving the central claim unsupported.","rationale":"The reader's verdict is UNVERDICTED because only the abstract was available. My stress-test focuses on the core causal identification step, which is indeed the weakest assumption: the abstract's claim that CRA 'corrects confounding by using backdoor adjustment' requires that the SAE features are a valid adjustment set for the causal model. The abstract does not specify the causal graph, the treatment/outcome variables, or any validation that the recovered features are the true confounders. This is not an external disagreement but an internal missing justification: if the features are not causal, the backdoor adjustment formula does not estimate a causal reward, and the reported accuracy gains could stem from an arbitrary feature transformation that happens to help ranking. My proposed test is a concrete way to check whether the SAE features correspond to a known confounder and whether the adjustment removes its effect. Since the concern directly supports the reader's UNVERDICTED verdict, I recommend no change.","tokens_in":624,"tokens_out":3892,"duration_ms":44556,"concrete_test":"Use a synthetic or semi-synthetic setup with a known confounder: take a PRM and a set of reasoning paths where a particular semantic feature (e.g., answer verbosity) is known to inflate PRM scores without affecting correctness. Split paths into strata of that feature. Run CRA (train SAE on PRM activations, compute backdoor-adjusted reward). Test whether, after adjustment, the probability of selecting the correct answer is uniform across confounder strata (i.e., the confounding bias is removed) and whether the adjusted reward rank-correlates with true correctness better than the raw PRM. If the adjusted reward still varies with the known confounder within fixed correctness, the SAE features did not capture the true confounder and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract, 'Causal Reward Adjustment') depends entirely on identifying a set of confounding semantic features and applying backdoor adjustment. The abstract states CRA 'trains sparse autoencoders on the PRM's internal activations to recover interpretable features' and 'corrects confounding by using backdoor adjustment,' but gives no causal graph, no definition of the target estimand, and no evidence that the SAE features are valid adjustment variables. Backdoor adjustment requires that the adjustment set Z blocks all backdoor paths from the treatment to the outcome and does not open any paths; if the SAE features are either incomplete (hidden confounder omitted) or non-causal (noise directions or colliders), the adjusted reward is not the true reward. Since the SAE is trained unsupervised on the same PRM activations whose bias is to be corrected, there is a risk that the features merely re-encode the PRM's own spurious correlations, making the adjustment circular. Without a demonstration that the recovered features correspond to the actual confounders (e.g., through causal discovery or intervention), the reported accuracy gains may be due to a general feature-selection artifact rather than causal correction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Causal Reward Adjustment (CRA), a method to mitigate reward hacking in PRM-based external reasoning systems. CRA trains sparse autoencoders on PRM internal activations to recover interpretable features, then uses backdoor adjustment to correct confounding semantic features, thereby estimating the true reward of a reasoning path. The abstract reports experiments on math solving datasets showing that CRA mitigates reward hacking and improves final accuracy without modifying the policy model or retraining the PRM.","tokens_in":949,"tokens_out":2428,"duration_ms":27540,"significance":"If CRA works as claimed, it introduces a causal perspective on reward hacking and offers a practical, training-free correction that could improve the reliability of process reward models in reasoning tasks. The use of sparse autoencoder features as adjustment variables is an interesting bridge between interpretability and causal inference. However, the abstract alone provides no verifiable evidence: no causal graph, no identification assumptions, no quantitative results, and no validation that the SAE features are valid confounders. The significance therefore remains conditional on the full paper supplying these missing elements.","major_comments":[{"comment":"The central claim that CRA 'corrects confounding by using backdoor adjustment' is not verifiable from the abstract. Backdoor adjustment requires a well-defined treatment, outcome, and adjustment set that satisfies the backdoor criterion in an explicit causal graph. The abstract gives none of these: no DAG, no target estimand, no justification that the SAE-recovered features block all backdoor paths and do not open collider paths. If the full paper does not provide this, the 'true reward' is not identified and the method is not causal.","section":"Abstract"},{"comment":"There is a potential circularity: the SAE is trained on the same PRM activations that are suspected of encoding spurious correlations. If the recovered features merely re-encode the PRM's biases, then adjusting for them may remove legitimate signal or reinforce the reward hacking. The abstract provides no evidence of causal validity of these feature directions, such as causal discovery, intervention tests, or at least a clear argument that the features correspond to semantic confounders rather than arbitrary linear directions.","section":"Abstract"},{"comment":"The empirical claim ('improves final accuracy') is reported without any numbers, baselines, or error bars. No comparison to existing reward-hacking mitigation methods is given, nor is there any indication of statistical significance. Since the paper's support rests on these experiments, the abstract's bare assertion is insufficient for the reader to judge whether the improvement is real or a feature-selection artifact.","section":"Abstract"}],"minor_comments":[{"comment":"The term 'reward hacking' should be defined explicitly; the abstract implies high-scoring but logically incorrect paths, but a formal definition and relationship to similar concepts in the literature would help.","section":"Abstract"},{"comment":"The phrase 'without modifying the policy model or retraining PRM' is clear at a high level, but it would help to specify what components are changed (e.g., only a post-hoc reward adjustment layer) and what overhead is introduced.","section":"Abstract"},{"comment":"Please provide a precise mathematical expression for the backdoor adjustment used, e.g., the corrected reward formula in terms of the SAE features and their conditional distributions.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript text provided to me consisted only of the abstract; I could not inspect the full derivation or experiments. The abstract itself is too thin to support the causal claims. I recommend the editor obtain the full paper and verify that it contains the missing causal graph, identifiability conditions, and empirical details before further consideration. If these are present, the paper may be viable; if not, the current abstract should be revised or the claims softened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this is a genuinely new combination—using sparse autoencoder features from PRM activations as adjustment variables in a backdoor correction for reward hacking. The abstract is clearly written and the causal framing is a reasonable instinct: if PRM scores are confounded by semantic features that correlate with style rather than correctness, adjusting for them is a sensible post hoc fix. Credit where due: the method doesn't require retraining the PRM or modifying the policy, which is practically attractive.\n\nThe soft spots are real, but they are soft spots in the abstract, not necessarily in the full paper. The load-bearing step is the backdoor adjustment. The abstract never shows the causal graph, never names the target estimand, and never justifies that the SAE-recovered features are the right adjustment set. Backdoor adjustment is only valid if you block all backdoor paths without opening new ones; if the SAE features are incomplete, non-causal noise directions, or colliders, the corrected reward is not the true reward. There is also a circularity risk: the SAE is trained on the same PRM activations whose bias you are trying to correct, so the features may simply re-encode the PRM's own spurious correlations. The abstract reports accuracy gains but no baselines, effect sizes, or error bars, so I can't tell whether the gains are meaningful or an artifact of feature selection.\n\nThat said, none of these are fatal if the full paper supplies the missing identification assumptions and validates the features—for example, through causal discovery, interventions, or ablations showing the features track known confounders. The stress-test note is on target, but it is a demand for evidence, not evidence of failure.\n\nThis paper is for researchers working on reward hacking, process reward models, and causal methods for LLM reasoning. If the full text delivers on the abstract's promise, it's worth citing. As it stands, the abstract alone isn't enough to trust the result, but the idea is serious enough to merit a real peer review. Send it to reviewers, with the expectation that they push hard on the causal identification and the feature validation.","headline":"A plausible new combination of SAE feature recovery and backdoor adjustment for reward hacking, but the abstract alone leaves the causal identification burden unmet.","tokens_in":1310,"tokens_out":1473,"would_cite":false,"duration_ms":18033,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Causal Reward Adjustment treats reward hacking in PRM-based reasoning as a confounding problem and corrects it with backdoor adjustment over sparse-autoencoder features, improving math accuracy without retraining the reward model.","keywords":["reward hacking","process reward models","causal inference","backdoor adjustment","sparse autoencoders","mathematical reasoning","external reasoning"],"falsifier":"On a math benchmark with known ground-truth labels, take a set of reasoning paths and artificially vary a known spurious feature, such as verbosity or overconfident phrasing, while holding the actual solution fixed. If CRA's adjusted score still ranks the spuriously confident wrong path above the correct one as often as the raw PRM score does, then the backdoor adjustment is not removing the confounding it claims to remove.","tokens_in":595,"feed_emoji":"🧮","tokens_out":5849,"duration_ms":68872,"temperature":0.7,"pith_summary":"This paper argues that reward hacking in PRM-based reasoning systems is not just a scoring flaw but a causal inference problem: the reward model's scores are distorted by confounding semantic features that are not actually tied to logical correctness. The authors propose Causal Reward Adjustment (CRA), which trains sparse autoencoders on the PRM's internal activations to isolate interpretable features, then applies backdoor adjustment to remove the confounding influence and estimate the true reward of a reasoning path. On mathematical problem-solving datasets, CRA reduces the number of high-scoring but wrong paths selected and improves final answer accuracy, without modifying the policy model or retraining the PRM. If the causal story is right, any PRM-based selection system can be made more trustworthy by correcting its scores rather than replacing the model.","feed_headline":"Backdoor correction stops reward hacking in math reasoning","feed_subtitle":"Adjusting PRM scores for confounding features selects more correct paths without retraining the model.","key_machinery":"The load-bearing identity is the backdoor adjustment formula from causal inference. Let $Z$ be the set of semantic features recovered from the PRM's internal activations by the sparse autoencoder, and let $R$ be the reward assigned to a reasoning path. Instead of using $R$ directly, CRA estimates the adjusted reward $\\sum_z \\mathbb{E}[R \\mid \\text{path quality}, z]\\,P(z)$, which strips out the spurious association between $Z$ and high scores. The sparse autoencoder is the enabler: it converts high-dimensional, opaque activations into a discrete set of interpretable features that can be conditioned on in the adjustment.","core_discovery":"The central claim is that high-scoring but logically incorrect reasoning paths arise because the process reward model's score is confounded by semantic features that correlate with surface characteristics rather than with correctness. CRA recovers these features from the PRM's internal activations using a sparse autoencoder, then applies backdoor adjustment to estimate the reward a path would receive if those confounders were held fixed. Experiments on math solving datasets show that this adjusted reward selects more correct reasoning paths than the raw PRM score, mitigating reward hacking and improving final accuracy. The method works without changing the policy model or retraining the PRM,","pith_inferences":["The same logic would apply beyond math: any verifier whose internal activations carry style or surface confounders could be corrected with the same recipe, as long as sparse autoencoders recover the confounders.","A direct corollary the paper does not draw: the difference between raw and adjusted reward on a given path is a measurement of how 'hacked' that path is, which could be used as a diagnostic or a search heuristic.","If the feature set is incomplete, the adjusted reward inherits residual confounding; this suggests the method's ceiling on a dataset is set by how faithfully the sparse autoencoder captures the PRM's actual shortcut features.","A testable extension: use the same adjustment on a reward model trained for a different task, such as code generation or planning, and compare selection accuracy, which would separate the causal claim from the specifics of math datasets."],"forward_implications":["Raw PRM scores are a biased selection signal because of confounding semantic features; CRA's adjusted score is the better signal for choosing reasoning paths.","CRA reduces reward hacking and improves final accuracy on math problem-solving datasets.","The method requires no modification of the policy model and no retraining of the PRM, so it can be layered onto existing systems.","Sparse autoencoder features learned from PRM activations can stand in for confounders in the backdoor adjustment."],"supporting_citations":[],"fun_headline_variants":["Backdoor fix stops reward hacking in math","Causal reward adjustment beats reward hacking","Correct PRM scores without retraining","Sparse autoencoder corrects confounded rewards"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The whole method rests on the assumption that the features recovered by the sparse autoencoder are exactly the confounders, and that no important confounder is left out.","fun_headline_variants_meta":{"raw":{"variants":["Backdoor fix stops reward hacking in math","Causal reward adjustment beats reward hacking","Correct PRM scores without retraining","Sparse autoencoder corrects confounded rewards"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000643,"raw_usage":{"total_tokens":2746,"prompt_tokens":650,"completion_tokens":2096,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":394,"completion_tokens_details":{"reasoning_tokens":2054}},"tokens_in":394,"tokens_out":2096,"duration_ms":18170,"temperature":1.0,"reasoning_tokens":2054,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:46:37.377362+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a math benchmark with known ground-truth labels, take a set of reasoning paths and artificially vary a known spurious feature, such as verbosity or overconfident phrasing, while holding the actual solution fixed. If CRA's adjusted score still ranks the spuriously confident wrong path above the correct one as often as the raw PRM score does, then the backdoor adjustment is not removing the confounding it claims to remove.","supporting_citations":[],"review_version":1}