{"arxiv_id":"2608.15445","title":"Measuring Reward Hacking and Reasoning-Answer Decoupling Under Position-Confounded Optimization","abstract":"When a reward is correct on every training example yet consistent with more than one goal, a model can acquire an unintended one, a failure known as goal misgeneralization. Endpoint accuracy on the training distribution cannot tell the two apart, because solving the task and exploiting a surface feature can satisfy the reward equally well. We treat this as a measurement problem: what does a benchmark score measure once a model has been optimized against a correct but confounded signal? We train language models with GRPO on multiple-choice math problems where the correct answer is always option A, then evaluate on an unseen test set with unbiased answer positions. Across Qwen2.5, Llama 3.x and Gemma-3 models, biased training often drives option-A rates above 0.90 in smaller models and collapses unbiased accuracy toward chance, so accuracy stops measuring math ability and instead measures an answer-position policy. We further find reasoning-answer decoupling: capable models generate reasoning that reaches the correct numeric answer while still selecting A. We track this with numeric extraction and an LLM judge (GPT-4.1-mini; Qwen2.5-3B decoupling rate is about 0.66). The broken construct generalizes beyond the training domain: biased models inflate A-rates on out-of-domain MMLU and value-laden prompts. Continued training on unbiased data reverses the in-domain shift unevenly and only partially reverses the out-of-domain one, so a model can appear restored on its training distribution while remaining biased on unseen inputs. Reasoning-answer decoupling rate, together with answer distributions and out-of-domain behavior, separates capability loss from a learned, transferable shortcut.","primary_cat":"cs.AI","cross_cats":[],"submitted_at":"2026-08-15T23:13:57+00:00","latest_version":1,"source":"arxiv","source_label":"arXiv","source_url":null,"has_pdf":true,"has_source":false,"verdict":"CONDITIONAL","confidence":"MODERATE","one_line_summary":"Biased training that always places the correct answer at option A makes many small language models select A over 90% of the time and collapse unbiased accuracy toward chance, while their reasoning traces still compute correct answers.","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","upvotes":0,"downvotes":0,"pith_score":0.0,"inbound_count":2,"_links":{"html":"https://pith.science/paper/2608.15445","verdict":"/api/v1/verdicts/2608.15445"}}