{"paper":{"title":"Uncertainty-Aware Reward Discounting for Mitigating Reward Hacking","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"A dual-source uncertainty framework using ensemble disagreement and preference variability reduces reward hacking by 93.7 percent in RL.","cross_cats":["cs.AI"],"primary_cat":"cs.LG","authors_text":"Disha Singha","submitted_at":"2026-04-29T07:14:01Z","abstract_excerpt":"Reinforcement learning from human feedback (RLHF) systems face a compounding alignment challenge: not only are learned reward models uncertain about unseen state-action pairs, but the human preference annotations they are trained on are themselves inconsistent, context-dependent, and noisy. Existing approaches address these uncertainty sources in isolation - epistemic uncertainty is used to guide exploration, while preference uncertainty is absorbed during reward model training but discarded during policy optimization. We introduce Uncertainty-Aware Reward Discounting (UARD), a principled fram"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Empirical results across multiple discrete grid configurations (6x6, 8x8, 10x10) and high-dimensional continuous control environments (Hopper-v4, Walker2d-v4) demonstrate that our approach yields more stable training dynamics and reduces exploitative behaviors under reward ambiguity, achieving a 93.7% reduction in reward-hacking behavior as measured by trap visitation frequency.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That ensemble disagreement reliably captures epistemic uncertainty relevant to reward hacking and that variability in reward annotations accurately reflects true preference uncertainty, allowing the Reliability Filter to correctly balance exploitation and caution without discarding useful actions.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Uncertainty-aware RL framework using ensemble disagreement and annotation variability reduces reward-hacking trap visits by 93.7% across grid and continuous control tasks while remaining robust to 30% label noise.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"A dual-source uncertainty framework using ensemble disagreement and preference variability reduces reward hacking by 93.7 percent in RL.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"60be28680cb8188bd071b19aebb57658e1840fe2ca2a847d26b9ee0a24b632ed"},"source":{"id":"2604.26360","kind":"arxiv","version":2},"verdict":{"id":"25f1b1c8-d859-4749-8029-2010b5b960bc","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-07T13:40:17.797121Z","strongest_claim":"Empirical results across multiple discrete grid configurations (6x6, 8x8, 10x10) and high-dimensional continuous control environments (Hopper-v4, Walker2d-v4) demonstrate that our approach yields more stable training dynamics and reduces exploitative behaviors under reward ambiguity, achieving a 93.7% reduction in reward-hacking behavior as measured by trap visitation frequency.","one_line_summary":"Uncertainty-aware RL framework using ensemble disagreement and annotation variability reduces reward-hacking trap visits by 93.7% across grid and continuous control tasks while remaining robust to 30% label noise.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That ensemble disagreement reliably captures epistemic uncertainty relevant to reward hacking and that variability in reward annotations accurately reflects true preference uncertainty, allowing the Reliability Filter to correctly balance exploitation and caution without discarding useful actions.","pith_extraction_headline":"A dual-source uncertainty framework using ensemble disagreement and preference variability reduces reward hacking by 93.7 percent in RL."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2604.26360/integrity.json","findings":[],"available":true,"detectors_run":[{"name":"ai_meta_artifact","ran_at":"2026-05-21T00:38:09.146708Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"doi_compliance","ran_at":"2026-05-19T20:13:22.791812Z","status":"completed","version":"1.0.0","findings_count":0}],"snapshot_sha256":"aff1ed6b046a8cd09078e95b495d50225e3a67058930dcb942e61a55832a962b"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}