{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2023:PYCDSKS33CWPGXPQZTU5GRC7S3","short_pith_number":"pith:PYCDSKS3","schema_version":"1.0","canonical_sha256":"7e04392a5bd8acf35df0cce9d3445f96eeee1e2aaeeeb630a6bb86c4eb5037cc","source":{"kind":"arxiv","id":"2312.09244","version":3},"attestation_state":"computed","paper":{"title":"Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"","cross_cats":[],"primary_cat":"cs.LG","authors_text":"Adam Fisch, Ahmad Beirami, Alekh Agarwal, Alex D'Amour, Chirag Nagpal, Deepak Ramachandran, Dj Dvijotham, Jacob Eisenstein, Jonathan Berant, Katherine Heller, Peter Shaw, Stephen Pfohl","submitted_at":"2023-12-14T18:59:04Z","abstract_excerpt":"Reward models play a key role in aligning language model applications towards human preferences. However, this setup creates an incentive for the language model to exploit errors in the reward model to achieve high estimated reward, a phenomenon often termed \\emph{reward hacking}. A natural mitigation is to train an ensemble of reward models, aggregating over model outputs to obtain a more robust reward estimate. We explore the application of reward ensembles to alignment at both training time (through reinforcement learning) and inference time (through reranking). First, we show that reward m"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2312.09244","kind":"arxiv","version":3},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.LG","submitted_at":"2023-12-14T18:59:04Z","cross_cats_sorted":[],"title_canon_sha256":"ab9920f313ad944c347bf2ed85e2598707334ad891bad6cf207e94c66f0ae623","abstract_canon_sha256":"8fee102c0fabe128d0e775b1e260f9a9adf9b90e6ca601a65d0a19c9860c7d39"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T08:56:15.816909Z","signature_b64":"6v9c/aROev5bIjyO7UH/YAKZqL6yjHIYZHg9qUWBsvITeHN3R/NBitW1o39j8d3lAHwvo7oCpz3As9jzlNh3CA==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"7e04392a5bd8acf35df0cce9d3445f96eeee1e2aaeeeb630a6bb86c4eb5037cc","last_reissued_at":"2026-07-05T08:56:15.816481Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T08:56:15.816481Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"","cross_cats":[],"primary_cat":"cs.LG","authors_text":"Adam Fisch, Ahmad Beirami, Alekh Agarwal, Alex D'Amour, Chirag Nagpal, Deepak Ramachandran, Dj Dvijotham, Jacob Eisenstein, Jonathan Berant, Katherine Heller, Peter Shaw, Stephen Pfohl","submitted_at":"2023-12-14T18:59:04Z","abstract_excerpt":"Reward models play a key role in aligning language model applications towards human preferences. However, this setup creates an incentive for the language model to exploit errors in the reward model to achieve high estimated reward, a phenomenon often termed \\emph{reward hacking}. A natural mitigation is to train an ensemble of reward models, aggregating over model outputs to obtain a more robust reward estimate. We explore the application of reward ensembles to alignment at both training time (through reinforcement learning) and inference time (through reranking). First, we show that reward m"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2312.09244","kind":"arxiv","version":3},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2312.09244/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2312.09244","created_at":"2026-07-05T08:56:15.816536+00:00"},{"alias_kind":"arxiv_version","alias_value":"2312.09244v3","created_at":"2026-07-05T08:56:15.816536+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2312.09244","created_at":"2026-07-05T08:56:15.816536+00:00"},{"alias_kind":"pith_short_12","alias_value":"PYCDSKS33CWP","created_at":"2026-07-05T08:56:15.816536+00:00"},{"alias_kind":"pith_short_16","alias_value":"PYCDSKS33CWPGXPQ","created_at":"2026-07-05T08:56:15.816536+00:00"},{"alias_kind":"pith_short_8","alias_value":"PYCDSKS3","created_at":"2026-07-05T08:56:15.816536+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":23,"internal_anchor_count":0,"sample":[{"citing_arxiv_id":"2606.19818","citing_title":"Uncertainty-Aware Reward Modeling for Stable RLHF","ref_index":7,"is_internal_anchor":false},{"citing_arxiv_id":"2606.09711","citing_title":"Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization","ref_index":87,"is_internal_anchor":false},{"citing_arxiv_id":"2606.09073","citing_title":"A Unifying Lens on Reward Uncertainty in RLHF","ref_index":5,"is_internal_anchor":false},{"citing_arxiv_id":"2606.03131","citing_title":"HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Models","ref_index":24,"is_internal_anchor":false},{"citing_arxiv_id":"2606.29805","citing_title":"Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation","ref_index":12,"is_internal_anchor":false},{"citing_arxiv_id":"2605.21602","citing_title":"Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs","ref_index":12,"is_internal_anchor":false},{"citing_arxiv_id":"2605.21351","citing_title":"The Human-AI Delegation-Verification Dilemma: Individual Strategies, Collective Equilibria and Sociotechnical Lock-in","ref_index":17,"is_internal_anchor":false},{"citing_arxiv_id":"2606.29805","citing_title":"Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation","ref_index":12,"is_internal_anchor":false},{"citing_arxiv_id":"2605.21602","citing_title":"Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs","ref_index":11,"is_internal_anchor":false},{"citing_arxiv_id":"2605.21602","citing_title":"Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs","ref_index":71,"is_internal_anchor":false},{"citing_arxiv_id":"2507.15698","citing_title":"CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoning","ref_index":4,"is_internal_anchor":false},{"citing_arxiv_id":"2509.03403","citing_title":"Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training","ref_index":9,"is_internal_anchor":false},{"citing_arxiv_id":"2601.21350","citing_title":"Factored Causal Representation Learning for Robust Reward Modeling in RLHF","ref_index":10,"is_internal_anchor":false},{"citing_arxiv_id":"2605.21351","citing_title":"The Human-AI Delegation-Verification Dilemma: Individual Strategies, Collective Equilibria and Sociotechnical Lock-in","ref_index":17,"is_internal_anchor":false},{"citing_arxiv_id":"2605.15855","citing_title":"Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?","ref_index":6,"is_internal_anchor":false},{"citing_arxiv_id":"2507.06419","citing_title":"Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modeling","ref_index":6,"is_internal_anchor":false},{"citing_arxiv_id":"2604.02686","citing_title":"Beyond Semantic Manipulation: Token-Space Attacks on Reward Models","ref_index":4,"is_internal_anchor":false},{"citing_arxiv_id":"2605.04542","citing_title":"Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation","ref_index":126,"is_internal_anchor":false},{"citing_arxiv_id":"2604.19193","citing_title":"How Far Are Video Models from True Multimodal Reasoning?","ref_index":13,"is_internal_anchor":false},{"citing_arxiv_id":"2604.18547","citing_title":"FUSE: Ensembling Verifiers with Zero Labeled Data","ref_index":28,"is_internal_anchor":false},{"citing_arxiv_id":"2605.06987","citing_title":"Response Time Enhances Alignment with Heterogeneous Preferences","ref_index":16,"is_internal_anchor":false},{"citing_arxiv_id":"2605.00754","citing_title":"Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring","ref_index":5,"is_internal_anchor":false},{"citing_arxiv_id":"2605.00754","citing_title":"Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring","ref_index":5,"is_internal_anchor":false}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/PYCDSKS33CWPGXPQZTU5GRC7S3","json":"https://pith.science/pith/PYCDSKS33CWPGXPQZTU5GRC7S3.json","graph_json":"https://pith.science/api/pith-number/PYCDSKS33CWPGXPQZTU5GRC7S3/graph.json","events_json":"https://pith.science/api/pith-number/PYCDSKS33CWPGXPQZTU5GRC7S3/events.json","paper":"https://pith.science/paper/PYCDSKS3"},"agent_actions":{"view_html":"https://pith.science/pith/PYCDSKS33CWPGXPQZTU5GRC7S3","download_json":"https://pith.science/pith/PYCDSKS33CWPGXPQZTU5GRC7S3.json","view_paper":"https://pith.science/paper/PYCDSKS3","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2312.09244&json=true","fetch_graph":"https://pith.science/api/pith-number/PYCDSKS33CWPGXPQZTU5GRC7S3/graph.json","fetch_events":"https://pith.science/api/pith-number/PYCDSKS33CWPGXPQZTU5GRC7S3/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/PYCDSKS33CWPGXPQZTU5GRC7S3/action/timestamp_anchor","attest_storage":"https://pith.science/pith/PYCDSKS33CWPGXPQZTU5GRC7S3/action/storage_attestation","attest_author":"https://pith.science/pith/PYCDSKS33CWPGXPQZTU5GRC7S3/action/author_attestation","sign_citation":"https://pith.science/pith/PYCDSKS33CWPGXPQZTU5GRC7S3/action/citation_signature","submit_replication":"https://pith.science/pith/PYCDSKS33CWPGXPQZTU5GRC7S3/action/replication_record"}},"created_at":"2026-07-05T08:56:15.816536+00:00","updated_at":"2026-07-05T08:56:15.816536+00:00"}