{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2024:JNAF7BYZKHOGMAWZXKXPPUSCK7","short_pith_number":"pith:JNAF7BYZ","schema_version":"1.0","canonical_sha256":"4b405f871951dc6602d9baaef7d24257ceca40379f5c447150ef9720f80fbcb9","source":{"kind":"arxiv","id":"2406.19185","version":2},"attestation_state":"computed","paper":{"title":"Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion","license":"http://creativecommons.org/licenses/by-nc-sa/4.0/","headline":"","cross_cats":[],"primary_cat":"cs.LG","authors_text":"Arash Ahmadian, Bill Wu, Chris Cremer, Eugene Choi, Florian Strub, Matthieu Geist, Mohammad Gheshlaghi Azar, Nathan Grinsztajn, Olivier Pietquin, Yannis Flet-Berliac, Yash Chandak","submitted_at":"2024-06-27T14:03:49Z","abstract_excerpt":"Reinforcement Learning (RL) has been used to finetune Large Language Models (LLMs) using a reward model trained from preference data, to better align with human judgment. The recently introduced direct alignment methods, which are often simpler, more stable, and computationally lighter, can more directly achieve this. However, these approaches cannot optimize arbitrary rewards, and the preference-based ones are not the only rewards of interest for LLMs (eg., unit tests for code generation or textual entailment for summarization, among others). RL-finetuning is usually done with a variation of "},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2406.19185","kind":"arxiv","version":2},"metadata":{"license":"http://creativecommons.org/licenses/by-nc-sa/4.0/","primary_cat":"cs.LG","submitted_at":"2024-06-27T14:03:49Z","cross_cats_sorted":[],"title_canon_sha256":"aa4458f92dd2b95ef7b850737547ccb94a6f7fb68ac94227fadffb4145eebb72","abstract_canon_sha256":"37e0239a9ae4a6944c563f6f5d1191cf597710a08ee2e40d9233c09bccef5441"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T10:01:34.387768Z","signature_b64":"KadW4DJZ+LNA9Jtq98hiMcatbEFhG3o0nY/EhUgFwctq8Vv3NDWSuCj/CJ9DUbww4SErc++QgtAXhmmhHhC6Bg==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"4b405f871951dc6602d9baaef7d24257ceca40379f5c447150ef9720f80fbcb9","last_reissued_at":"2026-07-05T10:01:34.387277Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T10:01:34.387277Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion","license":"http://creativecommons.org/licenses/by-nc-sa/4.0/","headline":"","cross_cats":[],"primary_cat":"cs.LG","authors_text":"Arash Ahmadian, Bill Wu, Chris Cremer, Eugene Choi, Florian Strub, Matthieu Geist, Mohammad Gheshlaghi Azar, Nathan Grinsztajn, Olivier Pietquin, Yannis Flet-Berliac, Yash Chandak","submitted_at":"2024-06-27T14:03:49Z","abstract_excerpt":"Reinforcement Learning (RL) has been used to finetune Large Language Models (LLMs) using a reward model trained from preference data, to better align with human judgment. The recently introduced direct alignment methods, which are often simpler, more stable, and computationally lighter, can more directly achieve this. However, these approaches cannot optimize arbitrary rewards, and the preference-based ones are not the only rewards of interest for LLMs (eg., unit tests for code generation or textual entailment for summarization, among others). RL-finetuning is usually done with a variation of "},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2406.19185","kind":"arxiv","version":2},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2406.19185/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2406.19185","created_at":"2026-07-05T10:01:34.387339+00:00"},{"alias_kind":"arxiv_version","alias_value":"2406.19185v2","created_at":"2026-07-05T10:01:34.387339+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2406.19185","created_at":"2026-07-05T10:01:34.387339+00:00"},{"alias_kind":"pith_short_12","alias_value":"JNAF7BYZKHOG","created_at":"2026-07-05T10:01:34.387339+00:00"},{"alias_kind":"pith_short_16","alias_value":"JNAF7BYZKHOGMAWZ","created_at":"2026-07-05T10:01:34.387339+00:00"},{"alias_kind":"pith_short_8","alias_value":"JNAF7BYZ","created_at":"2026-07-05T10:01:34.387339+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":0,"internal_anchor_count":0,"sample":[]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/JNAF7BYZKHOGMAWZXKXPPUSCK7","json":"https://pith.science/pith/JNAF7BYZKHOGMAWZXKXPPUSCK7.json","graph_json":"https://pith.science/api/pith-number/JNAF7BYZKHOGMAWZXKXPPUSCK7/graph.json","events_json":"https://pith.science/api/pith-number/JNAF7BYZKHOGMAWZXKXPPUSCK7/events.json","paper":"https://pith.science/paper/JNAF7BYZ"},"agent_actions":{"view_html":"https://pith.science/pith/JNAF7BYZKHOGMAWZXKXPPUSCK7","download_json":"https://pith.science/pith/JNAF7BYZKHOGMAWZXKXPPUSCK7.json","view_paper":"https://pith.science/paper/JNAF7BYZ","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2406.19185&json=true","fetch_graph":"https://pith.science/api/pith-number/JNAF7BYZKHOGMAWZXKXPPUSCK7/graph.json","fetch_events":"https://pith.science/api/pith-number/JNAF7BYZKHOGMAWZXKXPPUSCK7/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/JNAF7BYZKHOGMAWZXKXPPUSCK7/action/timestamp_anchor","attest_storage":"https://pith.science/pith/JNAF7BYZKHOGMAWZXKXPPUSCK7/action/storage_attestation","attest_author":"https://pith.science/pith/JNAF7BYZKHOGMAWZXKXPPUSCK7/action/author_attestation","sign_citation":"https://pith.science/pith/JNAF7BYZKHOGMAWZXKXPPUSCK7/action/citation_signature","submit_replication":"https://pith.science/pith/JNAF7BYZKHOGMAWZXKXPPUSCK7/action/replication_record"}},"created_at":"2026-07-05T10:01:34.387339+00:00","updated_at":"2026-07-05T10:01:34.387339+00:00"}