{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2023:ZLAFCHZ7JEKPUPNKOIHG5PMCOO","short_pith_number":"pith:ZLAFCHZ7","schema_version":"1.0","canonical_sha256":"cac0511f3f4914fa3daa720e6ebd82739efb4f737d6a4d6a9a66c91779b83191","source":{"kind":"arxiv","id":"2305.16381","version":3},"attestation_state":"computed","paper":{"title":"DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"","cross_cats":["cs.CV"],"primary_cat":"cs.LG","authors_text":"Craig Boutilier, Hao Liu, Kangwook Lee, Kimin Lee, Mohammad Ghavamzadeh, Moonkyung Ryu, Olivia Watkins, Pieter Abbeel, Ying Fan, Yuqing Du","submitted_at":"2023-05-25T17:35:38Z","abstract_excerpt":"Learning from human feedback has been shown to improve text-to-image models. These techniques first learn a reward function that captures what humans care about in the task and then improve the models based on the learned reward function. Even though relatively simple approaches (e.g., rejection sampling based on reward scores) have been investigated, fine-tuning text-to-image models with the reward function remains challenging. In this work, we propose using online reinforcement learning (RL) to fine-tune text-to-image models. We focus on diffusion models, defining the fine-tuning task as an "},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2305.16381","kind":"arxiv","version":3},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.LG","submitted_at":"2023-05-25T17:35:38Z","cross_cats_sorted":["cs.CV"],"title_canon_sha256":"e12813477ebfb7df2be616f1fdc3665e0c513aaa38bf104eaefe1648c294399c","abstract_canon_sha256":"cde2a1e416185a89253c16ac4968342c6bff53f878fb1fe12e0b1612b23f10d0"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T07:07:37.910573Z","signature_b64":"hqaC3am/TJZA7lwOrm1OBifm+EuDaSW95oaqKMxIfWSvIB5fkzp2mkGZF6ST3+GEq0a7zNJfQ/AjMaNbF9SmBw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"cac0511f3f4914fa3daa720e6ebd82739efb4f737d6a4d6a9a66c91779b83191","last_reissued_at":"2026-07-05T07:07:37.910070Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T07:07:37.910070Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"","cross_cats":["cs.CV"],"primary_cat":"cs.LG","authors_text":"Craig Boutilier, Hao Liu, Kangwook Lee, Kimin Lee, Mohammad Ghavamzadeh, Moonkyung Ryu, Olivia Watkins, Pieter Abbeel, Ying Fan, Yuqing Du","submitted_at":"2023-05-25T17:35:38Z","abstract_excerpt":"Learning from human feedback has been shown to improve text-to-image models. These techniques first learn a reward function that captures what humans care about in the task and then improve the models based on the learned reward function. Even though relatively simple approaches (e.g., rejection sampling based on reward scores) have been investigated, fine-tuning text-to-image models with the reward function remains challenging. In this work, we propose using online reinforcement learning (RL) to fine-tune text-to-image models. We focus on diffusion models, defining the fine-tuning task as an "},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2305.16381","kind":"arxiv","version":3},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2305.16381/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2305.16381","created_at":"2026-07-05T07:07:37.910131+00:00"},{"alias_kind":"arxiv_version","alias_value":"2305.16381v3","created_at":"2026-07-05T07:07:37.910131+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2305.16381","created_at":"2026-07-05T07:07:37.910131+00:00"},{"alias_kind":"pith_short_12","alias_value":"ZLAFCHZ7JEKP","created_at":"2026-07-05T07:07:37.910131+00:00"},{"alias_kind":"pith_short_16","alias_value":"ZLAFCHZ7JEKPUPNK","created_at":"2026-07-05T07:07:37.910131+00:00"},{"alias_kind":"pith_short_8","alias_value":"ZLAFCHZ7","created_at":"2026-07-05T07:07:37.910131+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":9,"internal_anchor_count":1,"sample":[{"citing_arxiv_id":"2607.07693","citing_title":"Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF","ref_index":8,"is_internal_anchor":true},{"citing_arxiv_id":"2606.22394","citing_title":"Curvature-Adaptive Consistency Flow Matching: Autonomous Trajectory Optimization via Reinforcement Learning","ref_index":10,"is_internal_anchor":false},{"citing_arxiv_id":"2604.27147","citing_title":"How to Guide Your Flow: Few-Step Alignment via Flow Map Reward Guidance","ref_index":54,"is_internal_anchor":false},{"citing_arxiv_id":"2606.28578","citing_title":"Surrogate-Gated Generation and Foundation-Model Embeddings for Bayesian Materials Design","ref_index":9,"is_internal_anchor":false},{"citing_arxiv_id":"2604.27147","citing_title":"How to Guide Your Flow: Few-Step Alignment via Flow Map Reward Guidance","ref_index":55,"is_internal_anchor":false},{"citing_arxiv_id":"2408.07199","citing_title":"Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents","ref_index":241,"is_internal_anchor":false},{"citing_arxiv_id":"2605.16951","citing_title":"Edit-GRPO: A Locality-Preserving Policy Optimization Framework for Image Editing","ref_index":9,"is_internal_anchor":false},{"citing_arxiv_id":"2604.27147","citing_title":"How to Guide Your Flow: Few-Step Alignment via Flow Map Reward Guidance","ref_index":55,"is_internal_anchor":false},{"citing_arxiv_id":"2305.13301","citing_title":"Training Diffusion Models with Reinforcement Learning","ref_index":6,"is_internal_anchor":false}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/ZLAFCHZ7JEKPUPNKOIHG5PMCOO","json":"https://pith.science/pith/ZLAFCHZ7JEKPUPNKOIHG5PMCOO.json","graph_json":"https://pith.science/api/pith-number/ZLAFCHZ7JEKPUPNKOIHG5PMCOO/graph.json","events_json":"https://pith.science/api/pith-number/ZLAFCHZ7JEKPUPNKOIHG5PMCOO/events.json","paper":"https://pith.science/paper/ZLAFCHZ7"},"agent_actions":{"view_html":"https://pith.science/pith/ZLAFCHZ7JEKPUPNKOIHG5PMCOO","download_json":"https://pith.science/pith/ZLAFCHZ7JEKPUPNKOIHG5PMCOO.json","view_paper":"https://pith.science/paper/ZLAFCHZ7","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2305.16381&json=true","fetch_graph":"https://pith.science/api/pith-number/ZLAFCHZ7JEKPUPNKOIHG5PMCOO/graph.json","fetch_events":"https://pith.science/api/pith-number/ZLAFCHZ7JEKPUPNKOIHG5PMCOO/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/ZLAFCHZ7JEKPUPNKOIHG5PMCOO/action/timestamp_anchor","attest_storage":"https://pith.science/pith/ZLAFCHZ7JEKPUPNKOIHG5PMCOO/action/storage_attestation","attest_author":"https://pith.science/pith/ZLAFCHZ7JEKPUPNKOIHG5PMCOO/action/author_attestation","sign_citation":"https://pith.science/pith/ZLAFCHZ7JEKPUPNKOIHG5PMCOO/action/citation_signature","submit_replication":"https://pith.science/pith/ZLAFCHZ7JEKPUPNKOIHG5PMCOO/action/replication_record"}},"created_at":"2026-07-05T07:07:37.910131+00:00","updated_at":"2026-07-05T07:07:37.910131+00:00"}