{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2026:ZCVZIOXALTXE3BLDU4NWNOWOJ4","short_pith_number":"pith:ZCVZIOXA","schema_version":"1.0","canonical_sha256":"c8ab943ae05cee4d8563a71b66bace4f214166effee64669864687fd34d7d3c3","source":{"kind":"arxiv","id":"2605.13639","version":1},"attestation_state":"computed","paper":{"title":"Achieving $\\epsilon^{-2}$ Sample Complexity for Single-Loop Actor-Critic under Minimal Assumptions","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Single-loop off-policy actor-critic reaches an ε-optimal policy with Õ(ε^{-2}) samples under only irreducibility of one policy.","cross_cats":["math.OC","stat.ML"],"primary_cat":"cs.LG","authors_text":"Ishaq Hamza, Zaiwei Chen","submitted_at":"2026-05-13T15:04:59Z","abstract_excerpt":"In this paper, we establish last-iterate convergence rates for off-policy actor--critic methods in reinforcement learning. In particular, under a single-loop, single-timescale implementation and a broad class of policy updates, including approximate policy iteration and natural policy gradient methods, we prove the first $\\tilde{\\mathcal{O}}(\\epsilon^{-2})$ sample complexity guarantee for finding an $\\epsilon$-optimal policy under minimal assumptions, namely, the existence of a policy that induces an irreducible Markov chain. This stands in stark contrast to the existing literature, where an $"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":false},"canonical_record":{"source":{"id":"2605.13639","kind":"arxiv","version":1},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.LG","submitted_at":"2026-05-13T15:04:59Z","cross_cats_sorted":["math.OC","stat.ML"],"title_canon_sha256":"03845fc45ceec497163534ed0e97652299513e483cff184f975b34cbf0f568a5","abstract_canon_sha256":"830565cf23fce4613fb81e2d6bcf04b77a035b7089f1d5861f7b20360b186c66"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-18T02:44:17.625801Z","signature_b64":"LMxOfnSR2r+kigztl53Vrd5v4v4w5c2iYdYU17UeQDbfacpE+KGiq/RHOUXlG2QZ11UYwy+jbyZ3rhmj97GWBQ==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"c8ab943ae05cee4d8563a71b66bace4f214166effee64669864687fd34d7d3c3","last_reissued_at":"2026-05-18T02:44:17.625374Z","signature_status":"signed_v1","first_computed_at":"2026-05-18T02:44:17.625374Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Achieving $\\epsilon^{-2}$ Sample Complexity for Single-Loop Actor-Critic under Minimal Assumptions","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Single-loop off-policy actor-critic reaches an ε-optimal policy with Õ(ε^{-2}) samples under only irreducibility of one policy.","cross_cats":["math.OC","stat.ML"],"primary_cat":"cs.LG","authors_text":"Ishaq Hamza, Zaiwei Chen","submitted_at":"2026-05-13T15:04:59Z","abstract_excerpt":"In this paper, we establish last-iterate convergence rates for off-policy actor--critic methods in reinforcement learning. In particular, under a single-loop, single-timescale implementation and a broad class of policy updates, including approximate policy iteration and natural policy gradient methods, we prove the first $\\tilde{\\mathcal{O}}(\\epsilon^{-2})$ sample complexity guarantee for finding an $\\epsilon$-optimal policy under minimal assumptions, namely, the existence of a policy that induces an irreducible Markov chain. This stands in stark contrast to the existing literature, where an $"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"we prove the first Õ(ε^{-2}) sample complexity guarantee for finding an ε-optimal policy under minimal assumptions, namely, the existence of a policy that induces an irreducible Markov chain.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"the existence of a policy that induces an irreducible Markov chain; the coupled Lyapunov drift framework and cross-domination property hold for the single-loop off-policy updates with unbounded iterates.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Single-loop actor-critic achieves the first Õ(ε^{-2}) sample complexity for ε-optimal policies under minimal irreducibility assumptions.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Single-loop off-policy actor-critic reaches an ε-optimal policy with Õ(ε^{-2}) samples under only irreducibility of one policy.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"2764f336ed3c3a0b4e7ff72df631b4b0848a072dc16277b58f4dc2b03c6a1332"},"source":{"id":"2605.13639","kind":"arxiv","version":1},"verdict":{"id":"c0707b4f-6d5f-4e58-92c0-e19ce0fe564c","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-14T19:24:50.031010Z","strongest_claim":"we prove the first Õ(ε^{-2}) sample complexity guarantee for finding an ε-optimal policy under minimal assumptions, namely, the existence of a policy that induces an irreducible Markov chain.","one_line_summary":"Single-loop actor-critic achieves the first Õ(ε^{-2}) sample complexity for ε-optimal policies under minimal irreducibility assumptions.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"the existence of a policy that induces an irreducible Markov chain; the coupled Lyapunov drift framework and cross-domination property hold for the single-loop off-policy updates with unbounded iterates.","pith_extraction_headline":"Single-loop off-policy actor-critic reaches an ε-optimal policy with Õ(ε^{-2}) samples under only irreducibility of one policy."},"references":{"count":69,"sample":[{"doi":"","year":2021,"title":"Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G. (2021). On the theory of policy gra- dient methods: Optimality, approximation, and distribution shift.Journal of Machine Learning Research, 22(9","work_id":"7944bdec-08fb-439b-b740-79e2cdc97ec4","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2022,"title":"Alacaoglu, A., Viano, L., He, N., and Cevher, V. (2022). A natural actor-critic framework for zero-sum Markov games. InInternational Conference on Machine Learning, pages 307–366. PMLR","work_id":"6178018b-2f46-42b6-bd4b-65915ad3a72e","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":1922,"title":"Banach, S. (1922). Sur les op´ erations dans les ensembles abstraits et leur application aux ´ equations int´ egrales.Fund. Math, 3(1):133–181","work_id":"d0013531-a137-453f-a361-27c18c3e4832","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2011,"title":"Bertsekas, D. P. (2011). Approximate policy iteration: A survey and some new methods. Journal of Control Theory and Applications, 9(3):310–335","work_id":"945f2728-8995-4602-802d-7e28f5155c45","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2018,"title":"Bhandari, J., Russo, D., and Singal, R. (2018). A finite-time analysis of temporal difference learning with linear function approximation. InConference On Learning Theory, pages 1691– 1692","work_id":"7c3c5f71-b5c6-4b83-8aa0-5e1c86e122cc","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":69,"snapshot_sha256":"04e9775dddc6aa50f92f8fe99f3feb5bbadf9052abd3fc8c100cdb3221c1efa9","internal_anchors":2},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2605.13639","created_at":"2026-05-18T02:44:17.625445+00:00"},{"alias_kind":"arxiv_version","alias_value":"2605.13639v1","created_at":"2026-05-18T02:44:17.625445+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2605.13639","created_at":"2026-05-18T02:44:17.625445+00:00"},{"alias_kind":"pith_short_12","alias_value":"ZCVZIOXALTXE","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_16","alias_value":"ZCVZIOXALTXE3BLD","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_8","alias_value":"ZCVZIOXA","created_at":"2026-05-18T12:33:37.589309+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":0,"internal_anchor_count":0,"sample":[]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/ZCVZIOXALTXE3BLDU4NWNOWOJ4","json":"https://pith.science/pith/ZCVZIOXALTXE3BLDU4NWNOWOJ4.json","graph_json":"https://pith.science/api/pith-number/ZCVZIOXALTXE3BLDU4NWNOWOJ4/graph.json","events_json":"https://pith.science/api/pith-number/ZCVZIOXALTXE3BLDU4NWNOWOJ4/events.json","paper":"https://pith.science/paper/ZCVZIOXA"},"agent_actions":{"view_html":"https://pith.science/pith/ZCVZIOXALTXE3BLDU4NWNOWOJ4","download_json":"https://pith.science/pith/ZCVZIOXALTXE3BLDU4NWNOWOJ4.json","view_paper":"https://pith.science/paper/ZCVZIOXA","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2605.13639&json=true","fetch_graph":"https://pith.science/api/pith-number/ZCVZIOXALTXE3BLDU4NWNOWOJ4/graph.json","fetch_events":"https://pith.science/api/pith-number/ZCVZIOXALTXE3BLDU4NWNOWOJ4/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/ZCVZIOXALTXE3BLDU4NWNOWOJ4/action/timestamp_anchor","attest_storage":"https://pith.science/pith/ZCVZIOXALTXE3BLDU4NWNOWOJ4/action/storage_attestation","attest_author":"https://pith.science/pith/ZCVZIOXALTXE3BLDU4NWNOWOJ4/action/author_attestation","sign_citation":"https://pith.science/pith/ZCVZIOXALTXE3BLDU4NWNOWOJ4/action/citation_signature","submit_replication":"https://pith.science/pith/ZCVZIOXALTXE3BLDU4NWNOWOJ4/action/replication_record"}},"created_at":"2026-05-18T02:44:17.625445+00:00","updated_at":"2026-05-18T02:44:17.625445+00:00"}