{"paper":{"title":"Achieving $\\epsilon^{-2}$ Sample Complexity for Single-Loop Actor-Critic under Minimal Assumptions","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Single-loop off-policy actor-critic reaches an ε-optimal policy with Õ(ε^{-2}) samples under only irreducibility of one policy.","cross_cats":["math.OC","stat.ML"],"primary_cat":"cs.LG","authors_text":"Ishaq Hamza, Zaiwei Chen","submitted_at":"2026-05-13T15:04:59Z","abstract_excerpt":"In this paper, we establish last-iterate convergence rates for off-policy actor--critic methods in reinforcement learning. In particular, under a single-loop, single-timescale implementation and a broad class of policy updates, including approximate policy iteration and natural policy gradient methods, we prove the first $\\tilde{\\mathcal{O}}(\\epsilon^{-2})$ sample complexity guarantee for finding an $\\epsilon$-optimal policy under minimal assumptions, namely, the existence of a policy that induces an irreducible Markov chain. This stands in stark contrast to the existing literature, where an $"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"we prove the first Õ(ε^{-2}) sample complexity guarantee for finding an ε-optimal policy under minimal assumptions, namely, the existence of a policy that induces an irreducible Markov chain.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"the existence of a policy that induces an irreducible Markov chain; the coupled Lyapunov drift framework and cross-domination property hold for the single-loop off-policy updates with unbounded iterates.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Single-loop actor-critic achieves the first Õ(ε^{-2}) sample complexity for ε-optimal policies under minimal irreducibility assumptions.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Single-loop off-policy actor-critic reaches an ε-optimal policy with Õ(ε^{-2}) samples under only irreducibility of one policy.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"2764f336ed3c3a0b4e7ff72df631b4b0848a072dc16277b58f4dc2b03c6a1332"},"source":{"id":"2605.13639","kind":"arxiv","version":1},"verdict":{"id":"c0707b4f-6d5f-4e58-92c0-e19ce0fe564c","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-14T19:24:50.031010Z","strongest_claim":"we prove the first Õ(ε^{-2}) sample complexity guarantee for finding an ε-optimal policy under minimal assumptions, namely, the existence of a policy that induces an irreducible Markov chain.","one_line_summary":"Single-loop actor-critic achieves the first Õ(ε^{-2}) sample complexity for ε-optimal policies under minimal irreducibility assumptions.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"the existence of a policy that induces an irreducible Markov chain; the coupled Lyapunov drift framework and cross-domination property hold for the single-loop off-policy updates with unbounded iterates.","pith_extraction_headline":"Single-loop off-policy actor-critic reaches an ε-optimal policy with Õ(ε^{-2}) samples under only irreducibility of one policy."},"references":{"count":69,"sample":[{"doi":"","year":2021,"title":"Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G. (2021). On the theory of policy gra- dient methods: Optimality, approximation, and distribution shift.Journal of Machine Learning Research, 22(9","work_id":"7944bdec-08fb-439b-b740-79e2cdc97ec4","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2022,"title":"Alacaoglu, A., Viano, L., He, N., and Cevher, V. (2022). A natural actor-critic framework for zero-sum Markov games. InInternational Conference on Machine Learning, pages 307–366. PMLR","work_id":"6178018b-2f46-42b6-bd4b-65915ad3a72e","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":1922,"title":"Banach, S. (1922). Sur les op´ erations dans les ensembles abstraits et leur application aux ´ equations int´ egrales.Fund. Math, 3(1):133–181","work_id":"d0013531-a137-453f-a361-27c18c3e4832","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2011,"title":"Bertsekas, D. P. (2011). Approximate policy iteration: A survey and some new methods. Journal of Control Theory and Applications, 9(3):310–335","work_id":"945f2728-8995-4602-802d-7e28f5155c45","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2018,"title":"Bhandari, J., Russo, D., and Singal, R. (2018). A finite-time analysis of temporal difference learning with linear function approximation. InConference On Learning Theory, pages 1691– 1692","work_id":"7c3c5f71-b5c6-4b83-8aa0-5e1c86e122cc","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":69,"snapshot_sha256":"04e9775dddc6aa50f92f8fe99f3feb5bbadf9052abd3fc8c100cdb3221c1efa9","internal_anchors":2},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}