{"work":{"id":"7c3dbac0-074a-462c-b23a-b35090f8bf48","openalex_id":"https://openalex.org/W4415277433","doi":"10.48550/arxiv.2510.13786","arxiv_id":"2510.13786","raw_key":null,"title":"The Art of Scaling Reinforcement Learning Compute for LLMs","authors":null,"authors_text":"Devvrit Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer","year":2025,"venue":"cs.LG","abstract":"Reinforcement learning (RL) has become central to training large language models (LLMs), yet the field lacks predictive scaling methodologies comparable to those established for pre-training. Despite rapidly rising compute budgets, there is no principled understanding of how to evaluate algorithmic improvements for scaling RL compute. We present the first large-scale systematic study, amounting to more than 400,000 GPU-hours, that defines a principled framework for analyzing and predicting RL scaling in LLMs. We fit sigmoidal compute-performance curves for RL training and ablate a wide range of common design choices to analyze their effects on asymptotic performance and compute efficiency. We observe: (1) Not all recipes yield similar asymptotic performance, (2) Details such as loss aggregation, normalization, curriculum, and off-policy algorithm primarily modulate compute efficiency without materially shifting the asymptote, and (3) Stable, scalable recipes follow predictable scaling trajectories, enabling extrapolation from smaller-scale runs. Combining these insights, we propose a best-practice recipe, ScaleRL, and demonstrate its effectiveness by successfully scaling and predicting validation performance on a single RL run scaled up to 100,000 GPU-hours. Our work provides both a scientific framework for analyzing scaling in RL and a practical recipe that brings RL training closer to the predictability long achieved in pre-training.","external_url":"https://arxiv.org/abs/2510.13786","cited_by_count":0,"metadata_source":"pith","metadata_fetched_at":"2026-08-05T02:28:24.338817+00:00","pith_arxiv_id":"2510.13786","created_at":"2026-05-09T06:40:40.678510+00:00","updated_at":"2026-08-05T02:28:24.338817+00:00","title_quality_ok":true,"display_title":"The Art of Scaling Reinforcement Learning Compute for LLMs","render_title":"The Art of Scaling Reinforcement Learning Compute for LLMs"},"hub":{"state":{"work_id":"7c3dbac0-074a-462c-b23a-b35090f8bf48","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":30,"external_cited_by_count":0,"distinct_field_count":5,"first_pith_cited_at":"2025-04-16T21:36:46+00:00","last_pith_cited_at":"2026-07-01T21:39:19+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-23T01:49:28.138019+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":6}],"polarity_counts":[{"context_polarity":"background","n":6}],"runs":{},"summary":{},"graph":{},"authors":[]}}