Pith. sign in

REVIEW 13 cited by

LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2505.24034 v2 pith:Q3D3J6G6 submitted 2025-05-29 cs.LG cs.AI

Bo Wu , Sid Wang , Yunhao Tang , Jia Ding , Eryk Helenowski , Liang Tan , Tengyu Xu , Tushar Gowda
show 6 more authors
Zhengxing Chen Chen Zhu Xiaocheng Tang Yundi Qian Beibei Zhu Rui Hou
This is my paper
classification cs.LGcs.AI
keywords llamarlasynchronousframeworkmodeltrainingdistributedefficiencyefficient
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Reinforcement Learning (RL) has become the most effective post-training approach for improving the capabilities of Large Language Models (LLMs). In practice, because of the high demands on latency and memory, it is particularly challenging to develop an efficient RL framework that reliably manages policy models with hundreds to thousands of billions of parameters. In this paper, we present LlamaRL, a fully distributed, asynchronous RL framework optimized for efficient training of large-scale LLMs with various model sizes (8B, 70B, and 405B parameters) on GPU clusters ranging from a handful to thousands of devices. LlamaRL introduces a streamlined, single-controller architecture built entirely on native PyTorch, enabling modularity, ease of use, and seamless scalability to thousands of GPUs. We also provide a theoretical analysis of LlamaRL's efficiency, including a formal proof that its asynchronous design leads to strict RL speed-up. Empirically during the Llama 3 post-training, by leveraging best practices such as colocated model offloading, asynchronous off-policy training, and distributed direct memory access for weight synchronization, LlamaRL achieves significant efficiency gains -- up to 10.7x speed-up compared to DeepSpeed-Chat-like systems on a 405B-parameter policy model. Furthermore, the efficiency advantage continues to grow with increasing model scale, demonstrating the framework's suitability for future large-scale RL training.

Discussion (0). Sign in to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-training

    cs.DC 2026-06 unverdicted novelty 7.0 of 10

    ForeMoE uses routing foresight from the rollout stage to enable micro-step load balancing in MoE RL post-training via a hierarchical planner and transfer engine, claiming up to 1.45x speedup on 64 GPUs.

  2. HetRL: Efficient Reinforcement Learning for LLMs in Heterogeneous Environments

    cs.DC 2025-12 unverdicted novelty 7.0 of 10

    HetRL delivers up to 9.17x higher throughput for LLM RL training on heterogeneous GPUs by using hybrid and ILP-based schedulers to solve a joint optimization problem over computation and data dependencies.

  3. Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Models

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Sparrow uses a dynamic sparsity schedule keyed to the lower tail of sparse-to-dense actor-policy mismatch to enable stable and faster rollouts in long-context RL for LLMs.

  4. AsyncWebRL: Efficient Multi-Step RL for Visual Web Agents

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    AsyncWebRL reports up to 2.9x training speedup and new SOTA on WebGym OOD split via async overlap plus constant normalizer in GRPO, with largest gains on harder tasks.

  5. Rollout-Level Advantage-Prioritized Experience Replay for GRPO

    cs.LG 2026-06 conditional novelty 6.0 of 10

    Rollout-level advantage-prioritized experience replay for GRPO recycles high-advantage individual rollouts with age eviction and fresh-anchored batches to outperform standard GRPO on math benchmarks, with gains increa...

  6. Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Pilot-Commit estimates per-prompt informativeness via a pilot stage and skips low-variance prompts, matching baseline accuracy with up to 4.0x fewer cumulative rollouts than DAPO on math reasoning tasks.

  7. When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVR

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Dynamic Gradient Gating monitors lm_head gradient norms to safely reuse rollout batches in RLVR, achieving up to 2.93x sample efficiency and 2.14x wall-clock speedup across math, ALFWorld, WebShop, and QA tasks.

  8. DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    DORA's multi-version streaming rollout enables 2-3x higher throughput in asynchronous RL for LLMs while preserving convergence by maintaining policy consistency, data integrity, and bounded staleness.

  9. TensorHub: Scalable and Elastic Weight Transfer for LLM RL Training

    cs.DC 2026-04 unverdicted novelty 6.0 of 10

    TensorHub uses Reference-Oriented Storage to enable scalable weight transfer in LLM RL training by referencing replicated GPU weights, achieving up to 19x reduction in cross-datacenter stall time.

  10. StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training

    cs.DC 2026-01 conditional novelty 6.0 of 10

    StaleFlow enforces per-trajectory staleness bounds and coordinates rollout data routing, achieving 1.42–2.68x higher RL post-training throughput with comparable convergence.

  11. Libra: Efficient Resource Management for Agentic RL Post-Training

    cs.LG 2026-06 unverdicted novelty 4.0 of 10

    Libra optimizes GPU allocation across rollout and training in agentic RL via an elastic hybrid pool and C-MLFQ scheduler based on tool-return causal signals, claiming up to 3.0x throughput and 2.5x faster reward conve...

  12. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

  13. Reinforcement Learning from Human Feedback

    cs.LG 2025-04 unverdicted novelty 2.0 of 10

    The book introduces the origins, mathematical setup, and optimization stages of RLHF including reward modeling, reinforcement learning, rejection sampling, and direct alignment algorithms.

Pith tools