Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

StaleFlow claims that trajectory-level staleness accounting can replace coarse in-flight limits, letting a disaggregated RL system bound staleness strictly and mitigate data skewness in one design.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 06:19 UTC pith:7U74AWXT

load-bearing objection Serious systems paper with measured throughput gains; the strict-staleness protocol is under-proved and convergence evidence is thin, but both are fixable. the 2 major comments →

arxiv 2601.12784 v2 pith:7U74AWXT submitted 2026-01-19 cs.DC

StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training

Haoyang Li , Sheng Lin , Fangcheng Fu , Yuming Zhou , Xiaodong Ji , Yanfeng Zhao , Lefeng Wang , Jie Jiang
show 1 more author
Bin Cui
This is my paper
classification cs.DC
keywords reinforcement learning post-trainingdata stalenessdata skewnessdisaggregated architecturerollout coordinationstaleness boundtrajectory serverparameter server
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

StaleFlow argues that fully disaggregated RL post-training does not have to choose between controlling data staleness and mitigating data skewness. The paper's central claim is that a trajectory-level version-tagging protocol, built around a virtual staleness buffer, can enforce a user-specified staleness bound strictly while still allowing partial rollout, redundant rollout, multi-version instances, and migration. On top of that protocol, a centralized coordinator with trajectory and parameter servers uses a throughput cost model to route, synchronize, and migrate work so that the system actually benefits from the slack that the staleness bound permits. The payoff, per the evaluations, is up to 2.68x (1.17-2.01x on average) higher throughput than state-of-the-art systems at the same convergence. A sympathetic reader would take this as evidence that the apparent tradeoff is an artifact of coarse-grained control, not an inherent limit.

Core claim

The central discovery is that staleness can be accounted for at the granularity of a single trajectory while the system retains full freedom to coordinate rollout. Each trajectory is tagged with V_traj, the oldest model version it is allowed to be generated under; each training buffer has a version V_buf; and the invariant V_traj + η ≥ V_buf is the only thing that needs to be maintained. The Reserve/Occupy/Consume primitives plus planned entry movement keep this invariant even when trajectories are interrupted, migrated, aborted, or generated across instances with different model versions. This means the standard "limit in-flight data" approach is not the only way to bound staleness, and the

What carries the argument

Virtual staleness buffer: a per-batch accounting structure holding an entry per trajectory group with metadata (trajectory ID and V_traj), where each buffer has a version V_buf and the invariant V_traj + η ≥ V_buf is enforced by Reserve/Occupy/Consume primitives. The buffer's entry deletion and movement rule — delete a completed reserved entry, then move the earliest admissible reserved entries forward — is what allows Occupy to place completed data in the earliest possible buffer and keeps training batches ready without letting stale data slip through. On the skewness side, the middleware trajectory server and parameter server decouple data movement from rollout instances, and a centralized

Load-bearing premise

The load-bearing assumption is that the buffer bookkeeping rule never admits a stale trajectory and never deadlocks under any interleaving of partial rollout, migration, and abort; the paper does not prove this invariant formally.

What would settle it

Instrument the staleness manager with η=0 and search, through random interleavings of Interrupt, Route, and Abort commands, for a state where an occupied entry has V_buf − V_traj > 0, or where Reserve cannot find an empty entry although total capacity is available. Finding either would break the strict-staleness claim. Short of that, re-running the throughput benchmark on the same 128-GPU setup and comparing tokens/second to the reported 1.17-2.01x average would test the measured gains.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Users get a real knob: set the staleness bound η and know that no consumed trajectory can be more than η model versions old, making the convergence/throughput tradeoff explicit and tunable per job.
  • All four rollout coordination techniques work together under one protocol, so skewness mitigation no longer depends on giving up staleness guarantees.
  • Larger η yields proportionally larger throughput gains (up to 2.68x), meaning jobs whose convergence tolerates more staleness can put idle GPUs to work.
  • The speculative-state snapshot validation prevents command races from causing oscillating routing decisions, which is a necessary condition for the coordinator's decisions to be stable.
  • Middleware overhead stays below 3% of step time, so the coordination machinery does not eat the gains it enables.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the buffer invariant is as robust as claimed, the same version-tag accounting could generalize to any asynchronous pipeline where data freshness is a correctness concern, such as federated learning or online data serving, by swapping model versions for data versions.
  • A natural next test is to make the cost-model coefficients adaptive, replacing offline profiling with runtime estimates; the current design depends on coefficients that may change with hardware or batch composition.
  • The convergence evidence is limited to math-reasoning RL with one training algorithm; extending the same staleness bound to general chat or agent RL would test whether the buffer protocol itself, rather than the algorithm's tolerance, is what preserves convergence.
  • The main throughput comparisons exclude redundant rollout; a head-to-head study with redundant rollout enabled could show whether the protocol's accommodation of redundancy translates into an independent, additional speedup.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. StaleFlow is a fully disaggregated RL post-training system that claims to jointly enforce a user-specified trajectory-level staleness bound (η) and mitigate data skewness. The system introduces a global consistency protocol built on a virtual "staleness buffer" with Reserve/Occupy/Consume primitives, plus two middleware data servers (trajectory server and parameter server) and a centralized coordinator that executes routing, synchronization, and migration strategies. The paper's central claims are (i) strict adherence to the global staleness constraint while supporting partial rollout, redundant rollout, multi-version instances, and migration, and (ii) throughput gains of up to 1.42–2.68× over state-of-the-art systems without compromising convergence. The evaluation is performed on a 128-GPU H20 cluster across four Qwen models, with convergence shown over 50–100 training steps.

Significance. If the protocol is correct and reproducible, StaleFlow is a meaningful systems contribution: it targets a real tension in disaggregated RL post-training and proposes a concrete architectural solution. The paper's empirical work is a strength: measured throughput on a 128-GPU cluster, a detailed ablation isolating the routing/synchronization/migration strategies, a cost model with a reported 10.52% average estimation error, and a time breakdown showing command overhead below 3%. The authors also state that source code is available. However, the central differentiator — the strict staleness guarantee — rests on an informally described buffer protocol with no invariant or liveness proof. Because the throughput measurements stand independently of that proof, the paper's usefulness would survive a fix, but the guarantee as stated is not currently established.

major comments (2)
  1. [§4.2, Figure 7 (right)] The strict-staleness guarantee is load-bearing, yet the Reserve/Occupy/movement procedure is only described informally. In particular, step (3) moves a reserved entry C into B's former buffer position without stating that C satisfies V_C + η ≥ V_buf at that newer buffer. Since the invariant is V_traj + η ≥ V_buf for every entry, moving C to a later buffer can violate the constraint if V_C is not large enough. The paper also does not prove that, under all interleavings of Reserve/Occupy/Abort and buffer advancement, Occupy always finds an admissible empty slot and the system never deadlocks (e.g., a Stuck buffer with aborted-but-not-cleaned reservations). I am not claiming the protocol is wrong; I am claiming the proof is missing, and the text as written is at least incomplete. This needs a formal invariant, a liveness argument, or an explicit enumeration of all transition cases.
  2. [§6.3, Figure 14] The claim "without compromising convergence" is supported only by 50–100 training steps on two models, with no repeated seeds or confidence intervals. The paper does show that η=1–3 matches VeRL's reward curve in that short horizon and that η=10 collapses, which is useful evidence. But the abstract and conclusion state the convergence claim as unconditional. Either extend the training horizon, add seed variance, or soften the claim to "on the tested horizon, no degradation was observed." As written, the convergence claim is stronger than the evidence.
minor comments (5)
  1. [Abstract / Conclusion] The throughput numbers are inconsistent: the abstract reports 1.18–1.91× average, while the full text and §6.2 report 1.17–2.01× average. Please reconcile.
  2. [Abstract / Artifact] The source-code link in the abstract points to github.com/psrl-project/psrl, but the paper is named StaleFlow. Please verify that this is the correct repository or provide a StaleFlow-specific link.
  3. [Figure 7 (right)] The four-step transition is hard to follow because the figure labels B and C in a way that does not unambiguously indicate buffer indices and version values. A small example with concrete (V_traj, V_buf) values would improve clarity and would also help the reader verify the invariant.
  4. [§6.1 / Figure 13] The text says red crosses denote NCCL timeouts in baselines, but the figure caption only says "Red crosses denote unsupported configurations." Please move the explanation into the caption or the body so readers understand the meaning of the crosses.
  5. [§2.2 / References] The paper categorizes VeRL-Async as "strict staleness control," citing a GitHub recipe rather than a peer-reviewed publication. This is acceptable, but the reference should be marked clearly as a repository/recipe, and the same applies to other non-archival citations.

Circularity Check

0 steps flagged

No significant circularity: headline results are empirical measurements against external baselines; the unproved buffer protocol is a correctness concern, not a circularity.

full rationale

StaleFlow's central claims rest on empirical measurements against external baselines (VeRL, VeRL-Pipeline, VeRL-Async, AReaL, Roll Flash) and on ablations that vary its own strategies while holding the system fixed; these are not derived from the cost model whose coefficients k1–k5 are regression-fitted. The cost model is used to steer routing decisions, and its accuracy is validated independently against measured throughput (Appendix B.2, Fig. 24, avg error 10.52%), but the headline 1.42–2.68x gains are measured end-to-end, not predicted by the model. The 'global consistency protocol' is a design artifact: the invariant V_traj+eta>=V_buf is the admission rule enforced by Reserve/Occupy; observing that measured trajectories satisfy it (Fig. 18) is a sanity check of the implementation, not a separate derivation. The informal buffer-movement step is a potential correctness gap (no proof that moving entry C preserves the invariant), but an unproved liveness/invariant claim is not circularity. Self-citations (prior PKU/SJTU parallel-training systems) appear only in related-work contexts and are not load-bearing. Hence no prediction or first-principles result reduces to its inputs by construction.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

No new physical entities are introduced. The virtual staleness buffer, trajectory server, and parameter server are software abstractions. The ledger shows that the central empirical claims rest on a fitted cost model and on an informally argued protocol invariant, while the headline throughput/convergence measurements are external and non-circular.

free parameters (6)
  • staleness bound eta = 1, 2, 3, 10 (evaluated)
    User-configurable knob studied in §6.2-6.3. Convergence is preserved for 1-3 and collapses at 10; it is chosen by hand, not fitted to the target result, but it is central to the claimed trade-off.
  • cost model coefficients k1-k4 = Table 4: 7.28e-8, 1.72e-3, 1.25e-4, 1.07e-2 for Qwen3-30B-A3B
    Fitted via offline profiling and linear regression (Appendix B). Used in Eq. 2 for routing and migration decisions; not used to compute the reported measured throughput, but affects coordination quality.
  • per-token KV cache footprint k5 = not stated numerically
    Obtained via profiling (§5.3); used in the marginal-gain model and routing admissibility checks.
  • routing acceptance threshold mu = 0.3
    Hand-chosen threshold in the waterfall model (§6.1); controls how aggressively trajectories are routed to lower-priority instances.
  • migration wait threshold phi_wait = 3
    Hand-chosen threshold for interrupting excess waiting trajectories (§6.1, Algorithm 5).
  • throughput imbalance threshold phi_throughput = 5
    Hand-chosen threshold for triggering migration between imbalanced instances (§6.1, Algorithm 5).
axioms (4)
  • domain assumption RL algorithms tolerate model mismatch up to staleness bound eta via importance sampling or additional corrections
    Invoked in §2.2 with citations [6,7,10,39,68,80,87,89,91]; not proved in this paper. It underpins the claim that bounded staleness preserves convergence.
  • domain assumption The constraint V_traj + eta >= V_buf is sufficient to control convergence
    Core of the global consistency protocol (§4.2). Supported empirically by Figures 3 and 14, but no formal RL convergence proof ties this specific buffer invariant to learning guarantees.
  • domain assumption The cost model T_i(S) accurately estimates instance throughput
    Appendix B reports average estimation error 10.52% (Figure 24). Routing, synchronization, and migration strategies rely on this model to make throughput-oriented decisions.
  • ad hoc to paper The virtual staleness buffer Reserve/Occupy/movement procedure maintains the global bound and does not deadlock
    Asserted informally in §4.2 and Figure 7 without a formal invariant or liveness proof. This is load-bearing for the strict-staleness guarantee and is a key reason for the CONDITIONAL verdict.

pith-pipeline@v1.3.0-alltime-deepseek · 43791 in / 14091 out tokens · 152605 ms · 2026-08-04T06:19:20.257750+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) post-training has become pivotal for enhancing the capabilities of modern large models. A recent trend is to develop RL systems with a fully disaggregated architecture, which decouples the three RL phases (rollout, reward, and training) onto separate resources and executes them asynchronously. However, two critical data-level concerns arise: (1) asynchronous execution leads to data staleness in trajectories (the data generated by rollout) as the model parameters used in rollout may not be up to date, which impairs RL convergence; and (2) the length variation of trajectories introduces severe data skewness, leading to workload imbalance and degraded system performance. Existing systems fail to address these two concerns in a unified manner. Techniques that tightly control data staleness often constrain effective data skewness mitigation, while aggressive data skewness mitigation tends to exacerbate data staleness. As a result, systems are forced to trade off convergence for performance, or vice versa. To address this, we propose StaleFlow, an RL post-training system that jointly tackles data staleness and skewness. First, to control staleness, StaleFlow introduces a global consistency protocol that tracks the full lifecycle of each trajectory and constrains staleness. Second, to mitigate skewness, StaleFlow re-designs the RL system architecture by constructing data servers for trajectories and parameters to achieve flexible rollout coordination. Subsequently, we develop a suite of staleness-aware, throughput-oriented strategies to enhance system performance. Evaluations show that StaleFlow achieves up to 1.42-2.68$\times$ (1.18-1.91$\times$ on average) higher throughput than state-of-the-art systems, without compromising convergence. Our source code is available: https://github.com/psrl-project/psrl.

Figures

Figures reproduced from arXiv: 2601.12784 by Bin Cui, Fangcheng Fu, Haoyang Li, Jie Jiang, Lefeng Wang, Sheng Lin, Xiaodong Ji, Yanfeng Zhao, Yuming Zhou.

Figure 1
Figure 1. Figure 1: (Left) RL workflow in a fully disaggregated architecture. (Right) Time breakdown of post-training a Qwen3-30B-A3B (64 H20 GPUs). Rollout and training overlap and jointly dominate the time. From the data management perspective, there are two kinds of data that move across different phases, namely trajectories and parameters. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of different RL systems. Higher values on the vertical axis indicate stronger support for rollout coordination. where instances update models at their own pace without global synchronization; and (4) Migration across instances [12, 16, 54, 62], which dynamically redistributes trajectories to rebalance workload. Limitations. Despite these advances, current systems fall short in striking a good ba… view at source ↗
Figure 3
Figure 3. Figure 3 [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: A rollout step of Qwen3-30B-A3B (128 H20 GPUs). (Left) Trajectory and token distributions are highly skewed both within each instance and across different instances. (Right) This causes intra-instance underutilization and inter-instance idle waiting. the 𝑉2 training phase to overlap with the 𝑉2 rollout phase. While this improves resource utilization, it introduces stale data, which may impair RL convergenc… view at source ↗
Figure 6
Figure 6. Figure 6: (Left) Overview of StaleFlow. A staleness manager en￾forces strict data staleness control, while a rollout coordinator and two data servers provide flexible, efficient rollout coordination to mitigate data skewness. (Right) Standard data flow in StaleFlow. Trajectories are sourced from the TS (○1 ) and flow through the roll￾out, reward, and training phases (○2 –○3 ). After training a batch, updated model p… view at source ↗
Figure 7
Figure 7. Figure 7: (Left) Staleness buffers enforce the bound 𝜂 using Reserve and Occupy. For illustration, Occupy is drawn immediately after rollout completion; in practice, it occurs after reward computation. In this example, 𝜂 = 1 (i.e., 𝑉𝑡𝑟𝑎 𝑗 + 1 ≥ 𝑉𝑏𝑢 𝑓 ), and each rollout instance supports up to three concurrent trajectories. (Right) Buffer transitions when a Reserve completes and a corresponding Occupy must be applie… view at source ↗
Figure 8
Figure 8. Figure 8: The staleness buffer is designed to be fully compatible with a wide range of advanced rollout coordination techniques. still be successfully claimed, the assignment is permitted. Otherwise, a larger 𝑉𝑡𝑟𝑎𝑗 is needed to unlock newer buffers for reservation, guaranteeing that the staleness bound 𝜂 remains satisfied. Buffer states. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p006_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: (Left) Rollout service architecture. The centralized coordinator periodically captures snapshots and issues commands to coordinate rollout instances, TS, and PS. (Right) Diverse rollout coordination techniques are supported via different commands based on TS and PS. 3 2 1 3 5 4 5’ 6 Pull 5’’ Rollout Coordinator Pull Snapshots Commands 5 + 5’ 7 8 Trajectory Server (TS) … Parameter Server (PS) W2… V1 TBD Int… view at source ↗
Figure 10
Figure 10. Figure 10: Decoupling rollout data movement (trajectories and model parameters) via middleware servers. (Top) TS: Trajectories flow through different rollout instances via the TS as an intermediary. (Bottom) PS: Training workers continuously Push updated model parameters to the PS, while rollout instances Pull them on demand. A read-write locking scheme is used to ensure correctness. (2) As shown in [PITH_FULL_IMAG… view at source ↗
Figure 11
Figure 11. Figure 11: A snapshot captures five fields for every rollout instance. the re-routed rollout instance is no older than the assigned 𝑉𝑡𝑟𝑎𝑗 , as it represents the oldest version tolerated over the entire generation. Parameter server. The PS stores the latest model parameters and, as shown in [PITH_FULL_IMAGE:figures/full_fig_p008_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: (a) To avoid outdated snapshots, we introduce a speculative state (𝑃) to verify whether the expected effect has occurred after issuing the commands. (b) After issuance, commands are executed concurrently. When dependencies exist between commands, we address them by either having commands wait or keeping data pending. “I”, “R” and “P” represent Interrupt, Route and Pull, respectively. (c) Our routing strat… view at source ↗
Figure 13
Figure 13. Figure 13: End-to-end throughput under different staleness bounds (𝜂). Values in parentheses are the gains achieved by StaleFlow compared to the baselines. Red crosses denote unsupported configurations: data skewness causes severe load imbalance, leading to NCCL [20, 49] timeouts. (a) DeepSeek-R1-Distill-Qwen-14B, 64 GPUs (b) DeepSeek-R1-Distill-Qwen-32B, 128 GPUs (c) Qwen2.5-32B, 128 GPUs (d) Qwen3-30B-A3B, 128 GPU… view at source ↗
Figure 14
Figure 14. Figure 14: RL convergence comparing StaleFlow with VeRL. that limit the number of in-flight trajectories, preventing the use of more flexible rollout coordination techniques ( [PITH_FULL_IMAGE:figures/full_fig_p011_14.png] view at source ↗
Figure 16
Figure 16. Figure 16: Ablation study of rollout coordination strategies. “R”, “S”, and “M” denote routing, synchronization, and migration strate￾gies, respectively. “✓” indicates StaleFlow ’s throughput-oriented strategy; “×” indicates the vanilla counterpart. 6.5 Performance Interpretation We further conduct a detailed analysis to identify the sources of throughput gain and provide a detailed breakdown of StaleFlow. The analy… view at source ↗
Figure 18
Figure 18. Figure 18: Trajectory staleness distribution across different stale￾ness buffers. The staleness bound 𝜂 is set to 3. synchronization performs greedy global updates even while many trajectories are still running. In [PITH_FULL_IMAGE:figures/full_fig_p012_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Overhead of model synchronization. other baselines, as shown in [PITH_FULL_IMAGE:figures/full_fig_p013_19.png] view at source ↗
Figure 21
Figure 21. Figure 21: Push is triggered automatically by training workers and can overlap with the next training step, whereas Pull is issued by the rollout coordinator and must block the generation of target instances. W1 ¼W2 ¼W2 2 1 3 5 4 Train Workers 2 4 1 5’ 6 Pull 5 3 6 5’’ Reserve Occupy Batch Weight Weight … Control Data Staleness (§5) Rollout Coordinator Push Pull Trajectory Staleness Manager Consume Snapshots Command… view at source ↗
Figure 23
Figure 23. Figure 23: Communication overhead across varying cluster scales. the communication load across senders, we track for each sender an accumulated estimate of the communication latency its assigned slices would incur. When assigning a sender for a receiver’s required slice, our planner selects the candidate sender with the smallest current accumulated latency, thereby greedily distributing traffic to minimize bottlenec… view at source ↗
Figure 24
Figure 24. Figure 24: Comparison of actual throughput and estimated throughput from our cost model. We present the results for a single instance over the first two post-training steps. 8250 8500 8750 9000 9250 9500 9750 9,744 8,622 9,481 ↓11.5% ↓2.7% Mean Response Length 32000 34000 36000 38000 40000 40,960 34,179 39,748 ↓16.6% ↓3.0% Max Response Length 360 380 400 420 440 460 460 382 422 ↓17.0% ↓8.3% Time per Step (s) 43000 4… view at source ↗
Figure 25
Figure 25. Figure 25: Effect of redundant rollout. We compare StaleFlow with and without redundant rollout at the batch and group levels. Green bars represent performance improvements, while red bars indicate performance degradation. with bf16 precision on an H20 GPU, this threshold is empirically found to be approximately 37. Thus, for 𝑛 > 37, L𝑚𝑎𝑡𝑚𝑢𝑙 ≈ 𝑘3 × 𝑛. Total per-decoding-step latency. Combining the latency of atten￾t… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-training

    cs.DC 2026-06 unverdicted novelty 7.0

    ForeMoE uses routing foresight from the rollout stage to enable micro-step load balancing in MoE RL post-training via a hierarchical planner and transfer engine, claiming up to 1.45x speedup on 64 GPUs.

Reference graph

Works this paper leans on

103 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwa- tra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24),

  2. [2]

    Longalign: A recipe for long context alignment of large language models, 2024

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. Longalign: A recipe for long context alignment of large language models, 2024. https://arxiv.org/abs/2401.18058

    Pith/arXiv arXiv 2024

  3. [3]

    A survey on mixture of experts in large language models.IEEE Transactions on Knowledge and Data Engineering, 2025

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts in large language models.IEEE Transactions on Knowledge and Data Engineering, 2025. http://dx.doi.org/10.1109/TKDE.2025. 3554028

    doi:10.1109/tkde.2025 2025

  4. [4]

    Respec: Towards optimizing speculative decoding in reinforcement learning systems, 2025

    Qiaoling Chen, Zijun Liu, Peng Sun, Shenggui Li, Guoteng Wang, Ziming Liu, Yonggang Wen, Siyuan Feng, and Tianwei Zhang. Respec: Towards optimizing speculative decoding in reinforcement learning systems, 2025. https://arxiv.org/ abs/2510.26475

    arXiv 2025

  5. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI, Daya Guo, Dejian Yang, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. https://arxiv.org/abs/2501. 12948

    2025

  6. [6]

    Ulorl:an ultra-long output reinforcement learning approach for advancing large language models’ reasoning abilities, 2025

    Dong Du, Shulin Liu, Tao Yang, Shaohua Chen, and Yang Li. Ulorl:an ultra-long output reinforcement learning approach for advancing large language models’ reasoning abilities, 2025. https://arxiv.org/abs/2507.19766

    Pith/arXiv arXiv 2025

  7. [7]

    Advances in importance sampling.Wiley Stat- sRef: Statistics Reference Online, 2021

    Víctor Elvira and Luca Martino. Advances in importance sampling.Wiley Stat- sRef: Statistics Reference Online, 2021. http://dx.doi.org/10.1002/9781118445112. stat08284

    doi:10.1002/9781118445112 2021

  8. [8]

    Aime24 dataset, 2025

    Hugging Face. Aime24 dataset, 2025. https://huggingface.co/datasets/math- ai/aime24

    2025

  9. [9]

    Dapo-math-17k dataset, 2025

    Hugging Face. Dapo-math-17k dataset, 2025. https://huggingface.co/datasets/ BytedTsinghua-SIA/DAPO-Math-17k. 13

    2025

  10. [10]

    Areal: A large-scale asynchronous reinforcement learning system for language reasoning, 2025

    Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhang Yuan, and Yi Wu. Areal: A large-scale asynchronous reinforcement learning system for language reasoning, 2025. https://arxiv.org/abs/2505.24298

    Pith/arXiv arXiv 2025

  11. [11]

    Apt-serve: Adaptive request scheduling on hybrid cache for scalable llm inference serving.Proc

    Shihong Gao, Xin Zhang, Yanyan Shen, and Lei Chen. Apt-serve: Adaptive request scheduling on hybrid cache for scalable llm inference serving.Proc. ACM Manag. Data, 2025. https://doi.org/10.1145/3725394

    doi:10.1145/3725394 2025

  12. [12]

    Rollpacker: Mitigating long-tail rollouts for fast, synchronous rl post-training, 2025

    Wei Gao, Yuheng Zhao, Dakai An, Tianyuan Wu, Lunxi Cao, Shaopan Xiong, Ju Huang, Weixun Wang, Siran Yang, Wenbo Su, Jiamang Wang, Lin Qu, Bo Zheng, and Wei Wang. Rollpacker: Mitigating long-tail rollouts for fast, synchronous rl post-training, 2025. https://arxiv.org/abs/2509.21009

    arXiv 2025

  13. [13]

    Enabling parallelism hot switching for efficient training of large language models

    Hao Ge, Fangcheng Fu, Haoyang Li, Xuanyu Wang, Sheng Lin, Yujie Wang, Xiaonan Nie, Hailin Zhang, Xupeng Miao, and Bin Cui. Enabling parallelism hot switching for efficient training of large language models. InProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, 2024. https: //doi.org/10.1145/3694715.3695969

    arXiv 2024

  14. [14]

    Search and score-based waterfall auction optimization

    Dan Halbersberg, Matan Halevi, and Moshe Salhov. Search and score-based waterfall auction optimization. InLearning and Intelligent Optimization: 16th International Conference, LION 16, Milos Island, Greece, June 5–10, 2022, Revised Selected Papers, 2022. https://doi.org/10.1007/978-3-031-24866-5_27

    work page doi:10.1007/978-3-031-24866-5_27 2022

  15. [15]

    Asyncflow: An asynchronous streaming rl framework for efficient llm post-training, 2025

    Zhenyu Han, Ansheng You, Haibo Wang, Kui Luo, Guang Yang, Wenqi Shi, Menglong Chen, Sicheng Zhang, Zeshun Lan, Chunshi Deng, Huazhong Ji, Wenjie Liu, Yu Huang, Yixiang Zhang, Chenyi Pan, Jing Wang, Xin Huang, Chunsheng Li, and Jianping Wu. Asyncflow: An asynchronous streaming rl framework for efficient llm post-training, 2025. https://arxiv.org/abs/2507.01663

    Pith/arXiv arXiv 2025

  16. [16]

    History rhymes: Accelerating llm reinforcement learning with rhymerl, 2025

    Jingkai He, Tianjian Li, Erhu Feng, Dong Du, Qian Liu, Tao Liu, Yubin Xia, and Haibo Chen. History rhymes: Accelerating llm reinforcement learning with rhymerl, 2025. https://arxiv.org/abs/2508.18588

    Pith/arXiv arXiv 2025

  17. [17]

    Verl recipe: Fully async policy trainer, 2025

    https://github.com/meituan search. Verl recipe: Fully async policy trainer, 2025. https://verl.readthedocs.io/en/latest/advance/fully_async.html

    2025

  18. [18]

    Verl recipe: One step off policy async trainer,

    https://github.com/meituan search. Verl recipe: One step off policy async trainer,

  19. [19]

    Openrlhf: An easy-to-use, scalable and high-performance rlhf framework, 2025

    Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Zilin Zhu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, Weikai Fang, Xianyu, Yu Cao, Haotian Xu, and Yiming Liu. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework, 2025. https://arxiv.org/abs/2405.11143

    Pith/arXiv arXiv 2025

  20. [20]

    Demystifying nccl: An in-depth analysis of gpu communication protocols and algorithms, 2025

    Zhiyi Hu, Siyuan Shen, Tommaso Bonato, Sylvain Jeaugey, Cedell Alexander, Eric Spada, James Dinan, Jeff Hammond, and Torsten Hoefler. Demystifying nccl: An in-depth analysis of gpu communication protocols and algorithms, 2025. https://arxiv.org/abs/2507.04786

    arXiv 2025

  21. [21]

    Qerl: Beyond efficiency – quantization-enhanced reinforcement learning for llms, 2025

    Wei Huang, Yi Ge, Shuai Yang, Yicheng Xiao, Huizi Mao, Yujun Lin, Hanrong Ye, Sifei Liu, Ka Chun Cheung, Hongxu Yin, Yao Lu, Xiaojuan Qi, Song Han, and Yukang Chen. Qerl: Beyond efficiency – quantization-enhanced reinforcement learning for llms, 2025. https://arxiv.org/abs/2510.11696

    arXiv 2025

  22. [22]

    Le, and Yonghui Chen

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Zhifeng Chen, Yanping Hu, Maxim Krikun, Quoc V. Le, and Yonghui Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. InAdvances in Neural Information Processing Systems (NeurIPS), 2019. https://proceedings. neurips.cc/paper/2019/file/093f65e080a295f8076b1c...

    2019

  23. [23]

    System optimizations for enabling training of extreme long sequence transformer models

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Reza Yaz- dani Aminadabi, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. System optimizations for enabling training of extreme long sequence transformer models. InProceedings of the 43rd ACM Symposium on Principles of Distributed Computing, 2024. https://doi.org/10.1145/3662158.3662806

    arXiv 2024

  24. [24]

    Dynapipe: Optimizing multi-task training through dynamic pipelines

    Chenyu Jiang, Zhen Jia, Shuai Zheng, Yida Wang, and Chuan Wu. Dynapipe: Optimizing multi-task training through dynamic pipelines. InProceedings of the Nineteenth European Conference on Computer Systems (EuroSys), 2024. https: //doi.org/10.1145/3627703.3629585

    arXiv 2024

  25. [25]

    Dhillon, David Brandfonbrener, and Rishabh Agarwal

    Devvrit Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit S. Dhillon, David Brandfonbrener, and Rishabh Agarwal. The art of scaling reinforcement learning compute for llms, 2025. https://arxiv. org/abs/2510.13786

    Pith/arXiv arXiv 2025

  26. [26]

    Muthukrishnan, Zheng Wen, and Yikun Xian

    Branislav Kveton, Saied Mahdian, S. Muthukrishnan, Zheng Wen, and Yikun Xian. Waterfall bandits: Learning to sell ads online, 2019. https://arxiv.org/abs/ 1904.09404

    Pith/arXiv arXiv 2019

  27. [27]

    Efficient mem- ory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient mem- ory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, 2023. https: //doi.org/10.1145/3600006.3613165

    arXiv 2023

  28. [28]

    Puzzle: efficiently aligning large language models through light-weight context switch

    Kinman Lei, Yuyang Jin, Mingshu Zhai, Kezhao Huang, Haoxing Ye, and Jidong Zhai. Puzzle: efficiently aligning large language models through light-weight context switch. InProceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference, 2024

    2024

  29. [29]

    {GS}hard: Scaling giant models with conditional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. {GS}hard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021. https://openreview. net/forum?id=qrwe7XHTmYb

    2021

  30. [30]

    Fast inference from transform- ers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transform- ers via speculative decoding. InProceedings of the 40th International Conference on Machine Learning, 2023. https://dl.acm.org/doi/10.5555/3618408.3619203

    arXiv 2023

  31. [31]

    Hetu v2: A general and scalable deep learning system with hierarchical and heterogeneous single program multiple data annotations,

    Haoyang Li, Fangcheng Fu, Hao Ge, Sheng Lin, Xuanyu Wang, Jiawen Niu, Xupeng Miao, and Bin Cui. Hetu v2: A general and scalable deep learning system with hierarchical and heterogeneous single program multiple data annotations,

  32. [32]

    Malleus: Straggler-resilient hybrid parallel training of large-scale models via malleable data and model parallelization.Proc

    Haoyang Li, Fangcheng Fu, Hao Ge, Sheng Lin, Xuanyu Wang, Jiawen Niu, Yujie Wang, Hailin Zhang, Xiaonan Nie, and Bin Cui. Malleus: Straggler-resilient hybrid parallel training of large-scale models via malleable data and model parallelization.Proc. ACM Manag. Data, 2025. https://doi.org/10.1145/3725322

    work page doi:10.1145/3725322 2025

  33. [33]

    Hydraulis: Balancing large transformer model training via co-designing parallel strategies and data assignment.Proc

    Haoyang Li, Fangcheng Fu, Sheng Lin, Hao Ge, Xuanyu Wang, Jiawen Niu, Jinbao Xue, Yangyu Tao, Di Wang, Jie Jiang, and Bin Cui. Hydraulis: Balancing large transformer model training via co-designing parallel strategies and data assignment.Proc. ACM Manag. Data, 2025. https://doi.org/10.1145/3769802

    doi:10.1145/3769802 2025

  34. [34]

    https://arxiv.org/abs/2504.20490

    Pith/arXiv arXiv

  35. [35]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe Twelfth International Conference on Learning Representations,

  36. [36]

    Lobra: Multi-tenant fine-tuning over heterogeneous data.Proc

    Sheng Lin, Fangcheng Fu, Haoyang Li, Hao Ge, Xuanyu Wang, Jiawen Niu, Yaofeng Tu, and Bin Cui. Lobra: Multi-tenant fine-tuning over heterogeneous data.Proc. VLDB Endow., 2025. https://doi.org/10.14778/3742728.3742752

    arXiv 2025

  37. [37]

    Hotprefix: Hotness-aware kv cache scheduling for efficient prefix sharing in llm inference systems.Proc

    Yuhang Li, Rong Gu, Chengying Huan, Zhibin Wang, Renjie Yao, Chen Tian, and Guihai Chen. Hotprefix: Hotness-aware kv cache scheduling for efficient prefix sharing in llm inference systems.Proc. ACM Manag. Data. https://doi.org/ 10.1145/3749168

    doi:10.1145/3749168

  38. [38]

    Ringattention with blockwise trans- formers for near-infinite context

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ringattention with blockwise trans- formers for near-infinite context. InThe Twelfth International Conference on Learning Representations, 2024. https://openreview.net/forum?id=WsRHpHH4s0

    2024

  39. [39]

    https://openreview.net/forum?id=v8L0pN6EOi

  40. [40]

    Flashrl: 8bit rollouts, full power rl, 2025

    Liyuan Liu, Feng Yao, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Flashrl: 8bit rollouts, full power rl, 2025. https://fengyao.notion. site/flash-rl

    2025

  41. [41]

    Spec-rl: Accelerating on-policy reinforce- ment learning with speculative rollouts, 2026

    Bingshuai Liu, Ante Wang, Zijun Min, Liang Yao, Haibo Zhang, Yang Liu, Xu Han, Peng Li, Anxiang Zeng, and Jinsong Su. Spec-rl: Accelerating on-policy reinforce- ment learning with speculative rollouts, 2026. https://arxiv.org/abs/2509.23232

    arXiv 2026

  42. [42]

    Deepcoder: A fully open-source 14b coder at o3- mini level, 2025

    Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open-source 14b coder at o3- mini level, 2025. https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open- Source-14B-Coder-at-O3-mini-Level-1cf81902c146...

    2025

  43. [43]

    When speed kills stability: Demystifying rl collapse from the inference- training mismatch, 2025

    Jiacai Liu, Yingru Li, Yuqian Fu, Jiawei Wang, Qian Liu, and Yu Shen. When speed kills stability: Demystifying rl collapse from the inference- training mismatch, 2025. https://yingru.notion.site/When-Speed-Kills- Stability-Demystifying-RL-Collapse-from-the-Inference-Training-Mismatch- 271211a558b7808d8b12d403fd15edda

    2025

  44. [44]

    Galvatron: Efficient transformer training over multiple gpus using automatic parallelism.Proc

    Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. Galvatron: Efficient transformer training over multiple gpus using automatic parallelism.Proc. VLDB Endow., 2022. https://doi.org/10.14778/3570690. 3570697

    work page doi:10.14778/3570690 2022

  45. [45]

    Part ii: Roll flash – accelerating rlvr and agentic training with asynchrony, 2025

    Han Lu, Zichen Liu, Shaopan Xiong, Yancheng He, Wei Gao, Yanan Wu, Weixun Wang, Jiashun Liu, Yang Li, Haizhou Zhao, Ju Huang, Siran Yang, Xiaoyang Li, Yijia Luo, Zihe Liu, Ling Pan, Junchi Yan, Wei Wang, Wenbo Su, Jiamang Wang, Lin Qu, and Bo Zheng. Part ii: Roll flash – accelerating rlvr and agentic training with asynchrony, 2025. https://arxiv.org/abs/2...

    arXiv 2025

  46. [46]

    Devanur, Gregory R

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: generalized pipeline parallelism for dnn training. InProceedings of the 27th ACM Symposium on Operating Systems Principles, 2019. https://doi.org/10.1145/ 3341301.3359646

    arXiv 2019

  47. [47]

    Real: Efficient RLHF training of large language models with parameter reallocation

    Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, and Yi Wu. Real: Efficient RLHF training of large language models with parameter reallocation. InEighth Conference on Machine Learning and Systems, 2025. https://openreview. net/forum?id=yLU1zRf95d

    2025

  48. [48]

    Asynchronous rlhf: Faster and more efficient off-policy rl for language models, 2025

    Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous rlhf: Faster and more efficient off-policy rl for language models, 2025. https://arxiv.org/abs/2410.18252

    Pith/arXiv arXiv 2025

  49. [49]

    A comprehensive survey of mixture-of-experts: Algo- rithms, theory, and applications, 2025

    Siyuan Mu and Sen Lin. A comprehensive survey of mixture-of-experts: Algo- rithms, theory, and applications, 2025. https://arxiv.org/abs/2503.07137

    arXiv 2025

  50. [50]

    Nvidia inference xfer library (nixl), 2025

    NVIDIA. Nvidia inference xfer library (nixl), 2025. https://github.com/ai- dynamo/nixl

    2025

  51. [51]

    Effi- cient large-scale language model training on gpu clusters using megatron-lm

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Effi- cient large-scale language model training on gpu clusters using megatron-lm. InProceedings of the International Conference for High Perfo...

    arXiv 2021

  52. [52]

    Unified communication x, 2025

    openucx. Unified communication x, 2025. https://github.com/openucx/ucx

    2025

  53. [53]

    Nvidia collective communication library (nccl) documentation, 2025

    NVIDIA. Nvidia collective communication library (nccl) documentation, 2025. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/index.html. 14

    2025

  54. [54]

    Seer: Online context learning for fast synchronous llm reinforcement learning, 2025

    Ruoyu Qin, Weiran He, Weixiao Huang, Yangkun Zhang, Yikai Zhao, Bo Pang, Xinran Xu, Yingdi Shan, Yongwei Wu, and Mingxing Zhang. Seer: Online context learning for fast synchronous llm reinforcement learning, 2025. https: //arxiv.org/abs/2511.14617

    Pith/arXiv arXiv 2025

  55. [55]

    Openai o1 system card,

    OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, et al. Openai o1 system card,

  56. [56]

    https://arxiv.org/abs/2412.16720

    Pith/arXiv arXiv

  57. [57]

    Rosberg and I

    Z. Rosberg and I. Adiri. Multilevel queues with extremal priorities.J. ACM, 1976. https://doi.org/10.1145/321978.321986

    arXiv 1976

  58. [58]

    Multi-step reasoning with large language models, a survey.ACM Comput

    Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki Van Stein, and Thomas Bäck. Multi-step reasoning with large language models, a survey.ACM Comput. Surv., 2025. https://doi.org/10.1145/3774896

    doi:10.1145/3774896 2025

  59. [59]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. https://arxiv.org/abs/1707.06347

    Pith/arXiv arXiv 2017

  60. [60]

    Qwen2.5 technical report,

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, et al. Qwen2.5 technical report,

  61. [61]

    https://arxiv.org/abs/2412.15115

    Pith/arXiv arXiv

  62. [62]

    Zero: Memory optimizations toward training trillion parameter models, 2020

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020. https: //arxiv.org/abs/1910.02054

    Pith/arXiv arXiv 2020

  63. [63]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. InProceedings of the Twentieth European Conference on Computer Systems, 2025. http://dx.doi.org/10.1145/3689031.3696075

    arXiv 2025

  64. [64]

    Trust region policy optimization

    John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. Trust region policy optimization. InProceedings of the 32nd Inter- national Conference on International Conference on Machine Learning, 2015. https://dl.acm.org/doi/10.5555/3045118.3045319

    arXiv 2015

  65. [65]

    Scaling behaviors of llm reinforcement learning post-training: An empirical study in mathematical reasoning, 2025

    Zelin Tan, Hejia Geng, Mulei Zhang, Xiaohang Yu, Guancheng Wan, Yifan Zhou, Qiang He, Xiangyuan Xue, Heng Zhou, Yutao Fan, Zhongzhi Li, Zaibin Zhang, Guibin Zhang, Chen Zhang, Zhenfei Yin, and Lei Bai. Scaling behaviors of llm reinforcement learning post-training: An empirical study in mathematical reasoning, 2025. https://arxiv.org/abs/2509.25300

    Pith/arXiv arXiv 2025

  66. [66]

    Beat the long tail: Distribution-aware speculative decoding for rl training, 2025

    Zelei Shao, Vikranth Srivatsa, Sanjana Srivastava, Qingyang Wu, Alpay Ariyak, Xiaoxia Wu, Ameen Patel, Jue Wang, Percy Liang, Tri Dao, Ce Zhang, Yiying Zhang, Ben Athiwaratkun, Chenfeng Xu, and Junxiong Wang. Beat the long tail: Distribution-aware speculative decoding for rl training, 2025. https://arxiv.org/ abs/2511.13841

    arXiv 2025

  67. [67]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. https://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv 2024

  68. [68]

    Laminar: A scalable asynchronous rl post-training framework, 2025

    Guangming Sheng, Yuxuan Tong, Borui Wan, Wang Zhang, Chaobo Jia, Xibin Wu, Yuqi Wu, Xiang Li, Chi Zhang, Yanghua Peng, Haibin Lin, Xin Liu, and Chuan Wu. Laminar: A scalable asynchronous rl post-training framework, 2025. https://arxiv.org/abs/2510.12633

    arXiv 2025

  69. [69]

    A survey on large language models for mathematical reasoning.ACM Comput

    Peng-Yuan Wang, Tian-Shuo Liu, Chenyang Wang, Ziniu Li, Yidi Wang, Shu Yan, Chengxing Jia, Xu-Hui Liu, Xinwei Chen, Jiacheng Xu, and Yang Yu. A survey on large language models for mathematical reasoning.ACM Comput. Surv., 2025. https://doi.org/10.1145/3786333

    doi:10.1145/3786333 2025

  70. [70]

    Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020. https://arxiv.org/abs/1909.08053

    Pith/arXiv arXiv 2020

  71. [71]

    Improving automatic parallel training via balanced memory workload optimization.IEEE Transactions on Knowledge and Data Engineering, August 2024

    Yujie Wang, Youhe Jiang, Xupeng Miao, Fangcheng Fu, Shenhan Zhu, Xiaonan Nie, Yaofeng Tu, and Bin Cui. Improving automatic parallel training via balanced memory workload optimization.IEEE Transactions on Knowledge and Data Engineering, August 2024. http://dx.doi.org/10.1109/TKDE.2024.3370614

    arXiv 2024

  72. [72]

    Kimi k2: Open agentic intelligence, 2025

    Kimi Team, Yifan Bai, Yiping Bao, et al. Kimi k2: Open agentic intelligence, 2025. https://arxiv.org/abs/2507.20534

    Pith/arXiv arXiv 2025

  73. [73]

    Kimi k1.5: Scaling reinforcement learning with llms, 2025

    Kimi Team, Angang Du, Bofei Gao, et al. Kimi k1.5: Scaling reinforcement learning with llms, 2025. https://arxiv.org/abs/2501.12599

    Pith/arXiv arXiv 2025

  74. [74]

    Tokdar and Robert E

    Surya T. Tokdar and Robert E. Kass. Importance sampling: a review.WIREs Comput. Stat., 2010. https://doi.org/10.1002/wics.56

    doi:10.1002/wics.56 2010

  75. [75]

    Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism

    Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism. InProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, 2024

    2024

  76. [76]

    Reinforcement learning optimization for large-scale learning: An efficient and user-friendly scaling library, 2025

    Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, et al. Reinforcement learning optimization for large-scale learning: An efficient and user-friendly scaling library, 2025. https://arxiv.org/abs/2506.06122

    Pith/arXiv arXiv 2025

  77. [77]

    An adaptive placement and parallelism framework for accelerating rlhf training, 2024

    Youshao Xiao, Zhenglei Zhou, Fagui Mao, Weichang Wu, Shangchun Zhao, Lin Ju, Lei Liang, Xiaolu Zhang, and Jun Zhou. An adaptive placement and parallelism framework for accelerating rlhf training, 2024. https://arxiv.org/abs/2312.11819

    Pith/arXiv arXiv 2024

  78. [78]

    Flexsp: Accelerating large language model training via flexible sequence parallelism

    Yujie Wang, Shiju Wang, Shenhan Zhu, Fangcheng Fu, Xinyi Liu, Xuefeng Xiao, Huixia Li, Jiashi Li, Faming Wu, and Bin Cui. Flexsp: Accelerating large language model training via flexible sequence parallelism. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 2025. http...

    arXiv 2025

  79. [79]

    Re- inforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms, 2025

    Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, Jiang Bian, and Mao Yang. Re- inforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms, 2025. https://arxiv.org/abs/2506.14245

    Pith/arXiv arXiv 2025

  80. [80]

    The multiqueue: A simple and fast relaxed concurrent priority queue.ACM Trans

    Marvin Williams and Peter Sanders. The multiqueue: A simple and fast relaxed concurrent priority queue.ACM Trans. Parallel Comput., 2025. https://doi.org/ 10.1145/3771738

    work page doi:10.1145/3771738 2025

Showing first 80 references.