DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning

Bin Cui; Fangcheng Fu; Longzan Luo; Siwei Chen; Xinyi Liu; Xupeng Miao; Yujie Wang

arxiv: 2605.30859 · v1 · pith:DDSM6DIOnew · submitted 2026-05-29 · 💻 cs.LG · cs.AI

DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning

Yujie Wang , Siwei Chen , Longzan Luo , Xinyi Liu , Xupeng Miao , Fangcheng Fu , Bin Cui This is my paper

Pith reviewed 2026-06-28 23:24 UTC · model grok-4.3

classification 💻 cs.LG cs.AI

keywords reinforcement learninglarge language modelsrollout efficiencytrajectory samplingdistribution shapingactive learning

0 comments

The pith

Active distribution shaping removes ineffective verbosity from LLM rollout trajectories to accelerate reinforcement learning.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that reinforcement learning for large language models is slowed by long-tail distributions in response lengths, specifically intra-prompt tails that are often just verbose but ineffective outputs. By characterizing these tails at a fine granularity, the authors show they can be mitigated by actively shaping the rollout distribution toward shorter, more certain responses. This is done via a sampling mechanism that picks trajectories from redundant spaces per prompt and an adaptive way to allocate redundancy. If successful, this would allow faster RL training cycles for LLMs without hurting the final model quality. Sympathetic readers would care because RL is crucial for aligning and improving LLMs but currently bottlenecked by computation on long responses.

Core claim

The central discovery is that intra-prompt long tails in LLM rollouts frequently consist of ineffective verbosity, and a distribution-aware trajectory sampling mechanism that selects from a redundant exploration space, paired with adaptive redundancy allocation, can shape the distribution toward conciseness and certainty, thereby resolving tail-induced overheads and achieving up to 1.77x acceleration over state-of-the-art systems without compromising model performance.

What carries the argument

The distribution-aware trajectory sampling mechanism, which selects trajectories from a redundant exploration space for each prompt, together with an adaptive redundancy allocation scheme.

If this is right

RL for LLMs becomes more efficient by focusing computation on concise trajectories.
State-of-the-art systems can be accelerated by up to 1.77x.
Model performance remains intact while reducing overhead from long tails.
The approach addresses the root source of inefficiency in the distribution itself rather than just scheduling.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

This method might generalize to other domains where RL involves generating long sequences with variable lengths.
It could lead to new ways of thinking about exploration in LLM training that prioritize certainty over exhaustive verbosity.
Practitioners might see reduced compute costs for fine-tuning LLMs with RL.

Load-bearing premise

Intra-prompt long tails consist mostly of ineffective verbosity that sampling can filter without losing the diversity required for effective policy improvement.

What would settle it

An experiment where removing the sampled trajectories leads to degraded final model performance or slower convergence in policy improvement compared to full sampling.

Figures

Figures reproduced from arXiv: 2605.30859 by Bin Cui, Fangcheng Fu, Longzan Luo, Siwei Chen, Xinyi Liu, Xupeng Miao, Yujie Wang.

**Figure 1.** Figure 1: Inter-prompt and intra-prompt rollout trajectory length distribution. (Qwen3-30B-A3B on DAPO-MATH dataset) Qwen3 (Yang et al., 2025), Kimi-K2 (Team et al., 2025a) and so on (Li et al., 2025). By shifting the focus from pre-training scaling to inference-time scaling, RL has empowered state-of-the-art models with robust reasoning capabilities, enabling them to tackle complex mathematical problems (Shao et … view at source ↗

**Figure 2.** Figure 2: Rollout length distribution within a single prompt, w/o and w/ distribution shaping. P99 denotes the 99th percentile. Rollout (Team et al., 2025b; Zhou et al., 2025) speculatively over-samples and truncates the unfinished long-tail prompts when sufficient data is collected, resuming them in subsequent steps. Fundamentally, these approaches identify the inter-prompt long-tail distribution ( [PITH_FULL_IMA… view at source ↗

**Figure 3.** Figure 3: Illustration of distinct rollout distribution patterns. from the policy model πθ, estimating the gradient as: J(θ) = Eqi∼P(Q),{o j i }∼πθ(·|qi) " 1 M XM j=1 A(o j i ) · ∇θ log πθ(o j i |qi) # (1) Here, ∇θ log πθ(o j i |qi) represents the gradient of logprobability, and A(o j i ) is the advantage computed by normalizing rewards within the group: A(o j i ) = r j i − µ(Yi) σ(Yi) (2) where r j i is the rewar… view at source ↗

**Figure 4.** Figure 4: Illustration of DARTS method: adaptive sampling strategies and redundancy allocations for different prompts (M = 8). reward correlation. Specifically, for a given prompt qi , we compare the expected response lengths of correct (E[l j i |r j i > 0]) and incorrect (E[l j i |r j i < 0]) responses, identifying two distinct rollout distribution patterns. Pattern I: Verbose and Ineffective Tails. In the first p… view at source ↗

**Figure 6.** Figure 6: Efficiency comparison of w/o and w/ variance-guided tail pruning scheme. (Qwen2.5-Math-7B on 32 H20) Rollout GPUs Training GPUs Saved time (a) Sample-level streaming. (b) Token-level streaming. Rollout GPUs Training GPUs [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗

**Figure 7.** Figure 7: Timeline of sample-level and token-level streaming. to high-variance prompts (large σ˜L(qi)) yields a larger reduction in distance than allocating to low-variance ones. Consequently, the optimization problem is defined as: min {M′ 1 ,...,M′ N } X N i=1 Norm(˜σL(qi)) M′ i s.t. X N i=1 M′ i = Mtotal, M′ i ∈ Z + Mlow ≤ M′ i ≤ Mup, ∀i ∈ {1, . . . , N} (5) where Mlow and Mup are hyper-parameters (set to M and … view at source ↗

**Figure 8.** Figure 8: End-to-end throughput comparison for various model sizes. Speedup ratios compared to VeRL are indicated. 200 400 600 800 steps 0.2 0.4 0.6 Accuracy VeRL DARTS (a) Qwen2.5-Math-7B 50 100 150 200 250 steps 0.2 0.4 0.6 Accuracy VeRL DARTS (b) Qwen2.5-32B Qwen2.5 3B Qwen2.5 Math-7B Qwen2.5 14B Qwen3 30B-A3B Qwen2.5 32B 20 30 40 50 VeRL DARTS (c) Benchmarks [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗

**Figure 9.** Figure 9: Training convergence (left) and benchmark scores (right). mechanism. We leverage the decision from our variancebased allocation (Section 4.2), and treat the saturation of the budget (M′ i = Mup) as a proxy for severe long-tail characteristics. In such cases, we dynamically switch from the dual-end sampling to shortest-only sampling strategy, which exclusively samples the top-M shortest trajectories. This… view at source ↗

**Figure 10.** Figure 10: Dataset-level rollout distribution shaping effect. 500 1000 1500 2000 2500 3000 3500 Response Length Density Pattern I 1000 1500 2000 2500 3000 Response Length Pattern II VeRL Reward>0 VeRL Reward<0 DARTS Reward>0 DARTS Reward<0 [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗

**Figure 11.** Figure 11: Distribution shaping effect for different prompt patterns. 32 GPUs. As shown in Tab. 3, the efficiency gains primarily stem from our distribution-aware sampling (Section 4.1) and adaptive redundancy allocation (Section 4.2). Our sampling scheme, with dual-end sampling as the core, effectively shapes the rollout distribution toward conciseness and compactness, achieving a 1.40× speedup even under a na¨ıve… view at source ↗

**Figure 12.** Figure 12: PTL curve fitting for TP=2 [PITH_FULL_IMAGE:figures/full_fig_p012_12.png] view at source ↗

**Figure 14.** Figure 14: Average length comparison of Qwen2.5-14B training process on DAPO-MATH dataset. 0 100 200 300 400 500 Steps 0 2000 4000 6000 8000 Max Response Length VeRL DARTS [PITH_FULL_IMAGE:figures/full_fig_p015_14.png] view at source ↗

read the original abstract

Reinforcement Learning (RL) has become pivotal for improving model capabilities yet suffers from rollout efficiency bottlenecks due to the long-tail response length distribution. While existing works mitigate the impact of long tails via prompt-level tail scheduling, we focus on the root source of inefficiency: the distribution itself. Specifically, we characterize the long-tail distribution at a finer granularity, identifying intra-prompt long tails, and revealing that they frequently consist of ineffective verbosity. To address this, we propose a novel paradigm of active distribution shaping to shape the rollout distribution towards conciseness and certainty, thereby fundamentally resolving tail-induced overheads. We achieve this through a distribution-aware trajectory sampling mechanism, which selects trajectories from a redundant exploration space for each prompt, and an adaptive redundancy allocation scheme to maximize both shaping effectiveness and system efficiency. Experiments demonstrate significant acceleration over state-of-the-art systems by up to 1.77x without compromising model performance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper shifts RL rollout shaping to intra-prompt granularity with a sampling mechanism to prune long tails, claiming 1.77x speedup, but the abstract gives almost no experimental controls or validation of the core assumption.

read the letter

The main idea here is to move beyond prompt-level tail scheduling and instead actively shape the trajectory distribution inside each prompt. They characterize intra-prompt long tails as mostly ineffective verbosity, then use a distribution-aware sampler to pick from a redundant exploration space plus an adaptive allocation scheme to balance shaping and efficiency.

What is new is the intra-prompt focus and the specific sampling-plus-allocation combination. The abstract positions this as a direct attack on the root distribution rather than a workaround. If the full paper shows that the mechanism actually reduces average rollout length while preserving the policy gradient signal, that would be a practical step for anyone running PPO-style RL on LLMs.

The reported 1.77x acceleration over state-of-the-art systems is the headline number, but the abstract supplies no details on baselines, statistical tests, model sizes, or task coverage. That leaves the central claim hard to evaluate. The weakest link is the assumption that long tails are safely prunable verbosity. If a non-trivial fraction of those extended trajectories contain rare high-reward reasoning paths or necessary diversity, the shaping step could quietly reduce exploration quality even if final benchmark scores look unchanged.

This work is aimed at people who already run RL post-training at scale and are looking for engineering levers on rollout cost. It is worth sending to peer review so the methods section and ablations can be checked; the idea is concrete enough that a referee could test whether the gains survive when the diversity assumption is stress-tested. I would not cite it yet without seeing the full experiments.

Referee Report

2 major / 0 minor

Summary. The paper claims that intra-prompt long tails in LLM RL rollouts are predominantly ineffective verbosity, and introduces DARTS: a distribution-aware trajectory sampling mechanism that selects from a redundant exploration space per prompt, combined with adaptive redundancy allocation, to actively shape the rollout distribution toward conciseness. This is asserted to yield up to 1.77x acceleration over SOTA systems without compromising model performance.

Significance. If the core assumption and empirical results hold, the work could meaningfully improve rollout efficiency in RL for LLMs by targeting the distribution itself rather than prompt-level scheduling. The paradigm of active distribution shaping is presented as a new approach that could apply more broadly to long-tailed RL settings.

major comments (2)

[Abstract] Abstract: The central claim of 'significant acceleration ... by up to 1.77x without compromising model performance' is stated without any reported metrics, baselines, controls, or statistical tests. This is load-bearing for the 'no performance compromise' assertion, as the method's safety depends on demonstrating that pruned trajectories do not remove necessary diversity for policy improvement.
[Abstract] Abstract: The characterization of intra-prompt long tails as 'frequently consist[ing] of ineffective verbosity' is presented without quantitative evidence (e.g., fraction of tail trajectories that are high-reward, diverse, or correct-but-extended). If a non-negligible portion carries useful exploration, the sampling and allocation scheme risks biasing gradients or reducing sample efficiency, directly undermining the acceleration result.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on the abstract. We agree that strengthening the abstract with explicit metrics and quantitative support will improve clarity and address the concerns about self-containment. We will revise the manuscript accordingly while preserving the core contributions.

read point-by-point responses

Referee: [Abstract] Abstract: The central claim of 'significant acceleration ... by up to 1.77x without compromising model performance' is stated without any reported metrics, baselines, controls, or statistical tests. This is load-bearing for the 'no performance compromise' assertion, as the method's safety depends on demonstrating that pruned trajectories do not remove necessary diversity for policy improvement.

Authors: We agree the abstract should be more self-contained. In the revision we will update the abstract to explicitly report the 1.77x speedup (with the corresponding baseline system), the evaluation benchmarks, and a concise statement that performance is preserved or improved (with reference to the reward curves, diversity metrics, and statistical tests already present in Sections 4 and 5). These sections already contain the controls showing that the shaped distribution does not remove high-reward or diverse trajectories required for policy improvement; the abstract revision will surface this evidence. revision: yes
Referee: [Abstract] Abstract: The characterization of intra-prompt long tails as 'frequently consist[ing] of ineffective verbosity' is presented without quantitative evidence (e.g., fraction of tail trajectories that are high-reward, diverse, or correct-but-extended). If a non-negligible portion carries useful exploration, the sampling and allocation scheme risks biasing gradients or reducing sample efficiency, directly undermining the acceleration result.

Authors: The full manuscript already contains quantitative support for this characterization (reward-vs-length scatter plots and histograms in Section 3.2, plus the fraction of tail trajectories that are both high-reward and low-diversity). To make the abstract self-contained we will add a short quantitative clause (e.g., “>70 % of intra-prompt tail trajectories exhibit low reward and low diversity”) and will reference the corresponding figures. This directly addresses the concern that useful exploration might be removed; the added numbers will show that the adaptive sampling preserves the minority of high-value tail trajectories. revision: yes

Circularity Check

0 steps flagged

No significant circularity; method presented as novel paradigm without reduction to fitted inputs or self-citations

full rationale

The abstract and description introduce a new active distribution shaping paradigm via distribution-aware trajectory sampling and adaptive redundancy allocation, characterizing intra-prompt long tails as ineffective verbosity. No equations, parameter fits, predictions derived from subsets of data, or self-citations appear in the provided text. The central claims rest on empirical characterization and a proposed mechanism rather than any self-definitional loop, fitted-input renaming, or load-bearing self-citation chain, rendering the derivation self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are stated or derivable from the provided text.

pith-pipeline@v0.9.1-grok · 5705 in / 972 out tokens · 16315 ms · 2026-06-28T23:24:00.949269+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

14 extracted references · 12 canonical work pages · 8 internal anchors

[1]

Training Verifiers to Solve Math Word Problems

Anthropic. Claude 4.5 opus. https://www. anthropic.com/news/claude-opus-4-5/ , 2025a. Anthropic. Claude 4.5 sonnet. https://www. anthropic.com/news/claude-sonnet-4-5/ , 2025b. Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv...

work page internal anchor Pith review Pith/arXiv arXiv
[2]

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

work page internal anchor Pith review Pith/arXiv arXiv
[3]

Asyncflow: An asynchronous streaming rl framework for efficient llm post-training.arXiv preprint arXiv:2507.01663,

Han, Z., You, A., Wang, H., Luo, K., Yang, G., Shi, W., Chen, M., Zhang, S., Lan, Z., Deng, C., et al. Asyncflow: An asynchronous streaming rl framework for efficient llm post-training.arXiv preprint arXiv:2507.01663,

work page arXiv
[4]

OpenAI o1 System Card

Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

work page internal anchor Pith review Pith/arXiv arXiv
[5]

Let’s verify step by step

Lightman, H., Kosaraju, V ., Burda, Y ., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. InInternational Conference on Learning Representations, volume 2024, pp. 39578–39601,

2024
[6]

Part ii: Roll flash– accelerating rlvr and agentic training with asynchrony

10 DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning Lu, H., Liu, Z., Xiong, S., He, Y ., Gao, W., Wu, Y ., Wang, W., Liu, J., Li, Y ., Zhao, H., et al. Part ii: Roll flash– accelerating rlvr and agentic training with asynchrony. arXiv preprint arXiv:2510.11345,

work page arXiv
[7]

Proximal Policy Optimization Algorithms

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

work page internal anchor Pith review Pith/arXiv arXiv
[8]

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

work page internal anchor Pith review Pith/arXiv arXiv
[9]

Kimi K2: Open Agentic Intelligence

Team, K., Bai, Y ., Bao, Y ., Chen, G., Chen, J., Chen, N., Chen, R., Chen, Y ., Chen, Y ., Chen, Y ., et al. Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025a. Team, K., Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., Li, C., Xiao, C., Du, C., Liao, C., et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arX...

work page internal anchor Pith review Pith/arXiv arXiv
[10]

Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math techni- cal report: Toward mathematical expert model via self- improvement.arXiv preprint arXiv:2409.12122,

work page internal anchor Pith review Pith/arXiv arXiv
[11]

Qwen3 Technical Report

Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

work page internal anchor Pith review Pith/arXiv arXiv
[12]

StreamRL: Scalable, hetero- geneous, and elastic RL for LLMs with disaggregated stream generation.arXiv preprint arXiv:2504.15930, 2025

Zhong, Y ., Zhang, Z., Song, X., Hu, H., Jin, C., Wu, B., Chen, N., Chen, Y ., Zhou, Y ., Wan, C., et al. Streamrl: Scalable, heterogeneous, and elastic rl for llms with disaggregated stream generation.arXiv preprint arXiv:2504.15930,

work page arXiv
[13]

Zhou, Y ., Li, J., Su, Y ., Ramesh, G., Zhu, Z., Long, X., Zhao, C., Pan, J., Yu, X., Wang, Z., et al

doi: 10.1007/S41019-023-00235-6. Zhou, Y ., Li, J., Su, Y ., Ramesh, G., Zhu, Z., Long, X., Zhao, C., Pan, J., Yu, X., Wang, Z., et al. April: Active partial rollouts in reinforcement learning to tame long-tail generation.arXiv preprint arXiv:2509.18521,

work page doi:10.1007/s41019-023-00235-6
[14]

This trend confirms that our method effectively mitigates the verbose long-tail issue early on, greatly enhancing training efficiency

As shown by the orange curves (DARTS), both the average and maximum response lengths exhibit a significant reduction during the early and middle stages of training compared to the baseline (VeRL). This trend confirms that our method effectively mitigates the verbose long-tail issue early on, greatly enhancing training efficiency. Crucially, the data revea...

2000

[1] [1]

Training Verifiers to Solve Math Word Problems

Anthropic. Claude 4.5 opus. https://www. anthropic.com/news/claude-opus-4-5/ , 2025a. Anthropic. Claude 4.5 sonnet. https://www. anthropic.com/news/claude-sonnet-4-5/ , 2025b. Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv...

work page internal anchor Pith review Pith/arXiv arXiv

[2] [2]

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

work page internal anchor Pith review Pith/arXiv arXiv

[3] [3]

Asyncflow: An asynchronous streaming rl framework for efficient llm post-training.arXiv preprint arXiv:2507.01663,

Han, Z., You, A., Wang, H., Luo, K., Yang, G., Shi, W., Chen, M., Zhang, S., Lan, Z., Deng, C., et al. Asyncflow: An asynchronous streaming rl framework for efficient llm post-training.arXiv preprint arXiv:2507.01663,

work page arXiv

[4] [4]

OpenAI o1 System Card

Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

work page internal anchor Pith review Pith/arXiv arXiv

[5] [5]

Let’s verify step by step

Lightman, H., Kosaraju, V ., Burda, Y ., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. InInternational Conference on Learning Representations, volume 2024, pp. 39578–39601,

2024

[6] [6]

Part ii: Roll flash– accelerating rlvr and agentic training with asynchrony

10 DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning Lu, H., Liu, Z., Xiong, S., He, Y ., Gao, W., Wu, Y ., Wang, W., Liu, J., Li, Y ., Zhao, H., et al. Part ii: Roll flash– accelerating rlvr and agentic training with asynchrony. arXiv preprint arXiv:2510.11345,

work page arXiv

[7] [7]

Proximal Policy Optimization Algorithms

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

work page internal anchor Pith review Pith/arXiv arXiv

[8] [8]

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

work page internal anchor Pith review Pith/arXiv arXiv

[9] [9]

Kimi K2: Open Agentic Intelligence

Team, K., Bai, Y ., Bao, Y ., Chen, G., Chen, J., Chen, N., Chen, R., Chen, Y ., Chen, Y ., Chen, Y ., et al. Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025a. Team, K., Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., Li, C., Xiao, C., Du, C., Liao, C., et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arX...

work page internal anchor Pith review Pith/arXiv arXiv

[10] [10]

Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math techni- cal report: Toward mathematical expert model via self- improvement.arXiv preprint arXiv:2409.12122,

work page internal anchor Pith review Pith/arXiv arXiv

[11] [11]

Qwen3 Technical Report

Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

work page internal anchor Pith review Pith/arXiv arXiv

[12] [12]

StreamRL: Scalable, hetero- geneous, and elastic RL for LLMs with disaggregated stream generation.arXiv preprint arXiv:2504.15930, 2025

Zhong, Y ., Zhang, Z., Song, X., Hu, H., Jin, C., Wu, B., Chen, N., Chen, Y ., Zhou, Y ., Wan, C., et al. Streamrl: Scalable, heterogeneous, and elastic rl for llms with disaggregated stream generation.arXiv preprint arXiv:2504.15930,

work page arXiv

[13] [13]

Zhou, Y ., Li, J., Su, Y ., Ramesh, G., Zhu, Z., Long, X., Zhao, C., Pan, J., Yu, X., Wang, Z., et al

doi: 10.1007/S41019-023-00235-6. Zhou, Y ., Li, J., Su, Y ., Ramesh, G., Zhu, Z., Long, X., Zhao, C., Pan, J., Yu, X., Wang, Z., et al. April: Active partial rollouts in reinforcement learning to tame long-tail generation.arXiv preprint arXiv:2509.18521,

work page doi:10.1007/s41019-023-00235-6

[14] [14]

This trend confirms that our method effectively mitigates the verbose long-tail issue early on, greatly enhancing training efficiency

As shown by the orange curves (DARTS), both the average and maximum response lengths exhibit a significant reduction during the early and middle stages of training compared to the baseline (VeRL). This trend confirms that our method effectively mitigates the verbose long-tail issue early on, greatly enhancing training efficiency. Crucially, the data revea...

2000