Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

UloRL:An Ultra-Long Output Reinforcement Learning Approach for Advancing Large Language Models' Reasoning Abilities

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training on 128k-token outputs with segmented rollouts and entropy masking lifts a 30B model's reasoning scores above a 235B model's.

desk verdict The segment rollout speedup and DMMPTs are the real contributions; the abstract's headline 128k numbers are actually the Yarn variant's, not the plain 128k model's. read the letter →

arxiv 2507.19766 v1 pith:4YRFZKTU submitted 2025-07-26 cs.CL cs.AI

Dong Du , Shulin Liu , Tao Yang , Shaohua Chen , Yang Li This is my paper
classification cs.CLcs.AI
keywords ultra-longoutputreinforcementlearningsegmentrolloutpseudoon-policyimportancesamplingdynamicmaskingofwell-masteredpositivetokensentropycollapsegenerativeverifiermathematicalreasoningbenchmarksQwen3-30B-A3B
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the practical ceiling on reinforcement learning for reasoning is not the model but the training harness: ultra-long outputs bottleneck batches and the policy collapses into repetition. Its proposed recipe, UloRL, splits 128k-token decoding into 16k-token segments, uses a pseudo on-policy importance-sampling estimator to keep training stable, and masks tokens the model already masters when entropy dips. If the reported experiments hold, these changes make 128k-token RL training about twice as fast per step and lift a 30B-parameter model's AIME2025 score from 70.9% to 85.1%, above a 235B-parameter model.

What carries the argument

The machinery is a staged training loop. Segment rollout divides the 128k-token maximum into eight 16k-token chunks, so samples that finish early enter the experience pool immediately while unfinished ones continue in later iterations; this is what yields the 2.06x speedup at four segments. Pseudo On-policy Importance Sampling (POIS) then recomputes every token's importance weight as if all segments came from the current policy, forcing every ratio to 1 and disabling clipping, which the paper says stabilizes entropy. Dynamic Masking of well-Mastered Positive Tokens (DMMPTs) masks any token with predicted probability above $\tau=0.99$ in a reward-1 sample whenever the current sequence entropy falls below a target $\sigma=0.2$. A generative verifier judges whether a predicted answer is equivalent to the reference, and a data-cleaning pipeline removes multi-part, trivial, or mislabeled questions; together these supply the reward signal.

What would settle it

Run the identical 128k-token recipe on the same 30B model but replace POIS with a consistent off-policy estimator that keeps each segment's true behavior policy, holding all other hyperparameters fixed; if the AIME2025 and BeyondAIME gains over the base model shrink substantially, or the entropy curves stop matching the on-policy case, the reported benefits are attributable to the biased estimator rather than to the segment-rollout scheme itself.

Watch

Extended reading notes

Core claim

The central claim is that entropy collapse in long-output RL comes primarily from overtraining positive-sample tokens the model already predicts with high probability, and that the right fix is to mask those tokens dynamically rather than add an entropy bonus or reweight samples. Combined with segment rollout, which lets finished samples train immediately instead of waiting for the longest samples in a batch, and with a generative verifier that replaces brittle string matching for answer equivalence, the paper reports that a 30B model trained with 128k-token responses surpasses a 235B reference model on AIME2025 and BeyondAIME. The same ablation shows that removing the dynamic masking drops average accuracy by about 3.6 points, and increasing the output cap from 32k to 128k produces a monotone climb in benchmark scores.

Load-bearing premise

The load-bearing premise is that pretending earlier rollout segments were produced by the current model, instead of by the older models that actually produced them, does not distort the training signal enough to change the outcome.

Editorial extensions

If this is right

  • A 4-segment rollout makes each RL update 2.06x faster than a single-segment 64k rollout, so ultra-long outputs become affordable to train on.
  • Length itself appears to be a training axis: average accuracy on AIME2025 and BeyondAIME climbs from 62.9% at 32k outputs to 71.5% at 128k outputs.
  • DMMPTs is a load-bearing component: without it the same recipe averages 67.9% instead of 71.5%.
  • A 30B-parameter model trained this way reports 82.8% on AIME2025 against 81.5% for a 235B-parameter model, and the context-extended variant reaches 85.1%.
  • Parameter count is not the only route to stronger reasoning: the smaller trained model surpasses the much larger reference on both benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because POIS replaces the true behavior-policy probabilities of earlier segments with the current policy, its importance ratios are not unbiased estimates of the segment-rollout objective; the paper does not bound this bias, so a direct comparison against a consistent off-policy estimator would tell whether POIS's stability is a real property of the objective or an artifact of the estimator.
  • The two DMMPTs thresholds, $\tau=0.99$ and $\sigma=0.2$, are presented as fixed choices; transferring the recipe to other model families or reward schemes would likely require re-tuning them, and the paper does not characterize that sensitivity.
  • The headline gain bundles segment rollout, POIS, DMMPTs, the generative verifier, and data cleaning; separate ablations of the verifier and data pipeline would reveal how much of the 14-point AIME improvement is due to the RL algorithm alone.
  • If the monotone length-performance trend continues, context extension beyond 128k may give further gains, but the paper gives no evidence about where the curve saturates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes UloRL, a collection of RLVR training techniques for long-output reasoning: segment rollout with two importance-sampling variants (SAIS and POIS), dynamic masking of well-mastered positive tokens (DMMPTs), a generative verifier model, data cleaning and transformation, and a simple overlong-punishment rule. Experiments on Qwen3-30B-A3B report a 2.06x training speedup with four segments at 64k output, and Table 2 reports AIME2025/BeyondAIME improvements from 70.9/50.7 to 82.8/60.2 for the 128k model, with 85.1/61.9 for an additional YaRN-extended 140k variant. The abstract and introduction attribute the higher numbers to the 128k model, which is inconsistent with Table 2. The paper also presents entropy-dynamics plots for token masking and one ablation of DMMPTs.

Significance. If the results hold, the paper demonstrates a practical recipe for RLVR at very long output lengths and offers a plausible mechanism for entropy stabilization via masking of high-confidence positive tokens. The direct speedup measurement in Table 1 is a concrete and reproducible contribution, and the planned release of code and models would help the community verify and build on the work. The main caveats are the abstract/table mismatch, the unanalyzed bias of POIS, and the mixed evaluation protocol; none of these destroys the core empirical finding, but they must be addressed before the claims can be accepted as stated.

major comments (4)
  1. [Abstract; §1; §4.2, Table 2] The abstract and Section 1 state that RL training with 128k-token outputs raises AIME2025 from 70.9% to 85.1% and BeyondAIME from 50.7% to 61.9%. Table 2, however, reports those exact numbers only for the UloRL-A3B-128k-Yarn variant, which additionally extends the output length to 140k via YaRN (Section 4.2, paragraph 3). The plain 128k model, UloRL-A3B-128k, scores 82.8% and 60.2%. The headline comparison to Qwen3-235B-A22B is therefore stated with inflated numbers, even though the 128k values still surpass the baseline. Please correct the abstract and introduction, and make the 128k vs. 140k distinction explicit wherever the result is summarized.
  2. [§3.1.2, POIS] The POIS estimator sets π_θ_old to the current rollout policy for all segments, forcing every importance ratio to 1. For segments generated by earlier policies, this is not an on-policy estimator; it is an unweighted off-policy update with no importance correction and no bias analysis. The paper attributes the entropy stability and accuracy advantage of POIS over SAIS to the benefits of on-policy training, but the comparison conflates the estimator bias with the rollout scheme. Since POIS is the only importance-sampling variant used in the final UloRL experiments, please provide a bias bound, a correction term, or an experiment that isolates the estimator (for example, comparing POIS against SAIS at the same 128k setting, or measuring the gradient difference on held-out segments).
  3. [§4.2, Table 2] The evaluation protocol is not uniform across the rows used for the headline comparison. The table footnote indicates that metrics marked with an asterisk are from the authors' evaluation, while unmarked metrics are from official reports; for the SOTA baselines this mixes official AIME-2025 numbers with self-evaluated BeyondAIME numbers, whereas all UloRL numbers are produced by the authors' pipeline. Because the claim that UloRL surpasses Qwen3-235B-A22B depends on this comparison, please either rerun all baselines under the same avg@32 protocol or restrict the claim to comparisons that use a single protocol.
  4. [§4.2, Table 2; §3.3–§3.5] The contribution attribution is underdetermined. Table 2 ablates only DMMPTs (UloRL-A3B-w/o-DMMPTs), while the generative verifier, data cleaning/transformation, and overlong-punishment rule are never individually ablated. The POIS-vs-SAIS comparison in Figure 2 is also run at 4k, 32k, and 64k output lengths, not at the final 128k configuration. Please add at least one ablation for the verifier and one for the data-cleaning pipeline, or soften the conclusion that each component is necessary for the reported gains.
minor comments (5)
  1. [§1] The introduction contains several typos: 'Pesudo On-Policy Importance Sampling' should be 'Pseudo', 'welll-Mastered' should be 'well-Mastered', and 'duiring' should be 'during'.
  2. [Eq. (8)] In the SAIS importance ratio, the denominator is written as π_{θ_old^{f(i)}}(o_{i,t} | q, o_{i,<|s_i|}); the context subscript appears inconsistent with the token index t and with Equation (6). Please define f(i) and the context notation precisely.
  3. [Table 2] The column header 'A VG' appears broken; it should read 'Avg' or 'Average'.
  4. [§3.1.1, Algorithm 1] The algorithm does not specify how many gradient updates are performed per rollout step, how the experience pool is sampled for the mini-batch of size 1024, or how samples with different segment counts are weighted in the loss. Please add these details for reproducibility.
  5. [§4.1] The hardware, distributed configuration, and total training steps/tokens are not reported; these details are needed to contextualize the 2.06x speedup and the overall cost of the 128k runs.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: UloRL's claims rest on external benchmark measurements and ablations; the abstract/Table 2 discrepancy over the Yarn variant is a reporting inconsistency, not a self-referential derivation.

full rationale

The paper derives no result from its own fitted values. Training efficiency (Table 1), benchmark scores (Tables 2 and 3), and entropy dynamics (Figures 4 and 5) are measured outcomes, not predictions implied by the definitions. The POIS estimator in Section 3.1.2 does replace all earlier-segment behavior policies with the current policy so that importance ratios equal 1 and clipping is disabled; this is an explicit estimator design that the authors acknowledge as pseudo on-policy, so it is a possible bias/validity concern but not a circular reduction. DMMPTs defines MPTs by a probability threshold and masks them when entropy is below sigma, but the entropy-stability conclusion is an empirical observation across three model sizes, not a tautology, and the final AIME/BeyondAIME scores are not fitted to sigma or tau. Hyperparameters such as sigma=0.2, tau=0.99, and epsilon_high=0.28 are hand-selected and could affect results, but tuning is not circularity. The prior-work citations (K1.5, DAPO, W-Reinforce, generative verifier literature) are external and are not used as self-supporting uniqueness arguments; the authors do not cite their own prior work as load-bearing. The one substantive discrepancy is internal: the Abstract attributes 85.1% AIME2025 / 61.9% BeyondAIME to 'RL training with 128k-token outputs,' while Table 2 and Section 4.2 list those numbers only for UloRL-A3B-128k-Yarn, the Yarn-extended 140k variant, with the actual 128k row at 82.8/60.2. That weakens the headline comparison to Qwen3-235B-A22B but is a reporting inconsistency, not an equivalence-by-construction or fitted-input-as-prediction, so it does not affect the circularity score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central claim rests on empirically chosen hyperparameters (tau, sigma), a biased importance-sampling shortcut, and data-filtering choices. These are not derived constants; they are hand-set and only weakly evaluated. No formal guarantees are provided for the unbiasedness or safety of the proposed training modifications.

free parameters (4)
  • MPT probability threshold tau = 0.99
    Defines well-mastered positive tokens in Eq. 9; set by hand with no sensitivity analysis, and it determines what gets masked.
  • Target entropy sigma = 0.2
    Threshold in Eq. 10 that triggers masking; hand-set per model and experiment without a reported tuning procedure.
  • Maximum segment count = 8 for 128k, 4 for 64k, 1 for 32k
    Rollout hyperparameter that controls speed and off-policy composition; Table 1 shows it affects training time and later experiments vary it.
  • Per-segment length = 16k tokens for 128k outputs
    Derived from global max length divided by segment count; part of the design.
assumptions (5)
  • domain assumption Increasing output length improves reasoning performance.
    Motivates the whole 128k target; cited to K1.5 but treated as a premise for the central comparison.
  • ad hoc to paper Entropy collapse is caused primarily by overtraining of well-mastered positive tokens.
    Section 3.2.1 supports this only by comparing baseline vs masking; no random-token control, so the causal role of MPTs specifically is assumed.
  • ad hoc to paper Masking MPTs when entropy is low does not harm final reasoning performance.
    Stated as expectation in Section 3.2.2; no independent test outside the one ablation.
  • domain assumption The generative verifier can judge answer equivalence accurately enough for reward computation.
    Section 3.3 introduces the verifier without training or evaluation details, so correctness of the reward signal is assumed.
  • ad hoc to paper Replacing behavior policies with the current policy in POIS yields a valid training signal.
    Section 3.1.2; no bias analysis is provided for off-policy segments.
invented entities (1)
  • Well-Mastered Positive Tokens (MPTs)
    purpose: Identify high-probability tokens in correct answers to be masked when entropy is low.
    Defined via threshold tau in Eq. 9; no external validation of the construct beyond the paper's own entropy curves.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UloRL:An Ultra-Long Output Reinforcement Learning Approach for Advancing Large Language Models' Reasoning Abilities." pith.science (2026). https://pith.science/paper/4YRFZKTU

@misc{pith2026250719766,
  author       = {Pith},
  title        = {Pith review of: UloRL:An Ultra-Long Output Reinforcement Learning Approach for Advancing Large Language Models' Reasoning Abilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4YRFZKTU}},
  note         = {Machine review of arXiv:2507.19766}
}
read the original abstract

Recent advances in large language models (LLMs) have highlighted the potential of reinforcement learning with verifiable rewards (RLVR) to enhance reasoning capabilities through extended output sequences. However, traditional RL frameworks face inefficiencies when handling ultra-long outputs due to long-tail sequence distributions and entropy collapse during training. To address these challenges, we propose an Ultra-Long Output Reinforcement Learning (UloRL) approach for advancing large language models' reasoning abilities. Specifically, we divide ultra long output decoding into short segments, enabling efficient training by mitigating delays caused by long-tail samples. Additionally, we introduce dynamic masking of well-Mastered Positive Tokens (MPTs) to prevent entropy collapse. Experimental results demonstrate the effectiveness of our approach. On the Qwen3-30B-A3B model, RL with segment rollout achieved 2.06x increase in training speed, while RL training with 128k-token outputs improves the model's performance on AIME2025 from 70.9\% to 85.1\% and on BeyondAIME from 50.7\% to 61.9\%, even surpassing Qwen3-235B-A22B with remarkable gains. These findings underscore the potential of our methods to advance the reasoning capabilities of LLMs with ultra-long sequence generation. We will release our code and model for further use by the community.

Figures

Figures reproduced from arXiv: 2507.19766 by the authors.

Figure 1
Figure 1. Illustration of a sample with segments from multiple models. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Training Dynamics of Different Importance Sampling Approaches. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Entropy changing direction of updating MPTs (left) and non-MPTs (right), where blue block [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Training dynamics of RL with masking MPTs. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The entropy dynamics of DMMPTs. time. This observation supports our hypothesis that the overtraining of MPTs is a key factor contributing to the reduction in entropy. By excluding MPTs, the model maintains a more diverse output distribution. 3.2.2 Dynamic Mask Well-mas…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training

    cs.DC 2026-01 conditional novelty 6.0 of 10

    StaleFlow enforces per-trajectory staleness bounds and coordinates rollout data routing, achieving 1.42–2.68x higher RL post-training throughput with comparable convergence.

  2. Hide and Seek with LLMs: An Adversarial Game for Sneaky Error Generation and Self-Improving Diagnosis

    cs.AI 2025-08 conditional novelty 6.0 of 10

    An adversarial training framework pairs a sneaky error generator with an error-diagnosing model, improving stealth and diagnosis on math problems.

Reference graph

Works this paper leans on

18 extracted references · 16 linked inside Pith · cited by 2 Pith papers

  1. [2]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

  2. [3]

    On-policy rl with optimal reward baseline

    Yaru Hao, Li Dong, Xun Wu, Shaohan Huang, Zewen Chi, and Furu Wei. On-policy rl with optimal reward baseline. arXiv preprint arXiv:2505.23585,

    arXiv

  3. [4]

    Skywork open reasoner 1 technical report

    Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner 1 technical report. arXiv preprint arXiv:2505.22312,

    arXiv

  4. [5]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720,

    arXiv

  5. [6]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole

    Notion Blog. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071,

    arXiv

  6. [10]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

    arXiv

  7. [12]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599,

    arXiv

  8. [13]

    Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939,

    arXiv

Show all 18 references
  1. [14]

    Confucius3-math: A lightweight high- performance reasoning llm for chinese k-12 mathematics learning

    Lixin Wu, Na Cai, Qiao Cheng, Jiachen Wang, and Yitao Duan. Confucius3-math: A lightweight high- performance reasoning llm for chinese k-12 mathematics learning. arXiv preprint arXiv:2506.18330 ,

    arXiv

  2. [15]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388,

    arXiv

  3. [16]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476,

    arXiv

  4. [18]

    The surprising effectiveness of negative reinforcement in llm reasoning

    Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The surprising effectiveness of negative reinforcement in llm reasoning. arXiv preprint arXiv:2506.01347,

  5. [2015]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  6. [2017]

    ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al. Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914,

  7. [2018]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240,

    arXiv

  8. [2023]

    High-dimensional continuous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438,

    arXiv

  9. [2024]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256,

    arXiv

  10. [2025]

    Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei

    URL https://hkunlp.github.io/blog/2025/Polaris. Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506.14758,

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.