Pith. sign in

REVIEW 4 major objections 6 minor 21 references

Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that RLVR training for reasoning models can match full-data performance while using only 20% of the data and roughly 40% of the rollouts, by selecting samples offline for diversity, influence, and difficulty, and then prun

desk verdict Useful, clearly-specified RLVR data-efficiency pipeline that likely works, but the headline speed-ups rest on test-set checkpoint selection and some quietly tuned hyperparameters, so treat the numbers as upper bounds. read the letter →

arxiv 2509.01321 v1 pith:WL63WTSC submitted 2025-09-01 cs.LG cs.CL

Xinyu Tang , Zhenduo Zhang , Yurou Liu , Wayne Xin Zhao , Zujie Wen , Zhiqiang Zhang , Jun Zhou This is my paper
classification cs.LGcs.CL
keywords reinforcementlearningverifiablerewarddataefficiencyselectionrolloutpruningdeterminantalpointprocessexplorabilityreasoningLLMs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DEPO is a two-stage data-selection pipeline for reinforcement learning with verifiable rewards (RLVR). The paper tries to show that most RLVR training data and most rollouts are redundant: a subset chosen offline for diversity, influence, and moderate difficulty can replace the full dataset, and an online explorability filter can decide which samples deserve expensive rollouts. The reported result is that with only 20% of the training data, DEPO reaches an average accuracy of 61.1 across five reasoning benchmarks, essentially matching the 61.7 of full-data GRPO, while using 57% of the training time and 40% of the rollouts; on AIME24 and AIME25 this is a 1.85x and 1.66x speed-up. If this is right, RLVR can be made substantially cheaper without changing the underlying policy-gradient algorithm, simply by being more deliberate about what gets trained on and rolled out.

What carries the argument

The load-bearing object is the weighted DPP kernel L_Y = diag(w^{1/2}_Y) S_Y diag(w^{1/2}_Y), whose determinant combines diversity (volume spanned by sample embeddings) with influence (PageRank weights); greedy MAP inference on this kernel produces the pruned offline subset. The second mechanism is the windowed explorability metric E, which sums reward-weighted rollout entropy over recent epochs, thresholds out pathological negative rollouts, and is used both to select which samples get rollouts and to identify under-explored samples for replay.

What would settle it

Replace the last-token embedding similarity matrix in the offline stage with random features (or a bag-of-words representation) while keeping the rest of DEPO fixed. If AIME24/AIME25 accuracy stays near 62.8/50.9, the graph construction is not load-bearing; if accuracy drops substantially, the representation assumption is confirmed. A second check: select the offline subset with one base model and train with a different base model; if the speed-up disappears, the subset is not transferable.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that data efficiency in RLVR is not a property of any single selection heuristic but of combining three complementary filters. Offline, DEPO builds a similarity graph over samples using last-token final-layer embeddings, prunes it with a PageRank-weighted determinantal point process that rewards both diversity and influence, and then samples problems whose rollout accuracy follows a normal distribution, keeping problems that are neither trivially easy nor hopelessly hard. Online, it defines a sample-level explorability score that averages reward-weighted, entropy-gated signals over a sliding window of recent epochs, rolls out only the top fr

Load-bearing premise

The offline stage assumes that the similarity graph built from last-token final-layer embeddings of the base model predicts which samples will be valuable during RLVR training; the paper does not ablate this choice.

Editorial extensions

If this is right

  • RLVR training on a fixed compute budget can be redirected from generating redundant rollouts to training on the samples that actually change the policy, so larger models or longer training become feasible at the same cost.
  • Offline data selection no longer requires warmup training on the full dataset or full-dataset rollouts, so the selection step itself is cheap enough to run as a standard preprocessing stage.
  • The 20% subset plus explorability pruning yields near-full-data accuracy, implying that typical RLVR datasets contain a large amount of low-value redundancy that can be removed a priori.
  • The method transfers across at least three different base models and three RLVR dataset sizes (5k, 17k, and 30k samples), so the selection criteria are not tied to one model or dataset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not ablate the embedding choice for the sample graph; a natural extension would be to test whether cheaper representations (e.g., bag-of-words or random features) preserve the offline selection gain, which would reveal whether the graph is carrying the signal or merely acting as a randomizer.
  • The explorability metric depends on rollout entropy; in domains where correct reasoning is naturally low-entropy (e.g., short code snippets or terse proofs), the filter may systematically under-rollout high-quality solutions, so the metric may need domain-specific tuning.
  • Because the ablation shows that removing difficulty-aware sampling hurts most, the offline stage may be doing the heavy lifting; a testable extension is to run online explorability pruning on the full dataset alone and measure how much of the total speed-up it recovers.
  • The plateau at 20% sampling ratio suggests diminishing returns; an interesting extension is to make the sampling ratio adaptive per epoch, increasing data when explorability stays high.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes DEPO, a two-stage data-selection pipeline for reinforcement learning with verifiable rewards (RLVR). Offline, DEPO builds a similarity graph from last-token embeddings of the base model, prunes the dataset with a PageRank-weighted determinantal point process to promote diversity/influence, and then samples a subset whose difficulty approximates a normal distribution. Online, DEPO computes a sample-level "explorability" metric from recent rollout dynamics, skips rollouts for low-explorability samples, and replays under-explored samples. Experiments on five reasoning benchmarks with three 7B/8B models report that using 20% of the training data and about 40% of rollouts yields accuracy comparable to full-dataset GRPO, with claimed 1.85x and 1.66x time-to-target speed-ups on AIME24 and AIME25.

Significance. If the empirical claims hold, DEPO would be a practically valuable contribution to a real bottleneck in RLVR training. The paper's strengths include component-level ablations (Table 2), evaluation across three base models and three RLVR training datasets (Figure 4), and a clear motivating problem. The two-stage combination of offline curation and online rollout pruning is reasonable and addresses a cost that dominates RLVR. However, the current evaluation protocol does not support the headline numbers: the reported results and speed-ups rely on oracle checkpoint selection on the five test benchmarks, and the method's hyperparameters are apparently tuned on the same test curves. These issues are load-bearing for the central claims and need to be fixed before the paper can be assessed fairly.

major comments (4)
  1. [Section 3.1, Appendix B, Table 1, Figure 1] The evaluation protocol uses oracle checkpoint selection on the test benchmarks: Section 3.1 and Appendix B state that models are evaluated every 50 steps and the checkpoint with the best average performance on the five benchmarks is reported. Thus Table 1's headline comparison (e.g., DEPO 61.1 vs GRPO 61.7; DEPO-Offline 61.4 vs 61.7) and the 1.85x/1.66x speed-ups in Figure 1 are upper envelopes over training checkpoints, not the output of a deployable early-stopping or validation rule. In addition, Figures 6-8 are the same test-benchmark accuracy curves used to select hyperparameters (mu=0.5, sigma=0.2, lambda=1.5, w=5, d=0.05, rho=0.05) reported in Appendix B. This is selection-on-the-test-set for both checkpoints and hyperparameters. Please re-evaluate with a held-out validation split for checkpoint selection (or report final-checkpoint performance) and with hyperparameters chosen wit
  2. [Equation (3) and Appendix B] All entries are single-run; there are no error bars, multiple seeds, or significance tests. Differences of 0.1-0.6 points (e.g., AIME24 62.8 vs 63.4, average 61.1 vs 61.7; Llama-8B average 57.3 vs 58.2) are within typical evaluation noise, especially with temperature=1 and 32 repeats. The claims "consistently outperforms" and "comparable performance" are not statistically grounded. Please report mean and standard deviation over at least three independent training runs, or provide bootstrap confidence intervals for the main comparisons.
  3. [Section 2.2.3 and Appendix B] Equation (3) defines mu and sigma as the mean and standard deviation of the accuracies in the final selected subset, but Appendix B sets mu=0.5 and sigma=0.2 as fixed values. This is internally contradictory: either the normal distribution is fitted from the data, in which case mu and sigma are not free hyperparameters, or it is controlled by the user, in which case the text misdescribes the method. The distinction matters because Table 2 shows difficulty-aware sampling is the largest offline ablation component (AIME24 drops from 62.8 to 60.3 when removed). Please clarify the actual procedure and the role of these parameters.
  4. [Section 2.2.1 and Table 2] The dynamic replay mechanism is described as sampling the rho% of samples that have been "least explored throughout the training," but Algorithm 2 computes B_Replay from the current raw batch B based on the smallest number of historical rollouts within that batch. These are not the same: a globally under-explored sample that does not appear in the current batch would never be replayed. The implementation should be made precise, and the indicator I in Eq. (5) should be reconciled with the algorithm's batching (the formula places I inside the expectation over the full batch, while the algorithm forms a pruned batch before rollout).
minor comments (6)
  1. [Figure 1 caption] The caption says "reduces training time by up to 1.6 times," while the abstract and main text claim 1.85x and 1.66x speed-ups. Please clarify whether these are speed-ups or time reductions, and reconcile the numbers.
  2. [Table 1] Row labels are inconsistent: the Qwen2.5-7B-Math block uses "w Online+ Random" while the earlier blocks use "+ Random". Please standardize the notation.
  3. [Eq. (4)] The symbol E is used for both the explorability score and token entropy, which is confusing. Also, E(o_i^+) is said to be the "mean entropy of all positive samples," but it is not defined whether this is within the current group, current batch, or the whole dataset.
  4. [Algorithm 1] The input line has a typo: "n.k" should be "n, k".
  5. [Section 2.2.1] No details are given for graph construction: how are edges derived from the embeddings (threshold? kNN?), and what is the PageRank restart parameter? Without this, the offline pruning step is not reproducible.
  6. [Table 1 efficiency columns] Please specify whether "Time" includes the offline rollout cost for difficulty scoring and the graph-construction cost. As written, DEPO's reported 57% of training time may undercount its total compute relative to GRPO's 100%.

Circularity Check

2 steps flagged · score 5.0 of 10

Headline speed-ups rely on oracle best-checkpoint selection on the test benchmarks and on hyperparameters tuned on the same five benchmarks; the offline/online selection method itself is not definitionally circular.

  1. fitted input called prediction [Section 3.1 and Appendix B (evaluation protocol)]
    "We follow Zheng et al. (2025) to evaluate models on these benchmarks every 50 steps and report the performance of the checkpoint that obtains the best average performance on five benchmarks."

    Table 1 and Figure 1 report accuracies and speed-ups computed from the checkpoint that maximizes the average on the five evaluation benchmarks. The 1.85x/1.66x speed-up is therefore measured from an oracle best-test-checkpoint, not from a fixed, deployable training budget or a validation-based early-stopping rule. For any method with a noisy or non-monotone accuracy curve, this rule can manufacture speed-ups that would not survive a held-out validation split. The claim that DEPO reaches GRPO full-data performance is, by construction, the best test-set value observed during training rather than an out-of-sample prediction.

  2. fitted input called prediction [Section 3.4 / Figures 6-8 and Appendix B (hyperparameter setting)]
    "The mean µ and standard deviation σ of the difficulty distribution for the final selected subset are set to 0.5 and 0.2, respectively. In the online data selection phase, we configure the window size of recent epoch w as 5, the initial sampling rate α0 as 1, the decay rate d as 0.05, the replay sample ratio ρ as 0.05, and the threshold λ for filtering poor negative rollouts as 1.5."

    DEPO's final hyperparameters (mu=0.5, sigma=0.2, lambda=1.5, w=5, d=0.05, rho=0.05) are fixed in Appendix B after analyses in Figures 6-8 that plot average accuracy on the five benchmarks while varying these hyperparameters. The final evaluation is then performed on the same five benchmarks in Table 1. Thus the configuration is fitted to maximize performance on the exact test sets used for reporting, making the reported gains partly a consequence of selection on the test set rather than independent evidence for the data-efficiency claim.

full rationale

The offline and online selection components are not definitionally circular: equations (1)-(5) define the DPP kernel, difficulty-aware sampling, explorability metric, and replay objective from model embeddings, rollout accuracies, entropy, and reward statistics, none of which assumes the benchmark accuracies used for evaluation. The self-citations (e.g., Liu et al. 2024, Tang et al. 2025) justify the representation choice but are not load-bearing. The circularity found is confined to the empirical claim: the headline final accuracies and speed-ups come from best-test-checkpoint selection and hyperparameters tuned on the same five test benchmarks, so the reported numbers are upper envelopes over test-evaluated checkpoints and test-tuned configurations rather than out-of-sample predictions. Because the method components retain independent algorithmic content and could in principle transfer, this is a moderate partial-circularity finding rather than a claim that the whole derivation is self-referential by definition.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or semantic entities. The free-parameter count is high: seven hyperparameters are hand-set or tuned on the evaluation benchmarks, and the two headline efficiency ratios (20% data, 40% rollouts) are also design choices. The main domain assumptions concern the validity of embedding-based similarity graphs and entropy as an exploration signal, both unproven beyond the paper's own ablations.

free parameters (7)
  • mu (difficulty distribution mean) = 0.5
    Fixed in Appendix B, varied in Figure 6b, and selected based on AIME benchmark performance. Eq. 3 describes it as the data mean, but it is a tuned hyperparameter.
  • sigma (difficulty distribution std) = 0.2
    Fixed in Appendix B, varied in Figure 6c, and selected based on AIME benchmark performance.
  • lambda (entropy threshold for negative rollouts) = 1.5
    Tuned in Figure 8a using AIME24 and AIME25.
  • w (explorability window size) = 5
    Tuned in Figure 8b using AIME24 and AIME25.
  • d (decay rate for alpha_e) = 0.05
    Tuned in Figure 7a; final value stated in Appendix B.
  • rho (replay sample ratio) = 0.05
    Tuned in Figure 7b; final value stated in Appendix B.
  • alpha_0 (initial high-explorability sample ratio) = 1.0
    Set in Appendix B; affects the rollout pruning schedule and is not derived from data.
assumptions (5)
  • domain assumption Last-token final-layer embeddings from the base model capture sample characteristics relevant to RL training value.
    Section 2.2.1; no ablation over embedding choice or graph construction; the offline selection relies on it.
  • domain assumption PageRank on the sample similarity graph measures sample influence and representativeness for RLVR.
    Section 2.2.2; no validation that PageRank scores correlate with learning value.
  • domain assumption Samples with mid-range rollout accuracy provide the best learning signal in RLVR.
    Section 2.2.3; supported by ablation (Table 2) but not by a theoretical argument.
  • domain assumption Average token entropy of rollouts, weighted by normalized reward, measures a sample's future exploration potential.
    Section 2.3.1; motivated by prior entropy-based RL work but the specific aggregation is new and not independently validated.
  • standard math Greedy DPP approximation preserves enough diversity for the selection objective.
    Appendix A; standard result from Kulesza & Taskar and Chen et al., used without re-derivation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward." pith.science (2026). https://pith.science/paper/WL63WTSC

@misc{pith2026250901321,
  author       = {Pith},
  title        = {Pith review of: Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WL63WTSC}},
  note         = {Machine review of arXiv:2509.01321}
}
read the original abstract

Recent advances in large reasoning models have leveraged reinforcement learning with verifiable rewards (RLVR) to improve reasoning capabilities. However, scaling these methods typically requires extensive rollout computation and large datasets, leading to high training costs and low data efficiency. To mitigate this issue, we propose DEPO, a Data-Efficient Policy Optimization pipeline that combines optimized strategies for both offline and online data selection. In the offline phase, we curate a high-quality subset of training samples based on diversity, influence, and appropriate difficulty. During online RLVR training, we introduce a sample-level explorability metric to dynamically filter samples with low exploration potential, thereby reducing substantial rollout computational costs. Furthermore, we incorporate a replay mechanism for under-explored samples to ensure adequate training, which enhances the model's final convergence performance. Experiments across five reasoning benchmarks show that DEPO consistently outperforms existing methods in both offline and online data selection scenarios. Notably, using only 20% of the training data, our approach achieves a 1.85 times speed-up on AIME24 and a 1.66 times speed-up on AIME25 compared to GRPO trained on the full dataset.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 7 canonical work pages

  1. [1]

    This objective aims to select a subset Y that is both diverse (as captured by det(SY)) and influential (as promoted by the product of weights ∏i∈Y wi)

    We begin with the original weighted determinant maximization problem: max Y⊆P det(SY) · ∏ i∈Y wi ! , (7) where SY ∈ R|Y|×|Y| denotes the similarity matrix over the subset Y, and wi represents the influential weight (i.e., the PageRank score) of each sample i. This objective aims to select a subset Y that is both diverse (as captured by det(SY)) and influe...

    work page 2025

  2. [3]

    12 DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Dama...

    arXiv

  3. [5]

    Determinantal point processes for machine learning

    Alex Kulesza and Ben Taskar. Determinantal point processes for machine learning. CoRR, abs/1207.6083, 2012a. Alex Kulesza and Ben Taskar. Determinantal point processes for machine learning. Found. Trends Mach. Learn., 5(2-3):123–286, 2012b. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion ...

    arXiv

  4. [8]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. CoRR, abs/2311.12022,

    arXiv

  5. [10]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haoz...

    arXiv

  6. [11]

    Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. CoRR, abs/2506.01939, 2025a. Yip...

    arXiv

  7. [12]

    DAPO: an open-source LLM reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, W...

    arXiv

  8. [13]

    Yue, Lovish Madaan, Ted Moskovitz, DJ Strouse, and Aaditya K

    Albert S. Yue, Lovish Madaan, Ted Moskovitz, DJ Strouse, and Aaditya K. Singh. HARP: A challenging human-annotated math reasoning benchmark. CoRR, abs/2412.08819,

    arXiv

Show all 21 references
  1. [14]

    V APO: efficient and reliable reinforcement learning for advanced reasoning tasks

    14 Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Cheng-Xiang Wang, Tiantian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Z...

    arXiv

  2. [15]

    A survey of large language models

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong W...

    arXiv

  3. [17]

    We train DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B on 64×H200 GPUs, and Qwen2.5-Math-7B on 32×H200 GPUs

    for rollout. We train DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B on 64×H200 GPUs, and Qwen2.5-Math-7B on 32×H200 GPUs. For training datasets, we use the DAPO-Math Yu et al. (2025) as the training dataset. During rollouts, we set the temperature to 1 and sampl...

    2025

  4. [20]

    We follow Zheng et al

    and LiveCodeBench (Jain et al., 2025)) to evaluate the model performance. We follow Zheng et al. (2025) to evaluate models on those benchmarks every 50 steps and report the performance of the checkpoint that obtains the best average performance on five benchmarks. All evaluati...

    2025

  5. [21]

    • Random: Randomly samples data from the training set

    and PPL-Middle (Ankner et al., 2025)), RLVR selection method (i.e., LIMR (Li et al., 2025b) and Learnalign (Li et al., 2025a)). • Random: Randomly samples data from the training set. • PPL-Top (Laurençon et al., 2022): Selects the data with the highest perplexity. • PPL-Middle...

    2025

  6. [256]

    Similar to Yue et al

    We apply the GRPO algorithm to train the model. Similar to Yue et al. (2025), we remove both the KL divergence loss and the entropy loss. We train all models for 1000 steps, and we optimize the actor model using the AdamW (Loshchilov and Hutter,

    2025

  7. [2018]

    Reasoning with exploration: An entropy perspective

    Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. CoRR, abs/2506.14758,

    arXiv

  8. [2019]

    User: \n [question] \n Please reason step by step, and put your final answer within \boxed{}. \n \n Assistant:

    optimizer with a constant learning rate of 2e-6 for DeepSeek-R1-Distill-Qwen-7B and Deepseek-R1-Distill-Llama-8B and 1e-6 for Qwen-Math-7B. The actor module is optimized using Fully Sharded Data Parallel (FSDP) (Zhao et al., 2023b) for efficient distributed training. The chat ...

    2021

  9. [2021]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya K...

    arXiv

  10. [2022]

    Learnalign: Reasoning data selection for reinforcement learning in large language models based on improved gradient alignment

    Shikun Li, Shipeng Li, Zhiqin Yang, Xinghua Zhang, Gaode Chen, Xiaobo Xia, Hengyu Liu, and Zhe Peng. Learnalign: Reasoning data selection for reinforcement learning in large language models based on improved gradient alignment. CoRR, abs/2506.11480, 2025a. Xuefeng Li, Haoyang ...

    arXiv

  11. [2023]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300,

    arXiv

  12. [2024]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. CoRR, abs/2503.20783, 2025a. Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, Ling Pan, Xinyu Hu, Shaopan ...

    arXiv

  13. [2025]

    Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L

    https://hkunlp.github.io/blog/2025/Polaris. Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L. Leavitt, and Mansheej Paul. Perplexed by perplexity: Perplexity-based data pruning with small reference models. In ICLR. OpenReview.net,

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.