Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

GradAlign's central claim: the best RL training problems are those whose policy gradients align with the gradient of a small trusted validation set, and reselecting by this alignment each round stabilizes training and improves final test ac

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:00 UTC pith:6VISOFI2

load-bearing objection Plausible and worth a look, but the paper overclaims 'consistently outperforms' and the central theorem has a real gap; this is a revise-and-resubmit, not a straight accept. the 3 major comments →

arxiv 2602.21492 v2 pith:6VISOFI2 submitted 2026-02-25 cs.LG cs.AIcs.CL

GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning

Ningyuan Yang , Weihua Du , Weiwei Sun , Sean Welleck , Yiming Yang This is my paper
classification cs.LG cs.AIcs.CL
keywords reinforcement learningdata selectionLLM post-trainingpolicy gradientgradient alignmentGRPOadaptive curriculumreward noise
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

GradAlign argues that in LLM reinforcement learning, a training problem is valuable only to the degree that updating on it improves a small trusted validation set. Since directly measuring that improvement is prohibitive, the paper uses the policy gradient as a first-order surrogate: it averages validation gradients to form a target direction, then scores each candidate problem by the cosine similarity between its gradient and that target. Each round the selected top fraction is used for GRPO training and the target is recomputed, giving an adaptive curriculum that tracks the evolving policy. Across three regimes—unreliable rewards, imbalanced domains, and low-utility corpora—GradAlign reports consistently higher final accuracy and more stable training than accuracy- and diversity-based baselines. The central object is the direction of policy gradients rather than scalar accuracy or difficulty.

Core claim

GradAlign's central claim is that a training problem is valuable for LLM RL only if updating the policy on it improves a small, trusted validation set that represents the downstream task. Because measuring that improvement directly is too expensive, the paper approximates it with a first-order gradient surrogate: the average GRPO policy gradient computed on the validation set is treated as the target direction, and each candidate training problem is scored by the cosine similarity between its own policy gradient and that target. The paper proves that, under on-policy sampling with binary rewards and unbiased advantage estimates, the GRPO gradient is an unbiased estimator of the expected-accu

What carries the argument

The load-bearing object is the validation target direction Gv: the average, per-round GRPO policy gradient over a small trusted validation set, recomputed on-policy at the start of each selection round. Each candidate problem gets a cheaper gradient estimate, and the selection score is the cosine similarity between that estimate and Gv. Two theorems legitimise the direction: Theorem 4.1 shows the GRPO gradient is an unbiased estimator of the expected-accuracy gradient under on-policy sampling, binary rewards, unbiased advantages, and no clipping or KL regularization; Theorem 4.2 shows advantage normalization preserves direction while changing magnitude, which is why cosine similarity is used

Load-bearing premise

The entire selection rests on the premise that the average policy gradient of a small validation set points toward updates that genuinely raise held-out test accuracy, and that cosine similarity to that direction remains a reliable ranking signal as the policy changes; if the validation gradient is misrepresentative or the gradient estimates are too noisy, the ranking loses meaning.

What would settle it

On a fixed policy at any round, compute GradAlign's ranking of candidates and compare it to the candidates' true measured validation improvement after a single GRPO update: if the Spearman correlation is near zero (say below 0.1) on a clean pool, the first-order surrogate has failed. A second direct check: draw the validation set from a distribution deliberately shifted from the test set; if GradAlign then performs no better than random selection on test accuracy, the method is relying on the validation set being representative rather than on any invariant property of gradient direction.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • With a small trusted validation set, RL practitioners can automatically filter noisy, mislabeled, or low-utility candidate pools without manual curation or per-sample retraining.
  • Accuracy-based heuristics such as keeping 50%-pass-rate problems actively select corrupted data; gradient alignment avoids that failure mode.
  • Because the target direction is recomputed each round, the curriculum adapts as the policy changes, which the paper shows yields more stable training curves than static selection.
  • The method transfers across target tasks and model scales, working for competition-math validation sets, domain-specific games, and broad reasoning benchmarks.
  • The method only requires on-policy gradient estimates, so it can be combined with on-policy RL objectives other than GRPO.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not explore is applying the same cosine-alignment score to individual rollouts or tokens rather than whole problems, which could prune low-utility steps inside long reasoning chains.
  • The per-round validation gradient could be reused as a cheap distribution-mismatch diagnostic: if most candidates have near-zero alignment with the target direction, the pool is off-target before any RL budget is spent.
  • The reported ranking-estimate correlations at different rollout budgets (0.30 at 32 samples, 0.79 at 512) imply a compute-quality frontier; spending extra rollout budget on candidate scoring may or may not beat spending the same compute on additional RL steps, and that trade-off is testable.
  • The directional-alignment logic suggests a harder test: if rewards become non-binary or the rollout distribution goes off-policy, selection quality should degrade in a predictable way; measuring that degradation would bound when the first-order surrogate stops being useful.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces GradAlign, an online data-selection method for LLM reinforcement learning that uses a small validation set to define a target policy-gradient direction and scores each candidate training problem by the cosine similarity between its GRPO policy gradient and that target direction. The method is evaluated in three regimes: noisy rewards, distribution imbalance, and low-utility large web corpora, against Random, Accuracy-Greedy, and LearnAlign baselines, with claims of 'consistently outperforming' all baselines. A theoretical argument (Theorems 4.1 and 4.2) is given to justify using the GRPO gradient as a surrogate for expected accuracy improvement.

Significance. Gradient-aligned data selection is a timely and plausible extension of SFT-style validation-guided data curation to RL, and the paper's focus on non-stationarity of the policy is well motivated. If the method genuinely improves RL training under noisy rewards and distribution shift, this would be a useful contribution. The authors also release code. However, as written, the central empirical claim is not supported by the paper's own tables, and the theoretical justification contains a non-trivial technical error; the paper's significance therefore remains unestablished.

major comments (3)
  1. [Section 4.2, proof of Theorem 4.1] The proof claims that GRPO subtracts 'the expected return baseline that is independent of the sampled action.' This is false: the GRPO advantage in Eq. (3.1) uses the group mean \bar r = (1/k)\sum_j r_j, which includes the reward of the action being scored. Conditional on a sampled action y_i, E[\hat A_i | x, y_i] = (k-1)/k (Q(x, y_i) - \bar Q(x)), not Q(x,y_i) - b(x) with b independent of y_i. Thus the estimator is not unbiased; it is scaled by (k-1)/k. While a positive scaling would preserve direction, the proof's statement is still incorrect. Similarly, Theorem 4.2's proof treats \mu and \sigma as constants when they are random functions of the sampled batch, so the 'directional in expectation' claim does not follow from the given argument. This needs to be corrected or the theoretical claims must be substantially weakened.
  2. [Section 6.3, Tables 1 and 5; abstract/conclusion] The abstract and conclusion claim GradAlign 'consistently outperforms existing baselines,' but the paper's own tables show multiple exceptions. In Table 5, GradAlign is not best on SuperGPQA-test@50 (31.5 vs Align 35.3 and AccGreedy 33.2), AIME2425-test@50 (17.5 vs AccGreedy 18.8), and MMLU-Pro-test@50 (56.0 vs Align 56.3). In Table 1, AMC23-test@50 is 59.5 vs Align 60.6. No error bars, multiple seeds, or significance tests are reported anywhere, so even the favorable average gaps (e.g., 45.7 vs 45.3 in Table 5) could be noise. Since the paper's main contribution is the empirical claim, this is a load-bearing deficiency: the claim 'consistently outperforms' is an overstatement and the evidence supports at most 'competitive on average.' The authors must provide multi-seed results with error bars or statistical tests and reconcile or remove the contradicting splits.
  3. [Section 7, Table 9 vs Table 10] The gradient-noise ablation reports Pearson correlations of 0.30, 0.49, and 0.79 for k_v = 32, 128, and 512, and concludes that a 'moderate correlation (e.g., 0.49)' is sufficient. However, the actual experiments use k_v = 16 for most settings (Table 10), which is below the smallest ablated value; k_v = 64 is used only in the noisy-rewards setting. At k_v = 32 the correlation is already only 0.30, and k_v = 16 is likely noisier. This undermines the reliability of the alignment scores used in the actual experimental configuration. The paper should report the correlation at the deployed k_v values and analyze the effect on selection stability.
minor comments (4)
  1. [Section 3.1, Eq. (3.1)] The normalization expression for \hat A_j is missing parentheses; it should be (r_j - \bar r) / sqrt((1/k)\sum_j (r_j - \bar r)^2 + \epsilon). This makes the definition ambiguous.
  2. [Section 4.3 and Table 10] The algorithm uses k_r rollouts for candidate problems, but no value for k_r appears in the hyperparameter table or in the experimental setup. Reporting the exact rollout counts and selection-ratio choices (q, k_v, k_r) is necessary for reproducibility.
  3. [Section 3, first paragraph] The problem formulation says candidates may be 'unlabeled' but later says every problem is associated with a reference answer and a judge binary reward. Clarify whether unlabeled candidates are actually used in the experiments.
  4. [Section 5.2] The claim that gradient computation adds only 'a constant-factor overhead compared to accuracy-based filtering' is unsubstantiated; scoring all M candidates requires rollouts and backpropagation per candidate each round. Report wall-clock or FLOP overhead, or at least discuss the trade-off more concretely.

Circularity Check

0 steps flagged

No significant circularity: GradAlign's validation-based target is legitimate, and the paper's main claims are evaluated on held-out test sets rather than the data used for selection.

full rationale

The derivation chain in GradAlign does not reduce to its own inputs. The selection criterion is the cosine similarity between a candidate training problem's policy gradient and the average validation gradient (Section 4.3, Algorithm 1), but the paper explicitly holds out test sets for evaluation: 'Pt is used only for the final evaluation' (Section 3). The central theoretical justification, Theorem 4.1, derives the GRPO gradient as an unbiased estimator of the expected-accuracy gradient from the standard policy gradient theorem, citing Sutton et al. 1999; it is not an imported self-citation. Theorem 4.2 is a stated approximation about advantage normalization, and the paper tests the resulting design choice (cosine vs inner product) in Section 7. The paper also runs a direct-training-on-validation ablation and reports that it 'yields higher in-domain performance but does not generalize to the test sets' (Section 7, Table 8), which directly counters the concern that validation-guided selection is merely memorizing the validation set. The only self-citations (e.g., Sun et al. 2025a for gradient alignment in instruction tuning, Sun et al. 2025b and Du et al. 2025 in related work/data context) are not load-bearing; they support background claims and are not used to justify the method's correctness. The empirical overstatement about 'consistently outperforms' and the lack of error bars are correctness/evidence concerns, not circularity. No fitted parameter is renamed as a prediction, and no equation is equivalent to its input by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central method rests on standard policy-gradient theory, a first-order Taylor approximation, the representative-ness of the validation set, and the (incorrectly asserted) unbiasedness of GRPO advantages. Hyperparameters q and k_v are hand-chosen and directly affect selection behavior; k_r is left unspecified. No new physical or conceptual entities are introduced.

free parameters (4)
  • selection ratio q = 4 or 20 (per scenario)
    Determines the fraction of candidates selected (1/q). Chosen by hand per experiment; directly controls the curriculum and affects all results.
  • validation rollouts k_v = 16 or 64 (Table 10)
    Number of rollouts per validation problem for gradient estimation. Ablation (Table 9) shows cosine similarity reliability varies with k_v, so this is a hand-chosen hyperparameter the method's performance depends on.
  • difficulty filter thresholds = pass rates in [0.2, 0.8]
    Pre-filter applied in Scenario 3 before RL selection (Section 6.1). Hand-chosen; changes the candidate pool for all methods but interacts with the selection behavior.
  • candidate rollout count k_r = not specified
    Algorithm 1 uses k_r rollouts per candidate (stated k_r < k_v), but no concrete value is given in Table 10; this unstated parameter affects the gradient estimates used for ranking.
axioms (5)
  • standard math Policy gradient theorem (Sutton et al. 1999)
    Used in Theorem 4.1 to express the expected-accuracy gradient as an expectation over policy gradients. Standard and accepted.
  • domain assumption First-order Taylor approximation of validation improvement
    Assumes validation loss change is proportional to the inner product of validation and training gradients (Section 4.2). This is a heuristic; second-order effects and the nonlinearity of RL updates are ignored.
  • domain assumption On-policy sampling and binary reward with reference answer
    Theorems and algorithm rely on rewards being binary judge outcomes and on rollouts being generated by the current policy. In practice, GRPO uses clipping and KL regularization, which are ignored in the theory.
  • ad hoc to paper Unbiasedness of GRPO advantage estimator
    Theorem 4.1 assumes an unbiased advantage estimator and claims GRPO satisfies it by subtracting a baseline independent of the sampled action. The group-mean baseline actually depends on all sampled actions, so this axiom is false as stated; it is introduced to make the theorem go through.
  • domain assumption Validation set is representative of the downstream task
    The method assumes gradient direction on a small validation set is a reliable target for held-out test performance. This is stated in Section 3 and is the core premise for transferring selection to test.

pith-pipeline@v1.3.0-alltime-deepseek · 15666 in / 14487 out tokens · 122322 ms · 2026-08-02T21:00:59.887807+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) has become a central post-training paradigm for large language models (LLMs), but its performance is highly sensitive to the quality of training problems. This sensitivity stems from the non-stationarity of RL: rollouts are generated by an evolving policy, and learning is shaped by exploration and reward feedback, unlike supervised fine-tuning (SFT) with fixed trajectories. As a result, prior work often relies on manual curation or simple heuristic filters (e.g., accuracy), which can admit incorrect or low-utility problems. We propose GradAlign, a gradient-aligned data selection method for LLM reinforcement learning that uses a small, trusted validation set to prioritize training problems whose policy gradients align with validation gradients, yielding an adaptive curriculum. We evaluate GradAlign across three challenging data regimes: unreliable reward signals, distribution imbalance, and low-utility training corpus, showing that GradAlign consistently outperforms existing baselines, underscoring the importance of directional gradient signals in navigating non-stationary policy optimization and yielding more stable training and improved final performance. We release our implementation at https://github.com/StigLidu/GradAlign

Figures

Figures reproduced from arXiv: 2602.21492 by Ningyuan Yang, Sean Welleck, Weihua Du, Weiwei Sun, Yiming Yang.

Figure 1
Figure 1. Figure 1: Overview of GradAlign. GradAlign uses a small validation set to estimate a coherent target gradient direction and scores large-scale training candidates by gradient alignment, selecting the top-ranked fraction to form an adaptive RL online-learning curriculum. model computes a binary reward indicating whether the generated output matches the reference answer. Because the candidate pool may be highly noisy,… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of three challenging data-selection scenarios. Each panel shows a failure mode where accuracy-based filtering fails to identify training samples that improve downstream performance. problems whose gradients align with the validation gradi￾ent, while downweighting target-irrelevant ones with low alignment. Low-Utility Training Data As RL training scales up, practitioners increasingly rely on la… view at source ↗
Figure 3
Figure 3. Figure 3: Training Accuracy Curve on AIME2425, AMC22 and AMC23 (Scenario 1). GradAlign (ours) achieves the strongest performance. B. Candidate Distribution We visualize the score distribution for Scenario 1 (Noisy Rewards), in which half of the training candidates are corrupted by random reward signals sampled from a Bernoulli distribution with p = 0.5. An effective data selection method should be able to distinguis… view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of cosine similarity and inner product similarity. Cosine similarity is more indicative than the inner product regarding detecting corrupted instances. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Let the Target Select for Itself: Data Selection via Target-Aligned Paths

    cs.LG 2026-05 unverdicted novelty 6.0

    Target-aligned data selection via normalized endpoint loss drop on a validation-induced reference path achieves competitive performance with reduced computational overhead.

Reference graph

Works this paper leans on

31 extracted references · 28 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Abbas, A., Tirumala, K., Simig, D., Ganguli, S., and Mor- cos, A. S. Semdedup: Data-efficient learning at web-scale through semantic deduplication.ArXiv, abs/2303.09540,

    Pith/arXiv arXiv

  2. [4]

    Dodge, J., Sap, M., Marasovi ´c, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., and Gardner, M. Documenting large webtext corpora: A case study 9 GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning on the colossal clean crawled corpus.arXiv preprint arXiv:2104.08758,

    Pith/arXiv arXiv

  3. [6]

    Dsdm: Model-aware dataset selection with datamodels.ArXiv, abs/2401.12926,

    Engstrom, L., Feldmann, A., and Madry, A. Dsdm: Model-aware dataset selection with datamodels.ArXiv, abs/2401.12926,

    Pith/arXiv arXiv

  4. [9]

    Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning.arXiv preprint arXiv:2504.11456,

    He, Z., Liang, T., Xu, J., Liu, Q., Chen, X., Wang, Y ., Song, L., Yu, D., Liang, Z., Wang, W., et al. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning.arXiv preprint arXiv:2504.11456,

    Pith/arXiv arXiv

  5. [10]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model

    Hu, J., Zhang, Y ., Han, Q., Jiang, D., Zhang, X., and Shum, H.-Y . Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290,

    Pith/arXiv arXiv

  6. [11]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    Pith/arXiv arXiv 2001

  7. [14]

    Learnalign: Reasoning data selec- tion for reinforcement learning in large language models based on improved gradient alignment.arXiv preprint arXiv:2506.11480, 2025a

    Li, S., Li, S., Yang, Z., Zhang, X., Chen, G., Xia, X., Liu, H., and Peng, Z. Learnalign: Reasoning data selec- tion for reinforcement learning in large language models based on improved gradient alignment.arXiv preprint arXiv:2506.11480, 2025a. Li, X., Zou, H., and Liu, P. Limr: Less is more for rl scaling. arXiv preprint arXiv:2502.11886, 2025b. Liang, ...

    Pith/arXiv arXiv

  8. [15]

    Let’s verify step by step.ArXiv, abs/2305.20050,

    Lightman, H., Kosaraju, V ., Burda, Y ., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step.ArXiv, abs/2305.20050,

    Pith/arXiv arXiv

  9. [16]

    Rho-1: Not all tokens are what you need.ArXiv, abs/2404.07965,

    Lin, Z.-W., Gou, Z., Gong, Y ., Liu, X., Shen, Y ., Xu, R., Lin, C., Yang, Y ., Jiao, J., Duan, N., and Chen, W. Rho-1: Not all tokens are what you need.ArXiv, abs/2404.07965,

    Pith/arXiv arXiv

  10. [17]

    Regmix: Data mixture as regression for language model pre-training.ArXiv, abs/2407.01492,

    Liu, Q., Zheng, X., Muennighoff, N., Zeng, G., Dou, L., Pang, T., Jiang, J., and Lin, M. Regmix: Data mixture as regression for language model pre-training.ArXiv, abs/2407.01492,

    Pith/arXiv arXiv

  11. [19]

    Mathematical Association of America

    URL https: //arxiv.org/abs/2505.14652. Mathematical Association of America. American Mathe- matics Competitions 2022,

    Pith/arXiv arXiv 2022

  12. [20]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. ArXiv, abs/1707.06347,

    Pith/arXiv arXiv

  13. [22]

    Enhancing training data attribution with representational optimization.ArXiv, abs/2505.18513, 2025a

    Sun, W., Liu, H., Kandpal, N., Raffel, C., and Yang, Y . Enhancing training data attribution with representational optimization.ArXiv, abs/2505.18513, 2025a. Sun, W., Zhou, X., Du, W., Wang, X., Welleck, S., Neubig, G., Sap, M., and Yang, Y . Training proactive and per- sonalized llm agents.arXiv preprint arXiv:2511.02208, 2025b. Sun, Y ., Shen, J., Wang,...

    arXiv

  14. [23]

    Wang, X., Zhou, W., Zhang, Q., Zhou, J., Gao, S., Wang, J., Zhang, M., Gao, X., Chen, Y ., and Gui, T

    URL https://arxiv.org/abs/2502.14739. Wang, X., Zhou, W., Zhang, Q., Zhou, J., Gao, S., Wang, J., Zhang, M., Gao, X., Chen, Y ., and Gui, T. Farewell to aimless large-scale pretraining: Influential subset se- lection for language model. InAnnual Meeting of the Association for Computational Linguistics,

    Pith/arXiv arXiv

  15. [24]

    Mmlu-pro: A more robust and challenging multi-task language under- standing benchmark.arXiv preprint arXiv:2406.01574,

    Wang, Y ., Ma, X., Zhang, G., Ni, Y ., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. Mmlu-pro: A more robust and challenging multi-task language under- standing benchmark.arXiv preprint arXiv:2406.01574,

    Pith/arXiv arXiv

  16. [25]

    Organize the web: Constructing domains en- hances pre-training data curation.ArXiv, abs/2502.10341,

    Wettig, A., Lo, K., Min, S., Hajishirzi, H., Chen, D., and Soldaini, L. Organize the web: Constructing domains en- hances pre-training data curation.ArXiv, abs/2502.10341,

    Pith/arXiv arXiv

  17. [26]

    Less: Selecting influential data for targeted instruction tuning.ArXiv, abs/2402.04333,

    Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. Less: Selecting influential data for targeted instruction tuning.ArXiv, abs/2402.04333,

    Pith/arXiv arXiv

  18. [27]

    M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y ., Liang, P., Le, Q

    Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y ., Liang, P., Le, Q. V ., Ma, T., and Yu, A. W. Doremi: Optimizing data mixtures speeds up language model pre- training.ArXiv, abs/2305.10429, 2023a. Xie, S. M., Santurkar, S., Ma, T., and Liang, P. Data se- lection for language models via importance resampling. ArXiv, abs/2302.03169, 2023b. Yang, J.,...

    Pith/arXiv arXiv

  19. [28]

    Data mixing laws: Optimizing data mixtures by predicting language modeling performance.ArXiv, abs/2403.16952,

    Ye, J., Liu, P., Sun, T., Zhou, Y ., Zhan, J., and Qiu, X. Data mixing laws: Optimizing data mixtures by predicting language modeling performance.ArXiv, abs/2403.16952,

    Pith/arXiv arXiv

  20. [29]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Yu, Q., Zhang, Z., Zhu, R., Yuan, Y ., Zuo, X., Yue, Y ., Dai, W., Fan, T., Liu, G., Liu, L., et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  21. [30]

    Mates: Model-aware data selection for efficient pretraining with data influence mod- els.ArXiv, abs/2406.06046,

    Yu, Z., Das, S., and Xiong, C. Mates: Model-aware data selection for efficient pretraining with data influence mod- els.ArXiv, abs/2406.06046,

    Pith/arXiv arXiv

  22. [31]

    Harnessing diversity for important data selection in pretraining large language models.ArXiv, abs/2409.16986,

    11 GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning Zhang, C., Zhong, H., Zhang, K., Chai, C., Wang, R., Zhuang, X., Bai, T., Qiu, J., Cao, L., Yuan, Y ., Wang, G., and He, C. Harnessing diversity for important data selection in pretraining large language models.ArXiv, abs/2409.16986,

    Pith/arXiv arXiv

  23. [32]

    Data-efficient rlvr via off-policy influence guidance.arXiv preprint arXiv:2510.26491,

    Zhu, E., Jiang, D., Wang, Y ., Li, X., Cheng, J., Gu, Y ., Niu, Y ., Zeng, A., Tang, J., Huang, M., et al. Data-efficient rlvr via off-policy influence guidance.arXiv preprint arXiv:2510.26491,

    Pith/arXiv arXiv

  24. [2017]

    Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  25. [2019]

    Y ., Bansal, H., Guha, E

    Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S. Y ., Bansal, H., Guha, E. K., Keh, S. S., Arora, K., Garg, S., Xin, R., Muennighoff, N., Heckel, R., Mercat, J.-P., Chen, M., Gururangan, S., Wortsman, M., Albalak, A., Bitton, Y ., Nezhurina, M., Abbas, A., Hsieh, C.-Y ., Ghosh, D., Gardner, J., Kilian, M., Zhang, H., Shao, R., Pratt, S., San...

    Pith/arXiv arXiv

  26. [2020]

    Understanding the effects of rlhf on llm generalisation and diversity

    Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Ham- bro, E., Grefenstette, E., and Raileanu, R. Understanding the effects of rlhf on llm generalisation and diversity. arXiv preprint arXiv:2310.06452,

    Pith/arXiv arXiv

  27. [2021]

    Generalizable end-to- end tool-use rl with synthetic codegym.arXiv preprint arXiv:2509.17325,

    Du, W., Gong, H., Ling, Z., Liu, K., Shen, L., Yao, X., Xu, Y ., Shi, D., Yang, Y ., and Chen, J. Generalizable end-to- end tool-use rl with synthetic codegym.arXiv preprint arXiv:2509.17325,

    arXiv

  28. [2022]

    Grosse, R. B., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., Hubinger, E., Lukovsiut.e, K., Nguyen, K., Joseph, N., McCandlish, S., Kaplan, J., and Bowman, S. Studying large language model generalization with influence func- tions.ArXiv, abs/2308.03296,

    Pith/arXiv arXiv

  29. [2023]

    Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models

    Albalak, A., Phung, D., Lile, N., Rafailov, R., Gandhi, K., Castricato, L., Singh, A., Blagden, C., Xiang, V ., Mahan, D., et al. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models. arXiv preprint arXiv:2502.17387,

    Pith/arXiv arXiv

  30. [2024]

    Gandhi, K., Lee, D., Grand, G., Liu, M., Cheng, W., Sharma, A., and Goodman, N. D. Stream of search (sos): Learning to search in language.arXiv preprint arXiv:2404.03683,

    Pith/arXiv arXiv

  31. [2025]

    Theoremqa: A theorem-driven question answering dataset

    Chen, W., Yin, M., Ku, M., Lu, P., Wan, Y ., Ma, X., Xu, J., Wang, X., and Xia, T. Theoremqa: A theorem-driven question answering dataset. InThe 2023 Conference on Empirical Methods in Natural Language Processing,

    2023