Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Gradient-space diversity, measured by G-Vendi, predicts out-of-distribution accuracy at Spearman's $\rho \approx 0.9$, and generating data into sparse gradient regions keeps improving these models past a million synthetic samples.

desk verdict The G-Vendi correlation result is solid and genuinely useful; the Prismatic Synthesis headline is promising but not yet isolated from generator and quality-filter confounds. read the letter →

arxiv 2505.20161 v2 pith:A67KMUY5 submitted 2025-05-26 cs.LG cs.AIcs.CL

Jaehun Jung , Seungju Han , Ximing Lu , Skyler Hallinan , David Acuna , Shrimai Prabhumoye , Mostafa Patwary , Mohammad Shoeybi
show 2 more authors
Bryan Catanzaro Yejin Choi
This is my paper
classification cs.LGcs.AIcs.CL
keywords gradient-spacediversityG-VendisyntheticdatagenerationLLMreasoningout-of-distributiongeneralizationcurationVendiscoresupervisedfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to identify what kind of diversity in training data actually drives generalization in LLM reasoning, beyond surface-level text variation. It claims the right notion is the diversity of the loss gradients that samples induce in a small off-the-shelf model, and proposes G-Vendi—the exponentiated entropy of projected gradient covariance—to measure it. Across more than 300 controlled training runs on math and natural-language-inference tasks, G-Vendi correlates with out-of-distribution benchmark performance (Spearman's $\rho \approx 0.9$) far better than embedding-based, perplexity, or skill-taxonomy measures. Building on that, the paper introduces Prismatic Synthesis, which clusters samples in gradient space and keeps only newly generated data that falls in sparse clusters, so synthetic data keeps improving models instead of saturating. If the paper is right, data diversity becomes a measurable, optimizable quantity that can substitute for scaling up ever-larger data generators.

What carries the argument

The load-bearing object is G-Vendi, the exponentiated entropy of the eigenvalues of a covariance matrix of projected loss gradients: for samples $(x,y)$ it uses $g_\theta(x,y) = -\nabla \log P(y|x;\theta)/\|\nabla \log P(y|x;\theta)\|$, computed with a fixed off-the-shelf proxy model and compressed by a Rademacher random projection (a Johnson-Lindenstrauss transform) to dimension $d=1024$, then aggregated as $\exp(-\sum_i \lambda_i \log \lambda_i)$ over the normalized covariance matrix. The machinery that turns measurement into data is Prismatic Synthesis: K-means clustering of samples in this gradient space, few-shot generation of new samples with a large language model, and rejection of generated samples that land in already-dense clusters, so only the sparsest-cluster samples join the pool.

What would settle it

Generate several data pools with a deliberately different generator—say a base model without instruction tuning, or a much weaker one—while holding quality constant, compute G-Vendi on subsets, train the same students, and check whether Spearman's $\rho$ with out-of-distribution accuracy stays near 0.9; if it collapses below embedding-based baselines, the transfer claim fails. A second check: corrupt labels or solutions in a gradient-diverse subset and see whether high G-Vendi still tracks performance, which would test whether the metric silently depends on the paper's quality-control pipeline.

Watch

Extended reading notes

Core claim

The central claim is that the diversity most relevant to generalization can be read off the geometry of the gradients that training samples produce. Each sample is represented by the normalized gradient of its log-likelihood under a fixed proxy model, projected to 1024 dimensions, and G-Vendi is the Vendi score—the exponentiated Shannon entropy of the eigenvalues of the samples' covariance matrix—of those projections. The paper reports $R^2 \approx 0.8$ and Spearman's $\rho \approx 0.9$ between G-Vendi and average relative out-of-distribution accuracy on both NLI and math tasks, and shows that a high-diversity 10K subset can outperform a lower-diversity 100K subset on OOD benchmarks, while scale matters more for in-distribution accuracy. It then claims that Prismatic Synthesis—iteratively clustering in gradient space, generating with few-shot prompting, and keeping only samples in the sparsest clusters—lets synthetic-data scaling improve performance past a million samples, yielding PrismMath-7B (distilled from a 32B generator) that beats the same-base 7B model distilled from a 671B teacher on 6 of 7 benchmarks, and PrismNLI that beats the best prior data mixture by 8 points in average OOD accuracy.

Load-bearing premise

Gradients from a small off-the-shelf model capture the same task-relevant diversity that decides how a different, larger student model generalizes, and this transfer holds for new generators, data pools, and tasks.

Editorial extensions

If this is right

  • A cheap, task-agnostic diversity signal from a 0.5B proxy model predicts generalization better than a 14-times-larger embedding model or an LLM skill taxonomy.
  • Diversity can substitute for scale on out-of-distribution benchmarks: a 10K high-diversity subset can beat a 100K low-diversity subset drawn from the same pool.
  • Synthetic-data scaling need not saturate: persona-guided generation plateaus around 50K–100K samples, while gradient-space diversification keeps improving beyond a million.
  • A student trained on a 32B generator's diversified output beats the same-base student trained on 671B-teacher data on 6 of 7 math benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If G-Vendi's strong correlation with generalization is causal, the same gradient-entropy signal could serve as an online filter in other training loops—pre-training, instruction tuning, code or planning—wherever gradients of a small proxy model are affordable.
  • The qualitative clusters in the paper's appendix share solution strategies rather than topics (tracking quantities step by step, computing remainders, age problems), suggesting G-Vendi tracks something like reasoning-skill coverage; a testable extension is whether the metric predicts generalization on code or planning tasks.
  • The paper's own limitation note says its strong correlation was found under careful quality control, so an untested boundary is whether G-Vendi orders datasets from genuinely different generators; comparing pools from differently sized or differently tuned generators would settle this.
  • Because the pipeline pairs a 32B solution generator with gradient diversification, the gains could come from coverage rather than teacher strength—implying weaker generators plus diversification might approach the same results at lower cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that gradient-space diversity, measured by G-Vendi (the exponentiated entropy of the covariance of projected loss gradients from a small off-the-shelf proxy model), strongly predicts out-of-distribution generalization of fine-tuned reasoning models. A controlled correlation study over 300 training runs in math and NLI supports this claim with R² ≈ 0.8 and Spearman's ρ ≈ 0.9. The paper then introduces Prismatic Synthesis, an iterative generate-cluster-sparse-reject framework for scaling synthetic data while increasing G-Vendi, and reports that the resulting PrismMath-7B and PrismNLI outperform strong baselines on several OOD benchmarks.

Significance. If the causal link between gradient-space diversification and OOD generalization were established, this would be a practically important and elegantly simple contribution: a cheap, model-based diversity metric that predicts generalization better than embedding or surface-form measures, plus an algorithm that turns that metric into a data-scaling recipe. The correlation study is well-designed in several respects: it uses many training runs, explicitly controls scale and quality by sampling from a single synthetic pool, does not fit G-Vendi to the target benchmarks, and evaluates on held-out OOD benchmarks. The metric is also computationally efficient (O(d²|D|) via the Gram matrix). These strengths are substantial and deserve credit.

major comments (3)
  1. [§3.2.2, Fig. 1 (right)] The scaling comparison does not isolate the effect of gradient-sparse selection. The 'Prismatic Synthesis' curve is produced by the full PrismMath pipeline (§3.1), which starts from OpenR1-Math, uses R1-32B for solution annotation, and applies majority-vote filtering (N=3, τ=2) plus decontamination, whereas the vanilla and persona curves are generated from a GSM8k/MATH seed pool with Qwen2.5-72B-Instruct and no majority-vote filtering (§A.2). The sustained improvement beyond 100K samples shown in Fig. 1 (right) therefore conflates Step 3 of Prismatic Synthesis with changes to the generator, seed pool, and verification filter, so the observed gain cannot be attributed to gradient-sparse cluster selection. Please add an ablation that holds the generator, quality filter, and student model fixed and varies only the sparse-cluster rejection, or explicitly frame the comparison as a system-level result rather than an isolation of the diversification step.
  2. [Section D and §3.1] The manuscript's own limitation statement says that G-Vendi's strong correlation 'can only be discovered via careful control of data quality' and that G-Vendi cannot be used to compare datasets from different data-generating processes. The PrismMath pipeline is a different data-generating process (OpenR1-Math seed, R1-32B generation, majority-vote verification) from the correlation study in §2.2 (GSM8k/MATH seed, Qwen2.5-72B generation, no verification). Since the causal attribution of Prismatic Synthesis's gains to gradient-space diversification rests on G-Vendi remaining predictive in this new regime, please provide direct evidence on the evolving PrismMath pool (e.g., G-Vendi of intermediate pools vs. held-out validation performance across iterations), or substantially temper the causal claims in the title and abstract.
  3. [§2.3.1, Table 1] The central claim that G-Vendi 'strongly predicts' generalization is based on R² and Spearman's ρ values reported without confidence intervals, standard errors, or significance tests. Given that this claim is load-bearing for the rest of the paper, please add bootstrap confidence intervals for ρ and R² (per data size and pooled), and state explicitly how many independent subsets contribute to each correlation value.
minor comments (5)
  1. [Appendix F.1] The heading 'Closet Sample' appears twice and should read 'Closest Sample'.
  2. [Table 2] The entry '87.864' has an extra digit; it should be '87.86' consistent with the other values.
  3. [§A.2 vs §3.1] The scaling-curve experiments are said to train 'Qwen2.5-7B-Math' in §A.2, while §3.1 states that PrismMath-7B is obtained by fine-tuning 'Qwen2.5-Math-7B-Instruct'; please clarify whether these are the same checkpoint.
  4. [Fig. 2 and Table 1] The text reports ρ ≈ 0.9, but Fig. 2 gives per-size ρ values (0.898–0.918 for math and 0.893–0.913 for NLI); please state clearly in the main text which correlation (pooled or per-size) is being cited.
  5. [NeurIPS Checklist, item 16] The checklist states that no LLM was used as an important development tool, yet the core method uses LLMs (Qwen2.5-72B-Instruct, R1-32B) as data generators; this response appears inconsistent with the paper's methodology.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: G-Vendi is an empirically validated correlate, and Prismatic Synthesis is assessed on separate downstream benchmarks.

full rationale

G-Vendi is defined in Eqs. (1)-(3) as the Vendi score of normalized loss gradients from an off-the-shelf proxy model; no term in the definition is fitted to the benchmark accuracies whose prediction is claimed. The correlation result in Section 2.3.1 is evaluated on 300 subsets against held-out OOD benchmarks and reported as Spearman's rho and R^2, i.e., an empirical association rather than a quantity forced by the metric or by a fitted parameter. Prismatic Synthesis uses cluster sparsity in the same gradient space as a selection rule, but its success is assessed by training separate student models and evaluating them on unseen benchmarks (Table 4), so the selection criterion and the evaluation signal are not the same object. The scaling comparison in Section 3.2.2 changes the solution generator and quality filtering alongside diversification, which weakens causal attribution of the gains to diversification, but that is an experimental confound rather than a definitional reduction. Self-citations [18, 19, 34] appear only in related-work positioning and do not carry the load-bearing argument. Section D explicitly concedes that the strong correlation requires careful quality control and cannot be assumed for arbitrary data-generating processes, which further confirms that the central claim is empirical rather than circular.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on a small number of hand-chosen hyperparameters (projection dimension, cluster fractions, voting thresholds) and on domain assumptions about gradient-based influence, the validity of the Vendi score as a diversity measure, and the transferability of proxy-model gradients. The assumptions are reasonable and partially ablated, but none are formally derived. No new physical or conceptual entities are introduced beyond the metric and algorithm themselves.

free parameters (4)
  • Random projection dimension d = 1024
    Chosen by hand for G-Vendi; no sensitivity analysis is provided, and it controls the rank and granularity of the gradient representations.
  • Cluster count k and kept-sparse-cluster fraction = k = 1% of pool size; keep smallest k/2 clusters
    Algorithm hyperparameters for Prismatic Synthesis; no ablation shows how performance depends on these choices.
  • Majority-voting filter N and threshold tau = N=3, tau=2 for PrismMath; N=2, tau=2 for PrismNLI
    Quality-filter thresholds chosen by hand; they affect final dataset composition and are not applied to the vanilla/persona baselines in the scaling comparison, creating a confound.
  • Similarity threshold tau in Lower Diversity Sampling = Not specified
    Controls the construction of low-diversity subsets in the correlation study; the exact value is not reported, which limits precise replication of the diversity spectrum.
assumptions (5)
  • domain assumption Loss-reduction influence is approximately proportional to the dot product of loss gradients (Pruthi et al. 2020).
    Motivates G-Vendi; cited from prior work and not re-derived or tested in this paper.
  • standard math Johnson-Lindenstrauss random projection preserves pairwise inner products of normalized gradients.
    Used to reduce gradient dimension from |theta| to 1024; standard result, no verification needed.
  • ad hoc to paper The Vendi score of the covariance matrix of projected gradients is a meaningful measure of data diversity.
    Core definitional choice; its validity is supported only through downstream correlations with performance, not by an independent theoretical argument.
  • domain assumption An off-the-shelf small instruction-tuned model's gradients are a valid proxy for task-relevant diversity for different student models.
    Load-bearing for transfer to Prismatic Synthesis; only partially ablated in Sections 2.3.3 and 2.3.4.
  • domain assumption Average relative accuracy across 7 OOD benchmarks is a valid measure of generalization.
    Defines Perf in Eq (5); the choice of benchmarks and the relative normalization can affect the strength of the observed correlations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning." pith.science (2026). https://pith.science/paper/A67KMUY5

@misc{pith2026250520161,
  author       = {Pith},
  title        = {Pith review of: Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A67KMUY5}},
  note         = {Machine review of arXiv:2505.20161}
}
abstract

Effective generalization in language models depends critically on the diversity of their training data. Yet existing diversity metrics often fall short of this goal, relying on surface-level heuristics that are decoupled from model behavior. This motivates us to ask: What kind of diversity in training data actually drives generalization in language models -- and how can we measure and amplify it? Through large-scale empirical analyses spanning over 300 training runs, carefully controlled for data scale and quality, we show that data diversity can be a strong predictor of generalization in LLM reasoning -- as measured by average model performance on unseen out-of-distribution benchmarks. We introduce G-Vendi, a metric that quantifies diversity via the entropy of model-induced gradients. Despite using a small off-the-shelf proxy model for gradients, G-Vendi consistently outperforms alternative measures, achieving strong correlation (Spearman's $\rho \approx 0.9$) with out-of-distribution (OOD) performance on both natural language inference (NLI) and math reasoning tasks. Building on this insight, we present Prismatic Synthesis, a framework for generating diverse synthetic data by targeting underrepresented regions in gradient space. Experimental results show that Prismatic Synthesis consistently improves model performance as we scale synthetic data -- not just on in-distribution test but across unseen, out-of-distribution benchmarks -- significantly outperforming state-of-the-art models that rely on 20 times larger data generator than ours. For example, PrismMath-7B, our model distilled from a 32B LLM, outperforms R1-Distill-Qwen-7B -- the same base model trained on proprietary data generated by 671B R1 -- on 6 out of 7 challenging benchmarks.

Figures

Figures reproduced from arXiv: 2505.20161 by the authors.

Figure 1
Figure 1. (Left) Overview of Prismatic Synthesis. We iteratively (1) cluster samples in a gradient space, (2) generate new samples, and (3) add to the pool only the samples in sparse clusters, consistently improving both the diversity and scale of generated dataset. (Right) Naive scaling of synthetic math data—with no diversification or with a heuristic persona-guided prompting [9]—faces early saturation, when measuring avera… view at source ↗
Figure 2
Figure 2. G-Vendi and model OOD performance. G-Vendi shows a strong log-linear relationship with model performance, when controlling for data scale and quality. In both tasks, models trained with datasets of high G-Vendi tend to generalize better in OOD benchmarks. Plots for baseline measures are shown in §B.1. College Math, GSM-IC, Aqua-RAT and Minerva-Math [71, 14, 52, 27, 22]—where the reference model Mref achieves meaning… view at source ↗
Figure 3
Figure 3. (Left) G-Vendi and in-distribution performance. Compared to OOD, ID performance is [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Relationship between baseline diversity measures and model OOD performance, measured [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. How Much Is a Dataset Worth? Scaling Laws, the Vendi Score, and Matrix Spectral Functions

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Vendi Score and scaling-law objectives belong to the class of matrix spectral functions, which are submodular, enabling efficient greedy selection of training data that outperforms random subsets in predicting held-ou...

  2. Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Coverage of sparse-autoencoder-identified task features predicts post-training performance and can guide synthesis of small, high-impact datasets (2,000 vs. 300,000 samples).

  3. Socratic-MCTS: Test-Time Visual Reasoning by Asking the Right Questions

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Socratic-MCTS uses the model's own subquestions and answers in a Monte Carlo Tree Search to improve multimodal multiple-choice accuracy without fine-tuning.

Reference graph

Works this paper leans on

103 extracted references · 73 canonical work pages · cited by 3 Pith papers

  1. [1]

    J. T. Ash, C. Zhang, A. Krishnamurthy, J. Langford, and A. Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds, 2020

    2020

  2. [2]

    Bukharin, S

    A. Bukharin, S. Li, Z. Wang, J. Yang, B. Yin, X. Li, C. Zhang, T. Zhao, and H. Jiang. Data diversity matters for robust instruction tuning. In Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 3411–3425, Miami, Florida, USA, Nov. 2024. Association for Computational Linguistics

    2024

  3. [3]

    H. Chen, A. Waheed, X. Li, Y . Wang, J. Wang, B. Raj, and M. I. Abdin. On the diversity of synthetic data and its impact on training large language models, 2024

    2024

  4. [4]

    Cobbe, V

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021

    2021

  5. [5]

    DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y . Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. W...

    2025

  6. [6]

    Didolkar, A

    A. Didolkar, A. Goyal, N. R. Ke, S. Guo, M. Valko, T. Lillicrap, D. Rezende, Y . Bengio, M. Mozer, and S. Arora. Metacognitive capabilities of llms: An exploration in mathematical problem solving, 2024

    2024

  7. [7]

    Fourrier, N

    C. Fourrier, N. Habib, H. Kydlíˇcek, T. Wolf, and L. Tunstall. Lighteval: A lightweight framework for llm evaluation, 2023

    2023

  8. [8]

    Friedman and A

    D. Friedman and A. B. Dieng. The vendi score: A diversity evaluation metric for machine learning, 2023

    2023

Show all 103 references
  1. [9]

    T. Ge, X. Chan, X. Wang, D. Yu, H. Mi, and D. Yu. Scaling synthetic data creation with 1,000,000,000 personas, 2024

    2024

  2. [10]

    Gunasekar, Y

    S. Gunasekar, Y . Zhang, J. Aneja, C. C. T. Mendes, A. D. Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O. Saarikivi, A. Salim, S. Shah, H. S. Behl, X. Wang, S. Bubeck, R. Eldan, A. T. Kalai, Y . T. Lee, and Y . Li. Textbooks are all you need, 2023. 10

    2023

  3. [11]

    Havrilla, A

    A. Havrilla, A. Dai, L. O’Mahony, K. Oostermeijer, V . Zisler, A. Albalak, F. Milo, S. C. Raparthy, K. Gandhi, B. Abbasi, D. Phung, M. Iyer, D. Mahan, C. Blagden, S. Gureja, M. Hamdy, W.- D. Li, G. Paolini, P. S. Ammanamanchi, and E. Meyerson. Surveying the effects of quality,...

    2024

  4. [12]

    C. He, R. Luo, Y . Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y . Huang, Y . Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024

    2024

  5. [13]

    P. He, X. Liu, J. Gao, and W. Chen. Deberta: Decoding-enhanced bert with disentangled attention, 2021

    2021

  6. [14]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  7. [15]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset, 2021

    2021

  8. [16]

    Open r1: A fully open reproduction of deepseek-r1, January 2025

    Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025

    2025

  9. [17]

    W. B. Johnson, J. Lindenstrauss, et al. Extensions of lipschitz mappings into a hilbert space. Contemporary mathematics, 26(189-206):1, 1984

    1984

  10. [18]

    J. Jung, X. Lu, L. Jiang, F. Brahman, P. West, P. W. Koh, and Y . Choi. Information-theoretic distillation for reference-less summarization. In First Conference on Language Modeling, 2024

    2024

  11. [19]

    J. Jung, P. West, L. Jiang, F. Brahman, X. Lu, J. Fisher, T. Sorensen, and Y . Choi. Impossible distillation for paraphrasing and summarization: How to make high-quality lemonade out of small, low-quality model. In K. Duh, H. Gomez, and S. Bethard, editors, Proceedings of the ...

    2024

  12. [20]

    Killamsetty, S

    K. Killamsetty, S. Durga, G. Ramakrishnan, A. De, and R. Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pages 5464–5474. PMLR, 2021

    2021

  13. [21]

    Killamsetty, D

    K. Killamsetty, D. Sivasubramanian, G. Ramakrishnan, and R. Iyer. Glister: Generalization based data subset selection for efficient and robust learning, 2021

    2021

  14. [22]

    Lewkowycz, A

    A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V . Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y . Wu, B. Neyshabur, G. Gur-Ari, and V . Misra. Solving quantitative reasoning problems with language models, 2022

    2022

  15. [23]

    J. LI, E. Beeching, L. Tunstall, B. Lipkin, R. Soletskyi, S. C. Huang, K. Rasul, L. Yu, A. Jiang, Z. Shen, Z. Qin, B. Dong, L. Zhou, Y . Fleureau, G. Lample, and S. Polu. Numina- math. [https://huggingface.co/AI-MO/NuminaMath-CoT](https://github.com/ project-numina/aimo-progre...

    2024

  16. [24]

    Y . Li, S. Bubeck, R. Eldan, A. D. Giorno, S. Gunasekar, and Y . T. Lee. Textbooks are all you need ii: phi-1.5 technical report, 2023

    2023

  17. [25]

    Z. Li, X. Zhang, Y . Zhang, D. Long, P. Xie, and M. Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281, 2023

    2023 arXiv

  18. [26]

    Lightman, V

    H. Lightman, V . Kosaraju, Y . Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step, 2023

    2023

  19. [27]

    W. Ling, D. Yogatama, C. Dyer, and P. Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. ACL, 2017

    2017

  20. [28]

    A. Liu, S. Swayamdipta, N. A. Smith, and Y . Choi. Wanli: Worker and ai collaboration for natural language inference dataset creation, 2022. 11

    2022

  21. [29]

    H. Liu, L. Cui, J. Liu, and Y . Zhang. Natural language inference in context - investigating con- textual reasoning over long texts. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15):13388–13396, May 2021

    2021

  22. [30]

    W. Liu, W. Zeng, K. He, Y . Jiang, and J. He. What makes good data for alignment? a compre- hensive study of automatic data selection in instruction tuning. In The Twelfth International Conference on Learning Representations, 2024

    2024

  23. [31]

    The llama 3 herd of models, 2024

    Llama Team. The llama 3 herd of models, 2024

    2024

  24. [32]

    Longpre, L

    S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y . Tay, D. Zhou, Q. V . Le, B. Zoph, J. Wei, and A. Roberts. The flan collection: Designing data and methods for effective instruction tuning, 2023

    2023

  25. [33]

    K. Lu, H. Yuan, Z. Yuan, R. Lin, J. Lin, C. Tan, C. Zhou, and J. Zhou. #instag: Instruction tagging for analyzing supervised fine-tuning of large language models, 2023

    2023

  26. [34]

    X. Lu, S. Han, D. Acuna, H. Kim, J. Jung, S. Prabhumoye, N. Muennighoff, M. Patwary, M. Shoeybi, B. Catanzaro, and Y . Choi. Retro-search: Exploring untaken paths for deeper and efficient reasoning, 2025

    2025

  27. [35]

    Maharana, P

    A. Maharana, P. Yadav, and M. Bansal. D2 pruning: Message passing for balancing diversity and difficulty in data pruning, 2023

    2023

  28. [36]

    Maini, S

    P. Maini, S. Seto, H. Bai, D. Grangier, Y . Zhang, and N. Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling, 2024

    2024

  29. [37]

    R. T. McCoy, E. Pavlick, and T. Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In A. Korhonen, D. Traum, and L. Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages...

    2019

  30. [38]

    Mirzasoleiman, J

    B. Mirzasoleiman, J. Bilmes, and J. Leskovec. Coresets for data-efficient training of machine learning models, 2020

    2020

  31. [39]

    Muennighoff, N

    N. Muennighoff, N. Tazi, L. Magne, and N. Reimers. Mteb: Massive text embedding benchmark, 2023

    2023

  32. [40]

    Muennighoff, Z

    N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Candès, and T. Hashimoto. s1: Simple test-time scaling, 2025

    2025

  33. [41]

    X. Ni, Y . Gong, Z. Gou, Y . Shen, Y . Yang, N. Duan, and W. Chen. Exploring the mystery of influential data for mathematical reasoning, 2024

    2024

  34. [42]

    Y . Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela. Adversarial NLI: A new benchmark for natural language understanding. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Com- putation...

    2020

  35. [43]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024

    2024

  36. [44]

    Open Thoughts

    OpenThoughts Team. Open Thoughts. https://open-thoughts.ai, Jan. 2025

    2025

  37. [45]

    J. Pang, J. Wei, A. P. Shah, Z. Zhu, Y . Wang, C. Qian, Y . Liu, Y . Bao, and W. Wei. Improving data efficiency via curating llm-driven rating systems, 2025

    2025

  38. [46]

    S. M. Park, K. Georgiev, A. Ilyas, G. Leclerc, and A. Madry. Trak: Attributing model behavior at scale, 2023

    2023

  39. [47]

    W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery.Numerical recipes in C (2nd ed.): the art of scientific computing. Cambridge University Press, USA, 1992

    1992

  40. [48]

    Pruthi, F

    G. Pruthi, F. Liu, M. Sundararajan, and S. Kale. Estimating training data influence by tracing gradient descent, 2020. 12

    2020

  41. [49]

    Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, ...

    2025

  42. [50]

    B. T. Rakhshan and G. Rabusseau. Tensorized random projections. ArXiv, abs/2003.05101, 2020

    2003 arXiv

  43. [51]

    V . Shah, D. Yu, K. Lyu, S. Park, J. Yu, Y . He, N. R. Ke, M. Mozer, Y . Bengio, S. Arora, and A. Goyal. Ai-assisted generation of difficult math questions, 2025

    2025

  44. [52]

    F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. Chi, N. Schärli, and D. Zhou. Large language models can be easily distracted by irrelevant context. arXiv preprint arXiv:2302.00093, 2023

    2023 arXiv

  45. [53]

    Srivastava, A

    A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, and A. G.-A. et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023

    2023

  46. [54]

    Toshniwal, W

    S. Toshniwal, W. Du, I. Moshkov, B. Kisacanin, A. Ayrapetyan, and I. Gitman. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data, 2024

    2024

  47. [55]

    Vendrow, E

    J. Vendrow, E. Vendrow, S. Beery, and A. Madry. Do large language model benchmarks test reliability?, 2025

    2025

  48. [56]

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In T. Linzen, G. Chrupała, and A. Alishahi, editors, Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpr...

    2018

  49. [57]

    P. Wang, Y . Shen, Z. Guo, M. Stallone, Y . Kim, P. Golland, and R. Panda. Diversity measurement and subset selection for instruction tuning datasets. In ICLR 2025 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2025

    2025

  50. [58]

    Y . Wang, S. Mishra, P. Alipoormolabashi, Y . Kordi, A. Mirzaei, A. Arunkumar, A. Ashok, A. S. Dhanasekaran, A. Naik, D. Stap, E. Pathak, G. Karamanolakis, H. G. Lai, I. Purohit, I. Mondal, J. Anderson, K. Kuznia, K. Doshi, M. Patel, K. K. Pal, M. Moradshahi, M. Parmar, M. Pur...

    2022

  51. [59]

    K. Wei, R. Iyer, and J. Bilmes. Submodularity in data subset selection and active learning. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, page 1954–1963. JMLR.org, 2015

    1954

  52. [60]

    S. Wu, K. Lu, B. Xu, J. Lin, Q. Su, and C. Zhou. Self-evolved diverse data sampling for efficient instruction tuning, 2023

    2023

  53. [61]

    M. Xia, S. Malladi, S. Gururangan, S. Arora, and D. Chen. Less: Selecting influential data for targeted instruction tuning, 2024

    2024

  54. [62]

    Yang, W.-L

    S. Yang, W.-L. Chiang, L. Zheng, J. E. Gonzalez, and I. Stoica. Rethinking benchmark and contamination for language models with rephrased samples, 2023

    2023

  55. [63]

    T. Ye, Z. Xu, Y . Li, and Z. Allen-Zhu. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process, 2024

    2024

  56. [64]

    Y . Ye, Z. Huang, Y . Xiao, E. Chern, S. Xia, and P. Liu. Limo: Less is more for reasoning, 2025

    2025

  57. [65]

    L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y . Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2024. 13

    2024

  58. [66]

    S. Yu, L. Chen, S. Ahmadian, and M. Fadaee. Diversify and conquer: Diversity-centric data selection with iterative refinement, 2025

    2025

  59. [67]

    Y . Yu, Y . Zhuang, J. Zhang, Y . Meng, A. Ratner, R. Krishna, J. Shen, and C. Zhang. Large language model as attributed training data generator: A tale of diversity and bias. In Thirty- seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023

    2023

  60. [68]

    Z. Yu, S. Das, and C. Xiong. Mates: Model-aware data selection for efficient pretraining with data influence models, 2024

    2024

  61. [69]

    L. Yuan, Y . Chen, G. Cui, H. Gao, F. Zou, X. Cheng, H. Ji, Z. Liu, and M. Sun. Revisiting out-of-distribution robustness in nlp: Benchmark, analysis, and llms evaluations, 2023

    2023

  62. [70]

    Zhang, J

    D. Zhang, J. Wang, and F. Charton. Instruction diversity drives generalization to unseen tasks, 2024

    2024

  63. [71]

    Zhong, R

    W. Zhong, R. Cui, Y . Guo, Y . Liang, S. Lu, Y . Wang, A. Saied, W. Chen, and N. Duan. AGIEval: A human-centric benchmark for evaluating foundation models. In K. Duh, H. Gomez, and S. Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, page...

    2024

  64. [72]

    C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y . Mao, X. Ma, A. Efrat, P. Yu, L. Yu, S. Zhang, G. Ghosh, M. Lewis, L. Zettlemoyer, and O. Levy. Lima: Less is more for alignment, 2023. 14 A Experimental Details A.1 Evaluating Diversity Measures Generating Data Pool We further illus...

    2023

  65. [73]

    This is given directly in the problem statement

    Jenny has 6 cards. This is given directly in the problem statement. ...[omitted] Therefore, the total number of Pokemon cards they have all together is 38 cards. Closest Sample in Gradient Space Problem: Levi has 5 lemons. Jayden has 6 more lemons than Levi. Jayden has one-thi...

  66. [74]

    23 Example 2 Original Sample Problem: The number of math problems that Marvin practiced today is three times as many as the number of problems he solved yesterday

    Calculate the number of cards Cindy has:...[omitted] So, the number of cards Rex has left is 150 . 23 Example 2 Original Sample Problem: The number of math problems that Marvin practiced today is three times as many as the number of problems he solved yesterday. His friend, Ar...

  67. [75]

    24 Example 3 Original Sample Problem: The $4.55 in Carol’s piggy bank consists of quarters and nickels

    **Calculate the number of problems Jenna answered:** ...[omitted] Therefore, Angela was able to finish 9 problems on her own. 24 Example 3 Original Sample Problem: The $4.55 in Carol’s piggy bank consists of quarters and nickels. There are seven more nickels than quarters. How...

  68. [76]

    Smallville

    **Calculate the total number of quarters Maria has now:**...[omitted] Therefore, the total amount of money Maria has in her piggy bank is $3.00 . 25 F.2 Example Clusters in Gradient Space We additionally analyze clusters of data points in gradient space, for both NLI and math ...

    2015

  69. [77]

    [omitted] Therefore, the total number of sticks the three boys need to collect is 129 sticks

    **Simon’s Raft**: Simon needs 36 sticks. ...[omitted] Therefore, the total number of sticks the three boys need to collect is 129 sticks. Problem: Bella bought stamps at the post office. Some of the stamps had a snowflake design, some had a truck design, and some had a rose de...

  70. [78]

    [omitted] Thus, the total number of stamps Bella bought is 38

    Bella bought 11 snowflake stamps. ...[omitted] Thus, the total number of stamps Bella bought is 38 . Problem: Marcy is the lunch monitor in an elementary school cafeteria. She gives 5 time-outs for running, 1 less than five times that number of time-outs for throwing food, and...

  71. [79]

    Problem: Rebecca makes her own earrings out of buttons, magnets, and gemstones

    **Calculate the number of time-outs for running**: ...[omitted] Thus, the students spend a total of 185 minutes in time-out. Problem: Rebecca makes her own earrings out of buttons, magnets, and gemstones. For every earring, she uses two magnets, half as many buttons as magnets...

  72. [80]

    28 Math Example 2: Computing the remainder Problem: At the height of cranberry season, there are 60000 cranberries in a bog

    **Calculate the total number of earrings**: ...[omitted] Therefore, to make 4 sets of earrings, Rebecca will need 24 gemstones. 28 Math Example 2: Computing the remainder Problem: At the height of cranberry season, there are 60000 cranberries in a bog. 40% are harvested by hum...

  73. [81]

    [omitted] Therefore, the number of cranberries left in the bog after being harvested by humans and eaten by elk is 16000 cranberries

    Calculate the total number of cranberries harvested by humans. ...[omitted] Therefore, the number of cranberries left in the bog after being harvested by humans and eaten by elk is 16000 cranberries. Problem: Barney’s grocery store sold out all of its items at the beginning of...

  74. [82]

    Problem: Out of 804 senior high school students, 75% passed their exams and so got their degree

    Calculate the total number of items Barney’s grocery store had after ordering extra items: ...[omitted] Therefore, the total number of items left in the whole store is 3472 . Problem: Out of 804 senior high school students, 75% passed their exams and so got their degree. The r...

  75. [83]

    [omitted] Therefore, the number of students who didn’t pass their exams is 201 students

    First, we calculate the number of students who passed their exams and got their degree. ...[omitted] Therefore, the number of students who didn’t pass their exams is 201 students. Problem: Miranda wants to buy a pair of heels she saw online. She saved money for 3 months. Her s...

  76. [84]

    [omitted] Therefore, Miranda saved $70 per month

    First, we need to determine how much money Miranda actually saved before her sister gave her the additional $50. ...[omitted] Therefore, Miranda saved $70 per month. 29 Math Example 3: How old is X now? Problem: In 10 years, Cornelia will be three times as old as Kilee. If Kil...

  77. [85]

    Problem: In 5 years, Raven will be 4 times as old as Phoebe

    Calculate Kilee’s age in 10 years: ...[omitted] Therefore, Cornelia is currently 80 years old. Problem: In 5 years, Raven will be 4 times as old as Phoebe. If Phoebe is currently 10 years old, how old is Raven? Solution: To solve this problem, we follow these steps:

  78. [86]

    Problem: After five years, Ron will be four times as old as Maurice

    Calculate Phoebe’s age in 5 years:...[omitted] Therefore, Raven is currently 55 years old. Problem: After five years, Ron will be four times as old as Maurice. If Ron’s age now is 43, how old is Maurice now? Solution: 1. Calculate Ron’s age in five years: - Ron’s current age i...

  79. [87]

    30 NeurIPS Paper Checklist

    Calculate Suzy’s age in four years: ...[omitted] Thus, Mary is 8 years old now. 30 NeurIPS Paper Checklist

  80. [88]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We propose G-Vendi and demonstrate that it strongly predicts model general- ization in reasoning tasks as empirically ...

  81. [89]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: See §D. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are no...

  82. [90]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 31 Answer: [NA] Justification: Our contribution focuses on empirical evaluation with its motivation theoreti- cally groun...

  83. [91]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  84. [92]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code 32 Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The code is provided as ...

  85. [93]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We include detailed experimen...

  86. [94]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We provide the standard errors for iterated experi...

  87. [95]

    • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We elaborate on GPU usa...

  88. [96]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We abide by the NeurIPS Code of Ethics. Guidelines: • The answer NA means that ...

  89. [97]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: See §E for the broader impacts of our work. Guidelines: • The answer NA means that there is no societal i...

  90. [98]

    We did not scrape any internet data, and our models are narrowly trained on these specific tasks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our domain of a...

  91. [99]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite all exis...

  92. [100]

    Guidelines: • The answer NA means that the paper does not release new assets

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We include documentations of our code and data released in the README accompanying the supplementary submission. Gui...

  93. [101]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  94. [102]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  95. [103]

    Answer: [NA] Justification: We did not use an LLM as a development tool in all phases of our research other than for formatting purposes

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.