Pith. sign in

REVIEW 3 major objections 4 minor 73 references

Efficient Data Selection at Scale via Influence Distillation

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that Influence Distillation selects better or equal fine-tuning data than state-of-the-art methods while cutting selection cost by up to 3.5x.

desk verdict A practical landmark-based data selection method with solid empirical results, but the 'mathematically justified' claim overreaches and the speedup excludes warm-up cost. read the letter →

arxiv 2505.19051 v1 pith:VIQYNRH6 submitted 2025-05-25 cs.CL cs.LG

Mahdi Nikdan , Vincent Cohen-Addad , Dan Alistarh , Vahab Mirrokni This is my paper
classification cs.CLcs.LG
keywords dataselectioninfluencedistillationinstructiontuninglandmarkapproximationJacobian-vectorproductembeddingsgradientprojectionlargelanguagemodelsAdamoptimizer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Influence Distillation, a data-selection method for fine-tuning large language models that assigns each training sample a weight reflecting how much training on it would reduce loss on a small target dataset. The weights are derived from a second-order Taylor expansion of the target loss after one gradient step, then regularized to be non-negative and sum to the source size. To make this tractable at 200k-sample scale, influence scores are computed exactly for a few thousand landmark samples and propagated to all other samples through a low-rank approximation built on Jacobian-vector-product embeddings. Experiments on instruction tuning with Llama and Qwen models show the selected subsets match or beat state-of-the-art selection on six benchmarks while cutting embedding-plus-selection cost by up to 3.5x. The paper also proves that, under unbiasedness and bounded mean-squared approximation error, the recovered sample weights are close to the true weights in high dimension.

What carries the argument

The central object is the landmark-based gradient approximation. Given a small set $L$ of landmark samples, the paper computes their exact gradients and fits a coefficient matrix $C$ by kernel ridge regression on low-dimensional embeddings, then approximates every sample's gradient as $\hat{g}_i = G_L^\top c_i$. Influence scores are then obtained from the first-order objective $\min_w \, -p^\top w + \tfrac{\lambda}{2}\|w\|^2$ subject to $w \ge 0$ and $w^\top 1 = |S|$, where $p = G_S g_T$. The crucial ingredient is the JVP embedding, which projects the Jacobian of intermediate logits onto random directions and is shown to correlate with true gradients much better than standard embeddings.

What would settle it

Train the same setup but replace JVP embeddings with random Gaussian embeddings of the same dimension, keeping the landmark set and all other hyperparameters fixed; if the selected subsets still match or beat RDS+, the embedding-to-gradient transfer assumption is not doing the work the paper claims. Alternatively, measure cosine similarity between reconstructed and true gradients on held-out samples; if similarity is at chance while selection accuracy remains high, the mechanism is not gradient recovery.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that second-order influence can be distilled at scale: instead of computing per-sample gradients for the whole pool, it suffices to compute them for a small landmark set and transfer the influence scores to all other samples using a linear reconstruction from cheap JVP embeddings. The paper shows this transfer is faithful enough that the resulting weights select subsets that consistently beat uniform sampling and often beat RDS+, and that the whole selection pipeline is 2.9-3.5x faster on the same problem. A formal bound (Theorem 4.1) ties the error in the final weights to the average squared recovery error divided by dimension and regularization, which is why high-dimensional models tolerate weak gradient recovery.

Load-bearing premise

The whole landmark pipeline rests on the assumption that the linear relationships among samples in the JVP-embedding space match the linear relationships among their true gradients; if embedding geometry does not transfer to gradient geometry, the computed influence scores are essentially noise.

Editorial extensions

If this is right

  • If correct, large instruction-tuning pools can be pruned to roughly 10k samples with equal or higher target-task accuracy than stronger baselines, at a fraction of the selection cost.
  • The Pareto-frontier results imply practitioners can afford to select from larger pools for a fixed compute budget, improving accuracy without extra training cost.
  • The derivation for Adam adapts the same weighting to the optimizer most commonly used for LLM fine-tuning.
  • The theoretical bound predicts that gains sharpen with model dimension $d$, since weight-error scales as $1/d$, benefiting exactly the largest models.
  • The per-target iteration over target samples yields task-specific rankings that can be aggregated across tasks, enabling multi-task data selection from one run.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The landmark approximation could be reused as a cheap gradient surrogate for other gradient-based influence and data-selection methods, not just the quadratic objective presented here.
  • The JVP embeddings, being model-dependent but much cheaper than full gradients, may serve as a general-purpose representation for data curation during pre-training, with periodic re-sampling as the paper suggests for longer horizons.
  • One could test the transfer assumption directly by comparing selected sets with true-gradient embeddings versus JVP embeddings across several pools; if JVP matches true-gradient selection, the assumption is validated in practice.
  • The explicit success with small target sets of 8-500 examples raises a scaling question worth probing: how small a target set can be before the influence signal vanishes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Influence Distillation (InfDist), a data-selection method for LLM instruction tuning. Starting from a second-order Taylor expansion of the target loss after one gradient step, the authors derive an optimal-weight objective for Gradient Descent and Adam. To make this scalable, they drop the second-order term (claimed to be negligible, justified in Appendix H), project gradients, and propose a landmark-based approximation: exact influence scores are computed on a small set of landmarks and propagated to all other samples via coefficients fit by Kernel Ridge Regression in a JVP-embedding space. Experiments on Tulu V2 (200k pool, 10k selected) with Llama2-7B, Llama3.2-3B, and Qwen2.5-1.5B/3B across six tasks show that InfDist matches or improves over RDS+ in three of four models and over uniform sampling in most settings, while reporting 2.9-3.5x lower embedding+selection FLOPs than RDS+. Ablations study the number of landmarks, pool size, and selected-set size. The paper closes with a limitations section acknowledging the lack of a target distribution, pre-training extension, and the exclusion of warm-up cost from the runtime.

Significance. If the results are reproducible, the landmark-based influence propagation with JVP embeddings is a useful practical contribution: it provides a way to approximate gradient-based influence scores without backpropagating through the full dataset, and the reported accuracy gains over RDS+ are non-trivial. The paper is honest about several limitations and includes 3-seed experiments, ablations, and a cost model. However, the formal justification (Theorem 4.1) rests on assumptions that are not verified for the deployed algorithm, and the headline speedup excludes a non-negligible warm-up phase. The method's practical value is therefore plausible, but the theoretical and computational claims need to be scaled back or substantiated.

major comments (3)
  1. [Section 4.3 (Theorem 4.1) and Appendix D] The theorem's assumptions are not verified for the deployed approximation. The deployed ĝ_i is a deterministic linear combination of a fixed landmark set with coefficients c_i obtained by minimizing ||E_S − C E_L||² via KRR; there is no mechanism to guarantee E[ĝ_i]=g_i over the random choice of landmarks and JVP directions, and no bias test is reported. Appendix F (Figure 7) shows average gradient-recovery cosine similarity of about 0.1 with 4096 JVP landmarks (versus over 0.9 when true gradients are used as embeddings), which indicates a large, systematic error rather than the zero-mean isotropic noise required by Lemma D.1. The theorem also assumes rotational equivariance of the approximation map, but the deployed map uses JVP embeddings as side information, so it is not a function of the gradient matrix alone. The claim that the landmark approximation is 'mathematically justified' should be revised to state that the theory applies to an idealized unbiased approximation, and the empirical validation of the approximation's bias and variance should be added.
  2. [Section 5.2 and Section 6 (Limitations)] The speedup claim of 'up to 3.5× faster selection' is based on the reported Embd+Sel cost, which explicitly excludes the warm-up phase (stated in the Limitations section). For the Llama2-7B/200k setting, the warm-up on 10k random samples requires approximately 10,000 × 6·d·2048 FLOPs (with d=7B), i.e., on the order of 860 TF, which is comparable to the reported 872 TF total for InfDist; including it would reduce the speedup over RDS+ from about 3.2× to about 1.6×. The two justifications in Section 6 (negligibility as pool grows, and possible shortening of warm-up) are plausible but not quantified for the experiments actually reported. Please report the speedup with warm-up cost included, or provide a sensitivity analysis using the short warm-up implied by Appendix A.
  3. [Appendix H] The decision to use the first-order variant across all main experiments is justified only by the small-scale study in Appendix H, which uses 128 source samples and 4 target examples for a single model (Qwen2.5-1.5B). The ratio |T2/T1| at practical learning rates may behave very differently for the 200k-sample pool and the specific tasks in Table 1. Since the first-order simplification is central to the reported efficiency (it avoids Hessian-vector products), the authors should validate it more directly, for example by comparing first-order and full second-order selection on a subset of the main setting, or by reporting the ratio for more models and target sets.
minor comments (4)
  1. [Section 3.3 (heading)] The heading 'Handling Variable Lebel Lengths' contains a typo: 'Lebel' should be 'Label'.
  2. [Related Work] DSIR is attributed to Do and Gaspers [2019], but it is actually introduced by Xie et al. [2023a]; the reference list also attributes 'SmallToLarge' to Xie et al. [2023b], whereas this method is due to Zhou et al. (2023). Please correct these citations.
  3. [Section 4.3 (Eq. 12)] The definition of JVP embeddings is not fully specified: Nℓ(x) is described as 'the logits of the next predicted token,' but the notation suggests a vector of logits; the shape of the Jacobian ∂Nℓ(x)/∂θℓ and how the average over V is normalized should be stated explicitly.
  4. [Table 1 and Figure 7] Several reported standard deviations are very large (e.g., SQuAD for Qwen2.5-1.5B, 54.4±13.13 for InfDist), and the number of seeds is not restated in each cell; please clarify whether the reported mean is over all seeds. In Figure 7, the two panels have different y-axis scales; using a shared scale would make the comparison between JVP and true-gradient embeddings easier to read.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the influence objective is a Taylor expansion of the stated target, lambda is tuned to a sparsity budget rather than to accuracy, and the main claims are evaluated against external baselines.

full rationale

Walking the paper's derivations, I find no step in which a prediction reduces by construction to a fitted input. The core objective (Eqs. 2-6) is obtained by a second-order Taylor expansion of the target loss after one gradient-descent step; p and Q are not defined in terms of the final selected set, so the weight objective carries independent content. The first-order simplification is explicitly aligned with LESS (Section 4.1), and the paper's added value is the landmark/JVP mechanism, whose end-to-end accuracy is checked against held-out tasks in Table 1 rather than fitted. The regularization parameter is selected by binary search to a sparsity budget (Section 4.4: 'we tune lambda via binary search to achieve a target sparsity level'), not to test accuracy, so no fitted constant is retrospectively reported as a prediction. The landmark theorem (Theorem 4.1) is conditional: it bounds weight error given unbiasedness and bounded MSE, but the deployed estimator's unbiasedness is not established; the paper itself states the transfer assumption in Section 4.3 ('This approximation implicitly assumes that the linear relationships learned in the embedding space transfer to the gradient space'). This is a validation gap and a correctness risk, not circularity. Self-citations (Axiotis et al. 2024; Peste et al. 2022) appear only in related-work and standard-technique contexts and do not carry the central claim. No uniqueness theorem is imported from the authors' prior work, and no known result is renamed as a new derivation without acknowledgment.

Assumptions & free parameters 7 free parameters · 6 assumptions · 1 invented entities

The method's core novelty rests on the landmark approximation and JVP embeddings. The influence objective itself is standard; the paper's free parameters are mostly engineering choices. The most fragile items are the transfer assumption and the unverified unbiasedness condition, both of which are required for the theoretical guarantee.

free parameters (7)
  • number of landmarks = 4096 for main experiments
    Chosen by hand; ablation in Figure 3 Left shows performance improves with more landmarks, so the result depends on this choice.
  • JVP embedding layers = 4 transformer blocks
    Chosen by hand; no ablation on layer count is reported.
  • JVP random vectors |V| = 2
    Chosen by hand; the JVP embedding averages over two random Gaussian directions.
  • gradient projection dimension = 131072
    Chosen as the largest Hadamard projection that fits in GPU memory, as described in Appendix I.
  • KRR dampening = 0.01
    Used when solving min_C ||E_S - C E_L|| with an RBF kernel; selected without reported sensitivity analysis.
  • regularization coefficient lambda = selected by binary search to target sparsity
    Not fitted to accuracy but determines which top-k samples are selected; still a chosen hyperparameter.
  • warm-up set size = 10000 random samples
    Used to stabilize gradients before computing JVP embeddings; the exact number of warm-up steps is not specified.
assumptions (6)
  • domain assumption Linear relationships in JVP embedding space transfer to gradient space
    Invoked in Section 4.3 to justify hat G_S = C G_L; the entire landmark approximation depends on it.
  • domain assumption Landmark gradient approximation is unbiased
    Theorem 4.1 and Corollary D.4 require E[hat g_i] = g_i for all i; no empirical verification of unbiasedness is provided.
  • domain assumption Second-order Hessian term is negligible for practical learning rates
    Appendix H checks the ratio on one small setting (128 source, 4 target, Qwen 1.5B) and then the main experiments use first-order only.
  • domain assumption Adam second moment v is approximately constant after warm-up
    Used in Appendix C to derive pAdam and QAdam; relies on beta_2 close to 1 and stabilized v.
  • domain assumption Gradient directions stabilize after a short warm-up
    Appendix A shows cosine similarities rise quickly; this justifies computing influence at a single checkpoint.
  • standard math Strong convexity and KKT conditions for active-set solution
    Appendix G solves the robust objective via Lagrange multipliers; assumes eta Q + lambda I is positive definite.
invented entities (1)
  • JVP embeddings
    purpose: Low-cost per-sample embeddings used to compute landmark coefficients C for gradient approximation
    The paper introduces JVP embeddings and validates them internally via gradient-recovery cosine similarity and end-to-end selection; no external benchmark is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Data Selection at Scale via Influence Distillation." pith.science (2026). https://pith.science/paper/VIQYNRH6

@misc{pith2026250519051,
  author       = {Pith},
  title        = {Pith review of: Efficient Data Selection at Scale via Influence Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VIQYNRH6}},
  note         = {Machine review of arXiv:2505.19051}
}
abstract

Effective data selection is critical for efficient training of modern Large Language Models (LLMs). This paper introduces Influence Distillation, a novel, mathematically-justified framework for data selection that employs second-order information to optimally weight training samples. By distilling each sample's influence on a target distribution, our method assigns model-specific weights that are used to select training data for LLM fine-tuning, guiding it toward strong performance on the target domain. We derive these optimal weights for both Gradient Descent and Adam optimizers. To ensure scalability and reduce computational cost, we propose a $\textit{landmark-based approximation}$: influence is precisely computed for a small subset of "landmark" samples and then efficiently propagated to all other samples to determine their weights. We validate Influence Distillation by applying it to instruction tuning on the Tulu V2 dataset, targeting a range of tasks including GSM8k, SQuAD, and MMLU, across several models from the Llama and Qwen families. Experiments show that Influence Distillation matches or outperforms state-of-the-art performance while achieving up to $3.5\times$ faster selection.

Figures

Figures reproduced from arXiv: 2505.19051 by the authors.

Figure 1
Figure 1. Average improvement over uniform sam￾pling across six tasks vs. runtime. The model used is Llama2-7B [Touvron et al., 2023], and the training dataset is Tulu V2 [Ivison et al., 2023]. The annotation “M/N” indicates that the method selected M samples from a pool of size N. Further details are provided in Section 5. Contribution. We introduce Influence Distilla￾tion, a novel framework for data selection that addresses… view at source ↗
Figure 2
Figure 2. (Left) Distribution of unconstrained weights, (Middle) Distribution of robust weights for [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (Left) Effect of the number of landmarks on the performance of Influence Distillation [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Average gradient cosine similarity on unseen samples from Tulu V2 (top) and BBH (bottom) [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Correlation between gradient norm and number of label tokens, across checkpoints on four [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: (Left) Distribution of theoretical robust weights for the linear case with [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: (Left) Gradient direction recovery vs number of landmarks, when different proxy embdeding [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: Ratio of second- to first-order terms for Qwen-2.5 1.5B across learning rates on two target [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]
Figure 9
Figure 9. Figure 9: Average gradient cosine similarity on unseen samples from GSM8k (top) and SQuAD [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

73 extracted references · 27 canonical work pages

  1. [1]

    Less: Selecting influential data for targeted instruction tuning

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333, 2024

    arXiv 2024

  2. [2]

    Compute-constrained data selection

    Junjie Oscar Yin and Alexander M Rush. Compute-constrained data selection. arXiv preprint arXiv:2410.16208, 2024

    arXiv 2024

  3. [3]

    Selecting Informative Contexts Improves Language Model Finetuning

    Richard Antonello, Nicole Beckage, Javier Turek, and Alexander Huth. Selecting informative contexts improves language model finetuning. arXiv preprint arXiv:2005.00175, 2020

    work page Pith review arXiv 2005

  4. [4]

    When less is more: Investigating data pruning for pretraining llms at scale

    Max Marion, Ahmet \"U st \"u n, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining llms at scale. arXiv preprint arXiv:2309.04564, 2023

    arXiv 2023

  5. [5]

    Perplexed by perplexity: Perplexity-based data pruning with small reference models

    Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L Leavitt, and Mansheej Paul. Perplexed by perplexity: Perplexity-based data pruning with small reference models. arXiv preprint arXiv:2405.20541, 2024

    arXiv 2024

  6. [6]

    From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning

    Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308.12032, 2023 a

    arXiv 2023

  7. [7]

    Large-scale data selection for instruction tuning

    Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi. Large-scale data selection for instruction tuning. arXiv preprint arXiv:2503.01807, 2025

    arXiv 2025

  8. [8]

    Woodruff, and Michael Wunder

    Kyriakos Axiotis, Vincent Cohen - Addad, Monika Henzinger, Sammy Jerome, Vahab Mirrokni, David Saulpic, David P. Woodruff, and Michael Wunder. Data-efficient learning via clustering-based sensitivity sampling: Foundation models and beyond. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenRevie...

    work page 2024

Show all 73 references
  1. [9]

    Data selection for language models via importance resampling

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 0 34201--34227, 2023 a

    2023

  2. [10]

    Dsdm: Model-aware dataset selection with datamodels, 2024

    Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels, 2024. URL https://arxiv. org/abs/2401.12926

    2024 arXiv

  3. [11]

    Dynimpt: A dynamic data selection method for improving model training efficiency

    Wei Huang, Yunxiao Zhang, Shangmin Guo, Yuming Shang, and Xiangling Fu. Dynimpt: A dynamic data selection method for improving model training efficiency. IEEE Transactions on Knowledge and Data Engineering, 2024

    2024

  4. [12]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  5. [13]

    Camels in a changing climate: Enhancing lm adaptation with tulu 2

    Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a changing climate: Enhancing lm adaptation with tulu 2. arXiv preprint arXiv:2311.10702, 2023

    2023 arXiv

  6. [14]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  7. [15]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/

    2024

  8. [16]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021 a

    2021

  9. [17]

    Aligning ai with shared human values

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021 b

    2021

  10. [18]

    Beyond neural scaling laws: beating power law scaling via data pruning

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 0 19523--19536, 2022

    2022

  11. [19]

    Semdedup: Data-efficient learning at web-scale through semantic deduplication

    Amro Abbas, Kushal Tirumala, D \'a niel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540, 2023

    2023 arXiv

  12. [20]

    Cross-lingual transfer learning with data selection for large-scale spoken language understanding

    Quynh Do and Judith Gaspers. Cross-lingual transfer learning with data selection for large-scale spoken language understanding. EMNLP, 2019

    2019

  13. [21]

    Smalltolarge (s2l): Scalable data selection for fine-tuning large language models by summarizing training loss trajectories of small models

    Sang Michael Xie et al. Smalltolarge (s2l): Scalable data selection for fine-tuning large language models by summarizing training loss trajectories of small models. arXiv preprint, 2023 b

    2023

  14. [22]

    Language models are few-shot learners

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Voss, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    2005 arXiv

  15. [23]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  16. [24]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  17. [25]

    Glam: Efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547--5569. PMLR, 2022

    2022

  18. [26]

    Intelligent selection of language model training data

    Robert C Moore and William Lewis. Intelligent selection of language model training data. In Proceedings of the ACL 2010 conference short papers, pages 220--224, 2010

    2010

  19. [27]

    Cynical selection of language model training data

    Amittai Axelrod. Cynical selection of language model training data. arXiv preprint arXiv:1709.02279, 2017

    2017 arXiv

  20. [28]

    Automatic document selection for efficient encoder pretraining

    Yukun Feng, Patrick Xia, Benjamin Van Durme, and Jo \ a o Sedoc. Automatic document selection for efficient encoder pretraining. arXiv preprint arXiv:2210.10951, 2022

    2022 arXiv

  21. [29]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  22. [30]

    Skill-it! a data-driven skills framework for understanding and training language models

    Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher R \'e . Skill-it! a data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  23. [31]

    Efficient online data mixing for language model pre-training

    Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training. In R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023

    2023

  24. [32]

    Lobass: Gauging learnability in supervised fine-tuning data

    Haotian Zhou et al. Lobass: Gauging learnability in supervised fine-tuning data. arXiv preprint arXiv:2310.13008, 2023 a

    2023 arXiv

  25. [33]

    Dataset cartography: Mapping and diagnosing datasets with training dynamics

    Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. EMNLP, 2020

    2020

  26. [34]

    An experimental design framework for label-efficient supervised finetuning of large language models

    Gantavya Bhatt et al. An experimental design framework for label-efficient supervised finetuning of large language models. arXiv preprint arXiv:2401.06692, 2024

    2024 arXiv

  27. [35]

    D4: improving LLM pretraining via document de-duplication and diversification

    Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: improving LLM pretraining via document de-duplication and diversification. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information P...

    2023

  28. [36]

    Dsdm: Model-aware dataset selection with datamodels

    Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=GC8HkKeH8s

    2024

  29. [37]

    Learning from less data: A unified data subset selection and active learning framework for computer vision

    Vishal Kaushal, Rishabh Iyer, Suraj Kothawade, Rohan Mahadev, Khoshrav Doctor, and Ganesh Ramakrishnan. Learning from less data: A unified data subset selection and active learning framework for computer vision. In 2019 IEEE Winter Conference on Applications of Computer Vision...

    2019

  30. [38]

    Retrieve: Coreset selection for efficient and robust semi-supervised learning

    Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient and robust semi-supervised learning. Advances in neural information processing systems, 34: 0 14488--14501, 2021

    2021

  31. [39]

    Submodularity in data subset selection and active learning

    Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International conference on machine learning, pages 1954--1963. PMLR, 2015

    1954

  32. [40]

    Alpagasus: Training a better alpaca with fewer data

    Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.08701, 2023

    2023 arXiv

  33. [41]

    Instruction mining: Instruction data selection for tuning large language models

    Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. Instruction mining: Instruction data selection for tuning large language models. arXiv preprint arXiv:2307.06290, 2023

    2023 arXiv

  34. [42]

    Active learning for convolutional neural networks: A core-set approach

    Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489, 2017

    2017 arXiv

  35. [43]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009

    2009

  36. [44]

    A software package for sequential quadratic programming

    Dieter Kraft. A software package for sequential quadratic programming. Forschungsbericht- Deutsche Forschungs- und Versuchsanstalt fur Luft- und Raumfahrt, 1988

    1988

  37. [45]

    Fundamental algorithms for scientific computing in python and scipy 1.0 contributors

    P Virtanen, R Gommers, TE Oliphant, M Haberland, T Reddy, D Cournapeau, E Burovski, P Peterson, W Weckesser, J Bright, et al. Fundamental algorithms for scientific computing in python and scipy 1.0 contributors. scipy 1.0. Nat. Methods, 17: 0 261--272, 2020

    2020

  38. [46]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  39. [47]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  40. [48]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022

    arXiv

  41. [49]

    Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki

    Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Lingu...

    2020

  42. [50]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  43. [51]

    SQ u AD : 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQ u AD : 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages...

    2016 doi

  44. [52]

    Alpacaeval: An automatic evaluator of instruction-following models, 2023 b

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models, 2023 b

    2023

  45. [53]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  46. [54]

    Second-order forward-mode automatic differentiation for optimization

    Adam D Cobb, At l m G \"u ne s Baydin, Barak A Pearlmutter, and Susmit Jha. Second-order forward-mode automatic differentiation for optimization. arXiv preprint arXiv:2408.10419, 2024

    2024 arXiv

  47. [55]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1 0 (2): 0 3, 2022

    2022

  48. [56]

    Trak: Attributing model behavior at scale

    Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303.14186, 2023

    2023 arXiv

  49. [57]

    Sharpness-aware minimization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020

    2010 arXiv

  50. [58]

    Cram: A compression-aware minimizer

    Alexandra Peste, Adrian Vladu, Eldar Kurtic, Christoph H Lampert, and Dan Alistarh. Cram: A compression-aware minimizer. arXiv preprint arXiv:2207.14200, 2022

    2022 arXiv

  51. [59]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25 0 (70): 0 1--53, 2024

    2024

  52. [60]

    o pf, Yannic Kilcher, Dimitri Von R \

    Andreas K \"o pf, Yannic Kilcher, Dimitri Von R \"u tte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Rich \'a rd Nagyfi, et al. Openassistant conversations-democratizing large language model alignment. Advances in Neural Info...

    2023

  53. [61]

    Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023

    2023

  54. [62]

    Instruction tuning with gpt-4

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023

    2023 arXiv

  55. [63]

    Code alpaca: An instruction-following llama model for code generation, 2023

    Sahil Chaudhary. Code alpaca: An instruction-following llama model for code generation, 2023

    2023

  56. [64]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36: 0 55006--55021, 2023 b

    2023

  57. [65]

    Wizardlm: Empowering large language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023

    2023 arXiv

  58. [66]

    Openorca: An open dataset of gpt augmented flan reasoning traces, 2023

    Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and “Teknium”. Openorca: An open dataset of gpt augmented flan reasoning traces, 2023

    2023

  59. [67]

    Sciriff: A resource to enhance language model instruction-following over scientific literature

    David Wadden, Kejian Shi, Jacob Morrison, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, et al. Sciriff: A resource to enhance language model instruction-following over scientific literature. arXiv preprint arXiv:2406.07835, 2024

    2024

  60. [68]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

    2023 arXiv

  61. [69]

    Nv-embed: Improved techniques for training llms as generalist embedding models

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428, 2024

    2024 arXiv

  62. [70]

    Large dual encoders are generalizable retrievers

    Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hern \'a ndez \'A brego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming-Wei Chang, et al. Large dual encoders are generalizable retrievers. arXiv preprint arXiv:2112.07899, 2021

    2021 arXiv

  63. [71]

    Galore: Memory-efficient llm training by gradient low-rank projection

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024

    2024 arXiv

  64. [72]

    Hadacore: Tensor core accelerated hadamard transform kernel

    Krish Agarwal, Rishi Astra, Adnan Hoque, Mudhakar Srivatsa, Raghu Ganti, Less Wright, and Sijia Chen. Hadacore: Tensor core accelerated hadamard transform kernel. arXiv preprint arXiv:2412.08832, 2024

    2024 arXiv

  65. [73]

    Fast hadamard transform in cuda, with a pytorch interface, 2023

    Tri Dao. Fast hadamard transform in cuda, with a pytorch interface, 2023. URL https://github.com/Dao-AILab/fast-hadamard-transform

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.