Pith. sign in

REVIEW 3 major objections 4 minor 52 references

Disentangling the Roles of Representation and Selection in Data Pruning

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that data representation, not the selection algorithm, drives data pruning: gradients beat losses and hidden states at choosing useful training instances, and several heuristics miss their own stated objective.

desk verdict Solid empirical study of data pruning decompositions; the theory for why gradients win has a load-bearing proof bug (monotonicity of an upper bound), but the findings are worth reviewing. read the letter →

arxiv 2507.03648 v1 pith:CPVIRAVN submitted 2025-07-04 cs.CL cs.LG

Yupei Du , Yingjin Song , Hugh Mee Wong , Daniil Ignatev , Albert Gatt , Dong Nguyen This is my paper
classification cs.CLcs.LG
keywords datapruningselectioninstancetraininggradientsrepresentationsalgorithmsNLPfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Data pruning — keeping a small training subset that preserves most of a model's performance — is usually evaluated as a single method, but this paper separates it into two decisions: how each training example is represented (training dynamics, hidden states, or gradients) and how a selection algorithm uses those representations (to maximize difficulty, diversity, or relevance to validation data). Its central claim is that the representation dominates: the same algorithm fed different representations produces more divergent selections than different algorithms sharing one representation, and the theoretically strongest representation — the training gradient — is also the one that performs best across tasks and budgets. Its second claim is that selection algorithms are unreliable guides to their own objectives: two difficulty-maximizing heuristics can choose nearly disjoint subsets, and on a synthetic task a difficulty heuristic visibly prefers easy, confidently classified examples. If these claims hold, data pruning research should concentrate effort on cheap, informative representations and on auditing whether a heuristic does what its name promises, since the selection step itself contributes less to the outcome.

What carries the argument

The carrying mechanism is the pairing of a representation with a selection algorithm, and the metric that exposes their relative weight is the overlap ratio between two selected subsets — at a 30% budget, random selection yields a ratio of about $0.3$, so any pair of methods far above or below that value is genuinely sharing or avoiding the same instances. The theoretical core is a last-layer geometry: writing $z = y\,w^\top h(x)$ for the signed distance of an instance to the decision boundary, the gradient of the binary cross-entropy loss is $-\frac{y h(x)}{1+e^z}$, which yields a squared gradient distance $D_g = \lVert \frac{y_i h(x_i)}{1+e^{z_i}} - \frac{y_j h(x_j)}{1+e^{z_j}} \rVert^2$ that is label-aware and boundary-weighted, a loss distance $D_\ell = \left( \log \frac{1+e^{-z_i}}{1+e^{-z_j}} \right)^2$ that is boundary-weighted but label-blind, and a hidden-state distance that is neither. The paper adds three criteria for a good representation (respect distance to the boundary, encode label information, and discriminate most sharply near the boundary) and proves that the gradient's discriminative power — the spectral norm of its Jacobian with respect to hidden states — peaks near the decision boundary once $\alpha = \lVert w \rVert \lVert h(x_i) \rVert$ exceeds about $1.544$, while the loss's discriminative power decays monotonically with confidence and the hidden state's is constant. These identities are what let the paper predict, before running models, that gradients should select more diverse and non-redundant instances than losses or hidden states.

What would settle it

A direct falsifier is a controlled crossing of three representations and three selection algorithms at 5-30% budgets on a large benchmark, asking whether any non-gradient representation with the same algorithm consistently beats the gradient version, and whether swapping representations moves final model performance more than swapping algorithms as measured by the pairwise overlap ratio. If the overlap pattern inverts — same-algorithm pairs overlapping more than same-representation pairs — the central claim fails. A cheaper variant targets the theory directly: measure the Jacobian discriminative power of the loss and gradient on a real multi-class or generation model's last layer; if $C_\ell$ and $C_g$ no longer peak near the decision boundary for softmax or per-token losses, the theoretical transfer that anchors the ordering is broken.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a decomposition plus an ordering. After splitting data pruning into a representation step and a selection step, the paper derives, for the classification layer of a binary classifier with binary cross-entropy loss, explicit distance identities: the loss distance is $D_\ell = \left( \log \frac{1+e^{-z_i}}{1+e^{-z_j}} \right)^2$ and the gradient distance is $D_g = \lVert \frac{y_i h(x_i)}{1+e^{z_i}} - \frac{y_j h(x_j)}{1+e^{z_j}} \rVert^2$, where $z = y\,w^\top h(x)$ is the signed distance to the decision boundary. The identities show that losses encode boundary distance, gradients additionally encode label agreement, and the gradient's discriminative power peaks near the decision boundary once $\alpha = \lVert w \rVert \lVert h(x_i) \rVert$ passes about $1.544$, while hidden-state distances are blind to both. The empirical companion claims are that representations move the selected subset more than algorithms do (same-representation pairs overlap more than same-algorithm pairs, with random overlap at roughly $0.3$ for a 30% budget), and that this carries through to model performance: gradient-based methods such as LESS and self-influence are competitive across tasks and budgets, while hidden-state-based methods track random selection and difficulty-based methods need large budgets.

Load-bearing premise

The paper's theoretical ranking — gradients above losses above hidden states — is proven only for the classification layer of a binary classifier trained with binary cross-entropy loss, and the argument assumes without a derivation that the same ordering carries over to multi-class classification and sequence generation, where the experiments are actually run.

Editorial extensions

If this is right

  • Future pruning research should fix the representation and vary algorithms second, because the representation is what mostly decides the selected subset.
  • Gradient-based methods (LESS, self-influence) should be the default choice for fine-tuning across tasks and budgets, while hidden-state-based methods (Prototypicality, SemDeDup) should not be expected to beat random selection on task-specific fine-tuning.
  • Difficulty-maximizing algorithms need auditing: Hard-to-Learn and Prototypicality both claim 'difficulty' yet select nearly disjoint instances, and Prototypicality on hidden states visibly favors easy, confidently correct examples.
  • Budget and distribution context decide the algorithm: relevance-based selection wins when train and test distributions diverge (DynaHate) and needs label matching on imbalanced data, while difficulty-based selection only pays off at budgets above roughly 30%.
  • Pretrained hidden states carry too little task-specific signal for pruning fine-tuning data, and fine-tuning them (early or late checkpoints) does not fix the gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit consequence the paper does not spell out: if representations dominate selections, then the real cost bottleneck of data pruning is representation computation, so cheap gradient surrogates (e.g., approximate influence from a few checkpoints, as the paper already uses) are the lever that decides whether pruning pays off at scale.
  • A testable extension of the theory: the three-criteria framework (boundary awareness, label awareness, boundary-adjacent discriminative power) could be measured directly for multi-class softmax and per-token generation losses; if gradients lose their boundary peak there, the empirical gradient advantage would need a different explanation than the one the paper gives.
  • A likely consequence for prior benchmarks: published comparisons that vary selection algorithms while accidentally varying representations may be ranking the wrong axis; re-running those comparisons with a single fixed representation could reshuffle the ordering of methods.
  • A cheap audit tool suggested by the paper's method: report the overlap ratio against random selection (about 0.3 at a 30% budget) whenever a new pruning method is introduced, since near-random overlap signals that the method's stated objective is not driving its selections.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper decomposes data pruning into two components—data representation and selection algorithm—and studies their relative influence through theory and experiments. The theoretical part (§3.1) analyzes, for binary logistic classification on the final layer, properties of distances based on hidden states, losses, and gradients, concluding that gradients are more discriminative near the decision boundary. The empirical part (§3.2, §4) evaluates six pruning methods across classification and generation tasks, reporting that representations, especially gradients, dominate selection algorithms in determining selected instances and downstream performance, and that selection algorithms often fail to follow their intended objectives.

Significance. If established, the paper's conclusion—that the choice of representation matters more than the choice of selection algorithm—would be a practically important and field-relevant message for data pruning. The paper contributes a clean conceptual decomposition, a taxonomy of existing methods, a large controlled empirical study (1200+ trained models, three seeds), public code, and several theory-derived hypotheses that are in principle falsifiable. The observation that algorithms designed for the same objective select very different instances is also valuable. The theoretical derivation in §3.1 is presented as parameter-free and is not fitted to the data, which is a strength. However, the central proof of the gradient-representation advantage has a gap, and the scope of the theory does not yet cover the multi-class and generation settings in which the headline claims are tested.

major comments (3)
  1. [§3.1, Appendix D.3, Corollary 3.4] The proof of Corollary 3.4 differentiates the upper bound on C_g from Theorem 3.3, not the exact spectral norm. Specifically, Eq. (22) takes the derivative of the right-hand side of the inequality C_g ≤ 1/(1+e^{z_i}) + e^{z_i}/(1+e^{z_i})^2 ||h||||w||, and the subsequent sign analysis concerns that bound. Monotonicity of an upper bound does not imply monotonicity of the bounded quantity, so the claimed boundary-peaked behavior of the exact C_g is not established. This is load-bearing because the theoretical explanation for why gradients outperform losses and hidden states rests on the peak near the decision boundary. The authors should either compute the exact spectral norm of the matrix in Eq. (17) and prove the claimed behavior, or present a different rigorous argument. If the exact norm does behave as claimed, this is a fixable gap rather than a fatal error.
  2. [§3.1, footnote 4, and §3.2/§4] The theoretical analysis is explicitly limited to binary cross-entropy on the classification layer, but the empirical validation is conducted on multi-class classification (CAD, WinoGrande with DeBERTaV3) and sequence generation (DialogSum with OPT), and the gradient representations in the experiments are LoRA gradients over all linear layers, not just the classification layer. Footnote 4 asserts that the analysis 'can be extended' to multi-class or generation by considering a specific class or token, but no derivation is provided, and no argument connects the classification-layer result to LoRA gradients. Since the abstract states that 'better representations, e.g., training gradients, generally lead to a better selection of instances, regardless of the chosen selection algorithm,' this transfer is part of the central claim. The authors should either provide the corresponding analysis for the settings actually tested or explicitly reframe the theoretical claim as supporting intuition for a binary-logistic setting and let the empirical results carry the general claim.
  3. [§4, Figures 3–6] The performance comparisons are reported without error bars, confidence intervals, or significance tests, despite the stated use of three random seeds. This makes it difficult to assess claims such as 'hidden-state-based methods perform worse than or similar to random selection on all tasks' and 'gradient-based methods like LESS and self-influence perform competitively across most tasks,' especially where the plotted curves are close. The authors should display seed-level variance or aggregate statistics and, where appropriate, perform tests of whether observed differences are meaningful relative to the random-selection baseline. Without this, the empirical component of the central claim is not fully supported.
minor comments (4)
  1. [Appendix D.2, Theorem D.2] The appendix restatement gives C_ℓ = e^{z_i}/(1+e^{z_i}) ||w||, which contradicts the main-text Theorem 3.3 where C_ℓ = 1/(1+e^{z_i}) ||w||. This appears to be a typo and should be corrected.
  2. [§4 Setup] The text states 'six different data budgets 5%, 15%, 30%, 50%, and 70%' but only five budgets are listed; the intended set should be clarified.
  3. [§3.2, last paragraph] There is a typo: 'the the sensitivity of selection algorithms' should read 'the sensitivity of selection algorithms.'
  4. [§3.2, Figure 2b] The overlap-ratio matrices are reported without any indication of how the three seeds are aggregated or whether the displayed values are stable across seeds; a short explanation or error estimates would strengthen the reproducibility of the claim that representations matter more than selection algorithms.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical results are algebraic derivations from stated definitions, the empirical comparisons are run on held-out performance and overlap ratios with a computed random baseline, and the only self-citation is not load-bearing.

full rationale

I walked the paper's derivation chain and found no step that reduces to its own inputs. In §3.1, the claimed properties of representations (Remark 3.1, Theorem 3.3, Corollary 3.4) are obtained by differentiating and manipulating the explicit expressions for loss and gradient distances; these are parameter-free derivations from the binary logistic-regression setup, not fits. The 'good representation' criteria are proposed and justified by external results (representer theorem, diversity arguments) rather than being defined as 'whatever gradients have', so the later conclusion that gradients meet more criteria is not self-definitional. The empirical claims in §3.2 and §4 are based on overlap ratios between actually selected subsets and on downstream model performance, with the random-selection overlap baseline (0.3) computed from the budget; no fitted parameter is renamed as a prediction. The one self-citation (Du et al., 2025) appears only in the introduction as one of several examples of data pruning and is not used to justify the paper's central claim, so it is not load-bearing. I also checked the skeptic-flagged issue: the proof of Corollary 3.4 differentiates the upper bound on C_g rather than the exact spectral norm, which is a mathematical validity concern about the theoretical argument, but it is not circularity because the bound is not assumed to equal C_g. Similarly, the extension from binary classification to multi-class/generation is stated explicitly as an assumption in footnote 4, not derived from the binary analysis. These are correctness or scope risks, not circular steps. The paper's conclusions therefore have independent empirical and derivational content, and no circularity score above 0 is warranted.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central theoretical claim (gradients are more discriminative than losses and hidden states) rests on explicit modeling choices: binary logistic classification layer, Euclidean distance between representations, spectral norm of the Jacobian as discriminative power, and the proxy that the reference model's decision boundary approximates the final model's. These are stated in the paper, but they are assumptions rather than derived facts. The empirical comparison adds hyperparameters chosen by search (cluster counts, projection dimension, checkpoint count), which are free parameters of the experiments. No invented entities are introduced.

free parameters (4)
  • number of k-means clusters = 100 (CAD, DialogSum), 200 (WinoGrande)
    Selected by hyperparameter search; affects S2L, Prototypicality, and SemDedup. The paper reports results are consistent across cluster counts, so this is not load-bearing.
  • gradient projection dimensionality = 1024
    Searched over 1024/2048/4096 on smaller models with one seed; chosen for efficiency. Not central to the representation ranking.
  • number of gradient checkpoints = first five
    Searched; first five were most consistent with all checkpoints (Spearman rank correlation at least 0.96).
  • training epochs = 15
    Searched 3 to 25 on smaller models; best at 15 across datasets. Standard experimental choice, not fitted to the claim.
assumptions (6)
  • domain assumption Binary logistic regression on the classification layer is sufficient to analyze representations, and the results extend to multi-class and generation tasks.
    Invoked in §3.1 and footnote 4. All empirical claims are tested on DeBERTaV3 and OPT with softmax and sequence losses, so the extension is load-bearing but unproven.
  • domain assumption The decision boundary of the final trained model can be approximated by the reference model's decision boundary.
    Footnote 6 in §3.1. The three criteria for good representations (close to boundary, non-redundant, diverse) rely on this proxy.
  • ad hoc to paper Good pruning should select non-redundant instances close to the decision boundary while preserving diversity.
    §3.1 'What makes a good representation?'. These criteria are proposed to justify why gradients are better; they are not derived from first principles.
  • ad hoc to paper Discriminative power is correctly measured by the spectral norm of the Jacobian of the representation with respect to hidden states.
    Definition 3.2. The choice of metric determines the theoretical ranking; other metrics could give different rankings.
  • domain assumption Three representation sources (training dynamics, hidden states, gradients) and three selection objectives (difficulty, diversity, relevance) span the space of studied pruning methods.
    §2. LLM-prompting methods are excluded in footnote 2, so the coverage is not complete.
  • standard math Representer theorem and long-tail memorization results from the literature justify focusing on decision-boundary instances and hard examples.
    Used in §3.1 (Yeh et al., 2018) and in the Self-Influence description (Feldman and Zhang, 2020).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Disentangling the Roles of Representation and Selection in Data Pruning." pith.science (2026). https://pith.science/paper/CPVIRAVN

@misc{pith2026250703648,
  author       = {Pith},
  title        = {Pith review of: Disentangling the Roles of Representation and Selection in Data Pruning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CPVIRAVN}},
  note         = {Machine review of arXiv:2507.03648}
}
read the original abstract

Data pruning, selecting small but impactful subsets, offers a promising way to efficiently scale NLP model training. However, existing methods often involve many different design choices, which have not been systematically studied. This limits future developments. In this work, we decompose data pruning into two key components: the data representation and the selection algorithm, and we systematically analyze their influence on the selection of instances. Our theoretical and empirical results highlight the crucial role of representations: better representations, e.g., training gradients, generally lead to a better selection of instances, regardless of the chosen selection algorithm. Furthermore, different selection algorithms excel in different settings, and none consistently outperforms the others. Moreover, the selection algorithms do not always align with their intended objectives: for example, algorithms designed for the same objective can select drastically different instances, highlighting the need for careful evaluation.

Figures

Figures reproduced from arXiv: 2507.03648 by the authors.

Figure 1
Figure 1. (Min-max normalized) discriminative power [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. The consistency of selections across different representations and selection algorithms. (a) Synthetic data: [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Results with DeBERTaV3Large and OPT-350M models. (a)–(d) show performance across data budgets, (e) presents label distributions, while (f) and (g) compare pretrained vs. fine-tuned hidden states, and (h) and (i) examine representation variation for WinoGrande and DynaHate. only the hardest instances will make models fail to converge. Second, gradient-based methods like LESS and self-influence perform competitively a… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Model performance under different data budgets. [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: Ablation studies on pretrained vs. fine-tuned hidden states. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Ablation studies on using different data representations with the same selection algorithm. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 32 canonical work pages

  1. [1]

    Amro Kamal Mohamed Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli, and Ari S. Morcos. 2023. https://openreview.net/forum?id=4vlGm9gv6c Semdedup: Data-efficient learning at web-scale through semantic deduplication . In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models

    work page 2023

  2. [2]

    Zeyuan Allen-Zhu and Yuanzhi Li. 2024. https://openreview.net/forum?id=5x788rqbcj Physics of language models: Part 3.1, knowledge storage and extraction . In Forty-first International Conference on Machine Learning

    work page 2024

  3. [3]

    Robert John Nicholas Baldock, Hartmut Maennel, and Behnam Neyshabur. 2021. https://openreview.net/forum?id=WWRBHhH158K Deep learning through the lens of example difficulty . In Advances in Neural Information Processing Systems

    work page 2021

  4. [4]

    Irina Bejan, Artem Sokolov, and Katja Filippova. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.625 Make every example count: On the stability and utility of self-influence for learning from noisy NLP datasets . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10107--10121, Singapore. Association for Comp...

    doi:10.18653/v1/2023.emnlp-main.625 2023

  5. [5]

    Rajat Bhatnagar, Ananya Ganesh, and Katharina Kann. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.540 CHIA : CH oosing instances to annotate for machine translation . In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7299--7315, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics

    work page doi:10.18653/v1/2022.findings-emnlp.540 2022

  6. [6]

    Sérgio Canuto, Daniel Xavier Sousa, Marcos André Gonçalves, and Thierson Couto Rosa. 2018. https://doi.org/10.1109/TKDE.2018.2820051 A thorough evaluation of distance-based meta-features for automated text classification . IEEE Transactions on Knowledge and Data Engineering, 30(12):2242--2256

    work page arXiv 2018

  7. [7]

    Joel Luis Carbonera and Mara Abel. 2015. https://doi.org/10.1109/ICTAI.2015.114 A density-based approach for instance selection . In 2015 IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI), pages 768--774

    work page doi:10.1109/ictai.2015.114 2015

  8. [8]

    Joel Luis Carbonera and Mara Abel. 2016. https://doi.org/10.1109/ICTAI.2016.0090 A novel density-based approach for instance selection . In 2016 IEEE 28th International Conference on Tools with Artificial Intelligence (ICTAI), pages 549--556

    work page arXiv 2016

Show all 52 references
  1. [9]

    Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. 2024. https://openreview.net/forum?id=FdVXgSJhvz Alpagasus: Training a better alpaca with fewer data . In The Twelfth Internationa...

    2024

  2. [10]

    Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. https://doi.org/10.18653/v1/2021.findings-acl.449 D ialog S um: A real-life scenario dialogue summarization dataset . In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5062--5074, Online...

    2021 doi

  3. [11]

    Yupei Du, Albert Gatt, and Dong Nguyen. 2025. https://aclanthology.org/2025.coling-main.86/ FTFT : Efficient and robust fine-tuning by transferring training dynamics . In Proceedings of the 31st International Conference on Computational Linguistics, pages 1294--1308, Abu Dhabi...

    2025

  4. [12]

    Logan Engstrom, Axel Feldmann, and Aleksander Madry. 2024. https://openreview.net/forum?id=GC8HkKeH8s Dsdm: Model-aware dataset selection with datamodels . In Forty-first International Conference on Machine Learning

    2024

  5. [13]

    Vitaly Feldman. 2020. https://doi.org/10.1145/3357713.3384290 Does learning require memorization? a short tale about a long tail . In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2020, page 954–959, New York, NY, USA. Association for Computi...

    2020

  6. [14]

    Vitaly Feldman and Chiyuan Zhang. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/1e14bfe2714193e7af5abc64ecbd6b46-Paper.pdf What neural networks memorize and why: Discovering the long tail via influence estimation . In Advances in Neural Information Processin...

    2020

  7. [15]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. https://openreview.net/forum?id=sE7-XhLxHA De BERT av3: Improving de BERT a using ELECTRA -style pre-training with gradient-disentangled embedding sharing . In The Eleventh International Conference on Learning Representations

    2023

  8. [16]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  9. [17]

    Osman Batur \.I nce, Tanin Zeraati, Semih Yagcioglu, Yadollah Yaghoobzadeh, Erkut Erdem, and Aykut Erdem. 2023. https://openreview.net/forum?id=BMIjPXooNq Harnessing dataset cartography for improved compositional generalization in transformers . In The 2023 Conference on Empir...

    2023

  10. [18]

    Smith, Iz Beltagy, and Hannaneh Hajishirzi

    Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. https://arxiv.org/abs/2311.10702 Camels in a changing climate: Enhancing lm adaptation with tulu 2...

    2023 arXiv

  11. [19]

    Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. 2021. https://proceedings.mlr.press/v139/jiang21k.html Characterizing structural regularities of labeled data in overparameterized models . In Proceedings of the 38th International Conference on Machine Learning, ...

    2021

  12. [20]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  13. [21]

    Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adi...

    2021 doi

  14. [22]

    Pang Wei Koh and Percy Liang. 2017. https://proceedings.mlr.press/v70/koh17a.html Understanding black-box predictions via influence functions . In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages...

    2017

  15. [23]

    Devin Kwok, Nikhil Anand, Jonathan Frankle, Gintare Karolina Dziugaite, and David Rolnick. 2024. https://arxiv.org/abs/2401.01867 Dataset difficulty and the role of inductive bias . Preprint, arXiv:2401.01867

    2024 arXiv

  16. [24]

    Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. 2024. https://aclanthology.org/2024.naacl-long.421 From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning . In P...

    2024

  17. [25]

    Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. 2024. https://openreview.net/forum?id=BTKAeLqLMw What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning . In The Twelfth International Conference on Learning Repre...

    2024

  18. [26]

    Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2024. https://openreview.net/forum?id=pszewhybU9 \#instag: Instruction tagging for analyzing supervised fine-tuning of large language models . In The Twelfth International Co...

    2024

  19. [27]

    Mohamed Malhat, Mohamed El Menshawy, Hamdy Mousa, and Ashraf El Sisi. 2020. https://doi.org/10.1016/j.eswa.2020.113297 A new approach for instance selection: Algorithms, evaluation, and comparisons . Expert Systems with Applications, 149:113297

    2020

  20. [28]

    Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. 2023. https://arxiv.org/abs/2309.04564 When less is more: Investigating data pruning for pretraining llms at scale . Preprint, arXiv:2309.04564

    2023 arXiv

  21. [29]

    o ren Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt H \

    S \"o ren Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt H \"o ltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. 2022. https://proceedings.mlr.press/v162/mindermann22a.html Prioritized training on points...

    2022

  22. [30]

    Moore and William Lewis

    Robert C. Moore and William Lewis. 2010. https://aclanthology.org/P10-2041/ Intelligent selection of language model training data . In Proceedings of the ACL 2010 Conference Short Papers , pages 220--224, Uppsala, Sweden. Association for Computational Linguistics

    2010

  23. [31]

    Patrik Okanovic, Roger Waleffe, Vasilis Mageirakos, Konstantinos Nikolakakis, Amin Karbasi, Dionysios Kalogerias, Nezihe Merve G \"u rel, and Theodoros Rekatsinas. 2024. https://openreview.net/forum?id=JnRStoIuTe Repeated random sampling for minimizing the time-to-accuracy of ...

    2024

  24. [32]

    Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. 2023. https://proceedings.mlr.press/v202/park23c.html TRAK : Attributing model behavior at scale . In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Pr...

    2023

  25. [33]

    Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. 2021. https://openreview.net/forum?id=Uj7pF-D-YvT Deep learning on a data diet: Finding important examples early in training . In Advances in Neural Information Processing Systems

    2021

  26. [34]

    Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/e6385d39ec9394f2f3a354d9d2b88eec-Paper.pdf Estimating training data influence by tracing gradient descent . In Advances in Neural Information Pr...

    2020

  27. [35]

    Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng

    Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H. Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. 2024. https://arxiv.org/abs/2402.09668 How to train data-efficient llms . Preprint, arXiv:2402.09668

    2024 arXiv

  28. [36]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. https://doi.org/10.1145/3474381 Winogrande: an adversarial winograd schema challenge at scale . Commun. ACM, 64(9):99–106

    2021 doi

  29. [37]

    Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. 2024. https://openreview.net/forum?id=0bmXrtTDUu Beyond chinchilla-optimal: Accounting for inference in language model scaling laws . In Forty-first International Conference on Machine Learning

    2024

  30. [38]

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. 2022. https://openreview.net/forum?id=UmvSlP-PyV Beyond neural scaling laws: beating power law scaling via data pruning . In Advances in Neural Information Processing Systems

    2022

  31. [39]

    Smith, and Yejin Choi

    Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.746 Dataset cartography: Mapping and diagnosing datasets with training dynamics . In Proceedings of the 2020 C...

    2020 doi

  32. [40]

    Megh Thakkar, Tolga Bolukbasi, Sriram Ganapathy, Shikhar Vashishth, Sarath Chandar, and Partha Talukdar. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.125 Self-influence guided data reweighting for language model pre-training . In Proceedings of the 2023 Conference on Empi...

    2023 doi

  33. [41]

    Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/a8f8cbd7f7a5fb2c837e578c75e5b615-Paper-Datasets_and_Benchmarks.pdf D4: Improving llm pretraining via document de-duplication and diversification . ...

    2023

  34. [42]

    Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. 2019. https://openreview.net/forum?id=BJlxm30cKm An empirical study of example forgetting during deep neural network learning . In International Conference on Lear...

    2019

  35. [43]

    Bertie Vidgen, Dong Nguyen, Helen Margetts, Patricia Rossini, and Rebekah Tromble. 2021. https://doi.org/10.18653/v1/2021.naacl-main.182 Introducing CAD : the contextual abuse dataset . In Proceedings of the 2021 Conference of the North American Chapter of the Association for ...

    2021 doi

  36. [44]

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. https://openreview.net/forum?id=Kw3ckB2Kfc LESS : Selecting influential data for targeted instruction tuning . In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models

    2024

  37. [45]

    Yu Yang, Siddhartha Mishra, Jeffrey N Chiang, and Baharan Mirzasoleiman. 2024. https://openreview.net/forum?id=K9IGlMQpif Smalltolarge (s2l): Scalable data selection for fine-tuning large language models by summarizing training trajectories of small models . In The Thirty-eigh...

    2024

  38. [46]

    Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. 2018. https://proceedings.neurips.cc/paper_files/paper/2018/file/8a7129b8f3edd95b7d969dfc2c8e9d9d-Paper.pdf Representer point selection for explaining deep neural networks . In Advances in Neural Information Pro...

    2018

  39. [47]

    Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. 2024. https://openreview.net/forum?id=5HCnKDeTws When scaling meets LLM finetuning: The effect of data, model and finetuning method . In The Twelfth International Conference on Learning Representations

    2024

  40. [48]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  41. [49]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://openreview.net/forum?id=uccHPGDlao Judging LLM -as-a-judge with MT -bench and chatbot ...

    2023

  42. [50]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  43. [51]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  44. [52]

    However, it remains unclear how different design choices of these methods impact their effectiveness, because previous studies typically treat them as cohesive units

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.