Pith. sign in

REVIEW 4 major objections 6 minor 45 references

CLUES: Collaborative High-Quality Data Selection for LLMs via Training Dynamics

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that collaborative fine-tuning of LLMs can be quality-controlled by scoring each private sample's influence on a small public anchor set, and that this beats local filtering baselines.

desk verdict Useful collaborative data-curation pipeline, but the headline numbers overclaim: the scoring is uncited TracIn, the Oracle is beaten, and the anchor-based threshold is unvalidated. read the letter →

arxiv 2507.03004 v1 pith:QMQKSZ6J submitted 2025-07-02 cs.CL cs.MA

Wanru Zhao , Hongxiang Fan , Shell Xu Hu , Wangchunshu Zhou , Bofan Chen , Nicholas D. Lane This is my paper
classification cs.CLcs.MA
keywords dataselectiontrainingdynamicsinfluencecollaborativefine-tuningfederatedlearningmodelmerginglow-rankadaptationanchor
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CLUES tries to establish that data quality for collaborative LLM fine-tuning can be judged without sharing raw data, by measuring how each private training sample moves the model along the same trajectory as a small public 'anchor' set. The method scores every sample as the learning-rate-weighted inner product of its gradient updates with anchor-sample gradient updates across saved checkpoints, then applies a single global threshold equal to the average anchor score, so clients with different contamination levels keep different fractions of their data. In medical, multilingual, and financial QA experiments, training on the selected samples often outperforms per-client perplexity, IFD, and DataInf baselines, reaching over 96% of the oracle upper bound that trains only on known-clean data in federated settings. If correct, this gives a privacy-preserving data curation pipeline that fits into both model merging and federated learning with only server-side changes.

What carries the argument

The load-bearing object is the training-dynamics influence score, $S(z)=\sum_{z'}\sum_{t=1}^{T}\bar{\eta}_t\,\mathcal{L}(z',\theta_t)\cdot\mathcal{L}(z,\theta_t)$, where $\mathcal{L}$ is the Adam/AdamW gradient transform used for the update, $\theta_t$ are checkpoints saved during local fine-tuning, and the inner product is taken between the update direction of a private sample $z$ and that of a public validation/anchor sample $z'$. Gradients come from the first low-rank layer of LoRA adapters, which the paper argues avoids the cancellation effect that weakens last-layer influence scores. The second load-bearing piece is the global threshold: the server computes the same score for 10 public anchor samples and uses their average as a unified quality bar that each client applies independently, turning data selection into a server-side decision that needs no raw data and no client-specific selection ratio.

What would settle it

Take a client whose private domain is far from the public anchor domain and whose low-quality data are adversarial paraphrases, then vary the anchor set from 1 to 100 samples and across domains; record selection precision/recall and downstream QA score. If the global threshold swings enough to drop selection F1 below the best per-client baseline, the claim that one small anchor set defines a reliable unified standard fails.

Watch

Extended reading notes

Core claim

The central claim is that a training sample's quality is readable from its training dynamics relative to a public anchor set: high-quality private data move the model in directions that resemble the anchor data's directions during optimization, while corrupted or mismatched samples diverge. Concretely, CLUES computes a score for each private sample $z$ by summing, over checkpoints and over anchor/validation samples $z'$, the learning-rate-weighted inner product of the AdamW update direction for $z$ and for $z'$, and the paper uses this score $S(z)$ as the quality signal. The server sets the global cutoff as the average score of 10 public anchor samples; every client drops private samples below that cutoff before collaborative fine-tuning. Across medical, multilingual, and financial QA, the authors report that this pipeline outperforms local baselines, retains medical knowledge better than the baselines, adapts its threshold to different contamination ratios, and approaches the oracle that trains only on the known-clean subset.

Load-bearing premise

The whole method hinges on the global cutoff, which is the average influence score of only 10 public anchor samples; if those anchors are unrepresentative of a client's domain or contain one noisy sample, the threshold shifts and can silently discard good data or keep bad data.

Editorial extensions

If this is right

  • In the federated setting, selected data reaches more than 96% of the oracle upper bound on GPT-4 scoring across medical and multilingual QA, and more than 91% in model-merging settings.
  • CLUES beats per-client perplexity, IFD, and DataInf baselines on QA quality metrics while keeping knowledge-benchmark performance in an acceptable range.
  • The global threshold adapts to contamination level: selection accuracy stays above 99% with 20%, 50%, and 80% low-quality data.
  • On quality-heterogeneous FiQA, the anchor-derived global threshold beats ratio-based and pre-determined-score selection, with 98.39% F1 versus 95.84% and 79.17%.
  • Because filtering happens before training and only the threshold broadcast changes on the server, CLUES drops into existing federated averaging and model-merging pipelines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the score is computed against a public anchor set, CLUES could double as a federated data-quality audit: clients report only scores, and the server can rank domains by how far their data drift from the anchor.
  • The 10-sample anchor rule is a free parameter the paper does not vary; a direct extension would measure selection stability as anchor size and anchor domain change, and pick per-client thresholds from score quantiles instead of a single global mean.
  • The gradient-trace score also ranks samples by influence on the anchor loss, so the same machinery could identify poisoned or mislabeled samples for removal before and after training, not just filter before fine-tuning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes CLUES, a data-selection pipeline for collaborative fine-tuning of LLMs under model merging or federated learning. Client models are trained on mixed-quality private data, per-sample influence scores are computed from gradient inner products against a public validation set, and a global threshold is determined on the server from the average score of a small set of public anchor samples. Samples below the threshold are discarded, and the clients retrain on the filtered data before merging or federated aggregation. The authors evaluate on medical, multilingual, and financial QA tasks with Llama-2 and Mistral, comparing against perplexity, IFD, and DataInf baselines, and they report that CLUES often outperforms the baselines and approaches the Oracle upper bound. The manuscript also includes ablations on threshold rules, merging techniques, gradient layer choice, and varying pollution ratios, with code released publicly.

Significance. If the empirical claims were reliable, CLUES would be a useful contribution to collaborative data curation: it is a general, scalable influence-based selection method that respects data-sharing constraints, and it provides a unified treatment of model merging and federated learning. The release of code and the bi-level optimization interpretation are strengths. However, the current empirical support is not yet convincing because several reported results violate the paper's own theoretical upper bound, the 96%-of-Oracle claim is contradicted by the authors' tables, and the global threshold mechanism, which is the key novel component, is underspecified and lacks sensitivity analysis. These issues are load-bearing for the central claim and must be resolved before the results can be accepted.

major comments (4)
  1. [Section 4.2, Tables 1 and 2] The Oracle is defined in Section 4.1 as the theoretical upper bound obtained by training only on the remaining high-quality data, yet CLUES exceeds the Oracle in multiple cells. In Table 1 (Llama2, GPT-4 Scoring) CLUES achieves 0.210 while the Oracle achieves 0.099; in Table 2 (Llama2, Federated) CLUES achieves 0.477 while the Oracle achieves 0.449; and in Table 1 (Mistral, GPT-4 Scoring) CLUES achieves 0.161 while the Oracle achieves 0.160. Under the paper's own definition, this is internally inconsistent, and it indicates either that the Oracle construction is flawed (for example, the 'low-quality' pollution may not be harmful under the GPT-4 metric) or that the evaluation metric has enough variance to invert rankings. The paper reports no variance or significance measures for the GPT-4 scores, so the two possibilities cannot be distinguished. This directly undermines the 'over 96% of the theoretical upper bound' claim in Section 4.2.
  2. [Section 4.2, Table 1] The claim that CLUES achieves over 96% of the theoretical upper bound is not supported by Table 1. On MedicalQA with Llama2, the Knowledge Avg score for CLUES is 0.356 versus an Oracle of 0.440, which is 80.9%; with Mistral, CLUES is 0.309 versus Oracle 0.233, which is 132.6%. The 96% figure appears to apply only to the multilingual setting in Table 2, and even there the Llama2 federated row shows 0.477 versus an Oracle of 0.449, exceeding 100%. The authors should either revise the claim to state precisely which table and which metric support it, or report corrected numbers if the Oracle or the evaluation is erroneous.
  3. [Section 3.3, Algorithm 1] The computation of the global threshold is underspecified in a way that affects comparability of scores. In the model-merging setting (communication round cr=1), clients send only their final LoRA adapters to the server, but Algorithm 1 Stage 3 instructs the server to compute anchor scores using per-checkpoint gradients; no checkpoints are available on the server in this setting. In the federated setting, the paper does not state which global checkpoints are used for anchor scoring or how their time indices align with the client-side checkpoints used for sample scoring. If the anchor scores and client scores are computed under different model checkpoints or different training trajectories, their scales are not comparable, and the threshold τ can retain or discard nearly arbitrary fractions of each client's data. This is a load-bearing issue because the global threshold is the only mechanism that distinguishes CLUES from per-client scoring baselines.
  4. [Section 3.3, Table 3] No sensitivity analysis is provided for the anchor set size or anchor sampling, even though the global threshold is the mean score of only 10 public anchor samples. Table 3 demonstrates that the choice of threshold rule changes selection accuracy from 75.00% (select-by-ratio) to 95.00% (pre-determined score) to 97.91% (global threshold), so the threshold is the primary determinant of success. With n=10, a single unrepresentative anchor can shift τ enough to change selection decisions for many samples, and no confidence intervals are reported. The authors should include experiments varying the anchor set size and multiple random anchor draws to show that the threshold is stable.
minor comments (6)
  1. [Table 1 caption] The caption contains the typo 'Webold' instead of 'We bold', which should be corrected.
  2. [Table 1 caption and references] The baseline citations in the Table 1 caption are inconsistent with the reference list and with the text: IFD is cited as [25] in Section 2 but as [20] in the caption, and DataInf is cited as [22] in Section 2 but as [25] in the caption. The caption should use [25] for IFD and [22] for DataInf.
  3. [Section 3.2, Equations (3) and (6)] The notation for the learning-rate weight is inconsistent: Equation (3) uses \bar{\eta}_i while Equation (6) uses \eta_i, and the summation limits differ between T and earlier definitions of T. The authors should unify the notation and clarify that T denotes the number of checkpoints.
  4. [Section 3.2, Algorithm 1] The text says the gradient computation process is 'based on one single checkpoint,' but Algorithm 1 loops over checkpoints t in T for each training sample. This apparent contradiction should be resolved by stating explicitly whether the score aggregates over multiple checkpoints.
  5. [Algorithm 1, Stage 4] The selection condition is written as 'D'(k)_train = {zi in D(k)_train, zi >= tau}', which compares a data sample directly to a score threshold. It should read 'S(zi) >= tau' to compare the score of the sample, not the sample itself.
  6. [Section 4.1 and Appendix F] No standard deviations or repeated-run statistics are reported for any of the main tables, so the reader cannot assess whether the differences between CLUES and the baselines are within evaluation noise. Please add at least three seeds or bootstrap confidence intervals for the key comparisons.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the influence score is a proxy for the paper's own quality definition, but the evaluation is external (held-out GPT-4 scoring and manual pollution labels) and the threshold is calibrated on 10 anchor samples, not on the evaluation target.

full rationale

The paper's derivation chain is self-contained rather than circular. Data quality is defined via validation-loss reduction (Definition 1.1), and the CLUES score S(z) is a first-order influence approximation of that same validation-loss reduction. This is deliberate consistency, not circularity: the score is not literally equal to the validation-loss difference by construction, so selecting on S(z) is an approximate optimization of the stated objective rather than a tautology. The global threshold is the average score of 10 public anchor samples drawn from a high-quality held-out set (Section 3.3). This is a calibration procedure; the evaluation does not use those anchor scores as labels, and no parameter is fitted to the data-selection correctness metric or to the GPT-4 downstream scores. The Oracle baseline is trained on ground-truth high-quality data, so the 'over 96% of theoretical upper bound' claim is an external benchmark, not an artifact of the selection rule. The paper contains no load-bearing self-citations: the references cited for influence, IFD, DataInf, and training dynamics are all external prior work, and the authors do not invoke a self-derived uniqueness theorem to force the method. Concerns about anchor-size sensitivity, underspecification of server-side anchor scoring in Algorithm 1, and the strong resemblance of the scoring formula to existing influence methods (e.g., TracIn-style gradient inner products) are correctness, robustness, and attribution issues, not circularity. Therefore the required standard for flagging circularity—exhibiting a specific reduction of a claimed prediction to its fitted input or to a self-citation chain—is not met.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method depends on a small number of hand-picked design choices (anchor size 10, first-layer gradients) and several unproven assumptions about the relationship between training dynamics, anchor data, and data quality. No new physical or mathematical entities are introduced.

free parameters (2)
  • anchor set size = 10
    The global threshold is the average influence score of exactly 10 public anchor samples. The choice of 10 is not justified or varied in sensitivity analysis, and it directly controls how many samples are kept or dropped.
  • gradient layer index = first layer
    The influence score is computed using gradients from the first LoRA layer. The paper motivates this by a cancellation effect but does not systematically tune the layer choice; using the last layer reduces selection accuracy from 90.69% to 83.01%.
assumptions (4)
  • domain assumption High-quality data have similar training dynamics to the anchor dataset.
    This is the central premise of the method (Section 2.1 and Introduction), stating that samples inducing similar gradient trajectories as the anchor data are high quality. It is asserted, not proven.
  • standard math The sum of learning-rate-weighted gradient inner products over checkpoints approximates the contribution to validation loss reduction.
    Eq. 3 relies on the TracIn approximation (Pruthi et al., 2020) that influence can be estimated by inner products of updates; the paper does not derive this bound or its error.
  • domain assumption All participating models share the same architecture, initialization, and LoRA rank.
    Stated in the Limitations section: the method assumes homogeneous model architectures; the merging and scoring depend on this.
  • domain assumption The validation set used for scoring is representative of the test distribution.
    The anchor and validation data are held-out high-quality samples from the same source as the training and test splits, ensuring that selecting data to reduce anchor loss also improves test performance in this setup. This may not generalize to genuinely out-of-domain deployment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLUES: Collaborative High-Quality Data Selection for LLMs via Training Dynamics." pith.science (2026). https://pith.science/paper/QMQKSZ6J

@misc{pith2026250703004,
  author       = {Pith},
  title        = {Pith review of: CLUES: Collaborative High-Quality Data Selection for LLMs via Training Dynamics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QMQKSZ6J}},
  note         = {Machine review of arXiv:2507.03004}
}
read the original abstract

Recent research has highlighted the importance of data quality in scaling large language models (LLMs). However, automated data quality control faces unique challenges in collaborative settings where sharing is not allowed directly between data silos. To tackle this issue, this paper proposes a novel data quality control technique based on the notion of data influence on the training dynamics of LLMs, that high quality data are more likely to have similar training dynamics to the anchor dataset. We then leverage the influence of the training dynamics to select high-quality data from different private domains, with centralized model updates on the server side in a collaborative training fashion by either model merging or federated learning. As for the data quality indicator, we compute the per-sample gradients with respect to the private data and the anchor dataset, and use the trace of the accumulated inner products as a measurement of data quality. In addition, we develop a quality control evaluation tailored for collaborative settings with heterogeneous domain data. Experiments show that training on the high-quality data selected by our method can often outperform other data selection methods for collaborative fine-tuning of LLMs, across diverse private domain datasets, in medical, multilingual and financial settings. Our code is released at github.com/Ryan0v0/CLUES.

Figures

Figures reproduced from arXiv: 2507.03004 by the authors.

Figure 1
Figure 1. Validation loss and training loss. Definition 1.1 (Data Quality on Specific Domain k). Given a model architecture θ, a training configuration (optimizer, etc.), and a validation set Dval in a specific domain k, the quality of training data z is defined as follows: for z1, z2 ∈ Dtrain, if Lval(θ(z1), Dval) < Lval(θ(z2), Dval), then the quality of z1 is considered higher than that of z2. Here, Lval denotes the validat… view at source ↗
Figure 2
Figure 2. Performance drop on the performance of collaborative fine-tuning of LLMs when we change [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overall workflow diagram consists of two phases: 1) Step One: client-side computes each [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Experimental results for different levels of low-quality data [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Left: Comparison of different merging techniques. Right: First layer v.s. last layer for low-rank tracing gradient. Varying Merging Techniques [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 18 canonical work pages

  1. [1]

    Ensemble of averages: Im- proving model selection and boosting performance in domain generalization

    Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Im- proving model selection and boosting performance in domain generalization. arXiv preprint arXiv:2110.10832, 2021

    arXiv 2021

  2. [2]

    Swad: Domain generalization by seeking flat minima

    Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    work page 2021

  3. [3]

    Redpajama: an open dataset for training large language models, 2023

    Together Computer. Redpajama: an open dataset for training large language models, 2023

    work page 2023

  4. [4]

    Cold fusion: Collaborative descent for distributed multitask finetuning

    Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning. arXiv preprint arXiv:2212.01378, 2022

    arXiv 2022

  5. [5]

    fingpt-fiqa_qa

    FinGPT. fingpt-fiqa_qa. https://huggingface.co/datasets/FinGPT/fingpt-fiqa_qa,

    work page

  6. [6]

    Arcee’s mergekit: A toolkit for merging large language models

    Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee’s mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403.13257, 2024

    arXiv 2024

  7. [7]

    Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well

    Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste. Stochastic weight averaging in parallel: Large-batch training that generalizes well. arXiv preprint arXiv:2001.02312, 2020

    work page Pith review arXiv 2001

  8. [8]

    Medalpaca–an open-source collection of medical conversational ai models and training data

    Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. Medalpaca–an open-source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247, 2023

    arXiv 2023

Show all 45 references
  1. [9]

    Aligning ai with shared human values

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  2. [10]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  3. [11]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685, 2021

    2021 arXiv

  4. [12]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022

    2022 arXiv

  5. [13]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=6t0Kwf8-jrj

    2023

  6. [14]

    Averaging weights leads to wider optima and better generalization

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018. 11

    2018 arXiv

  7. [15]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  8. [16]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021

    2021

  9. [17]

    Pubmedqa: A dataset for biomedical research question answering

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146, 2019

    1909 arXiv

  10. [18]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09849, 2022

    2022 arXiv

  11. [19]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015

    2015

  12. [20]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885–1894. PMLR, 2017

    2017

  13. [21]

    Federated learning: Strategies for improving communication efficiency

    Jakub Koneˇcn`y, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016

    2016 arXiv

  14. [22]

    Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models

    Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models. arXiv preprint arXiv:2310.00902, 2023

    2023 arXiv

  15. [23]

    Deduplicating training data makes language models better

    Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021

    2021 arXiv

  16. [24]

    Branch-train-merge: Embarrassingly parallel training of expert language models

    Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208.03306, 2022

    2022 arXiv

  17. [25]

    From quantity to quality: Boosting llm performance with self- guided data selection for instruction tuning

    Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting llm performance with self- guided data selection for instruction tuning. ArXiv, abs/2308.12032, 2023

    2023 arXiv

  18. [26]

    Convergence analysis of two-layer neural networks with relu activation

    Yuanzhi Li and Yang Yuan. Convergence analysis of two-layer neural networks with relu activation. Advances in neural information processing systems, 30, 2017

    2017

  19. [27]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019

    2019

  20. [28]

    Peft: State-of-the-art parameter-efficient fine-tuning methods, 2022

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Ben- jamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods, 2022

    2022

  21. [29]

    Mondschein and Cosimo Monda

    Christopher F. Mondschein and Cosimo Monda. The EU’s General Data Protection Regulation (GDPR) in a Research Context, pages 55–71. Springer International Publishing, Cham, 2019. ISBN 978-3-319-99713-1. doi: 10.1007/978-3-319-99713-1_5

    2019 doi

  22. [30]

    Scaling data-constrained language models

    Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. arXiv preprint arXiv:2305.16264, 2023

    2023 arXiv

  23. [31]

    Gpt-4 technical report, 2023

    OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff B...

    2023

  24. [32]

    Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning, pages 248–260. PMLR, 2022

    2022

  25. [33]

    Pytorch: An imperative style, high- performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  26. [34]

    Towards building multilingual language model for medicine, 2024

    Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards building multilingual language model for medicine, 2024

    2024

  27. [35]

    Smith, and Yejin Choi

    Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with 13 training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language P...

    2020 doi

  28. [36]

    Luck matters: Understanding training dynamics of deep relu networks

    Yuandong Tian, Tina Jiang, Qucheng Gong, and Ari Morcos. Luck matters: Understanding training dynamics of deep relu networks. arXiv preprint arXiv:1905.13405, 2019

    1905 arXiv

  29. [37]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langua...

    2023

  30. [38]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023

  31. [39]

    Wilson, James L

    Charles L. Wilson, James L. Blue, and Omid M. Omidvar. Training dynamics and neural network performance. Neural Networks, 10(5):907–923, 1997. ISSN 0893-6080. doi: https: //doi.org/10.1016/S0893-6080(96)00119-0

    1997 doi

  32. [40]

    Transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  33. [41]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

    2022

  34. [42]

    Pmc-llama: Further finetuning llama on medical papers

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-llama: Further finetuning llama on medical papers. arXiv preprint arXiv:2304.14454, 2023

    2023 arXiv

  35. [43]

    TIES-merging: Resolving interference when merging models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging: Resolving interference when merging models. In Thirty-seventh Conference on Neural Informa- tion Processing Systems, 2023

    2023

  36. [44]

    surprised

    Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. Fingpt: Open-source financial large language models. arXiv preprint arXiv:2306.06031, 2023. 14 A Data Attribution Perplexity (PPL) serves as a fundamental metric in language modeling to measure the model’s ability to predic...

    2023

  37. [2023]

    Accessed: [Insert Access Date Here]

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.