Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimization

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that choosing a training-data mixture for a large language model is a black-box Bayesian optimization problem, and that treating it that way finds good mixtures for 7B models over five times faster than existing methods.

desk verdict Solid, useful BO-for-data-mixture paper with a valuable released dataset, but the headline speed-ups only hold over a finite precomputed candidate set, not the continuous simplex. read the letter →

arxiv 2508.11551 v2 pith:XQBSJW46 submitted 2025-08-15 stat.ML cs.AIcs.LG

Shengzhuang Chen , Xu Ouyang , Michael Arthur Leopold Pearce , Thomas Hartvigsen , Jonathan Richard Schwarz This is my paper
classification stat.MLcs.AIcs.LG
keywords datamixtureoptimizationBayesianmulti-fidelityGaussianprocessregressionlargelanguagemodelsinstructionfine-tuningre-weightingpre-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the best mix of training domains for a language model can be found by treating the mixture itself as a black-box objective and searching it with Bayesian optimization. Instead of fitting a fixed linear or exponential formula, ADMIRE-BayesOpt trains a Gaussian process that predicts validation performance for any mixture of source datasets, then picks the next training run where improvement is most promising. The multi-fidelity variant also chooses which model size to train, letting the algorithm spend most of its budget on cheap small models while still recommending a mixture for a much larger target model. On pre-training with The Pile and instruction fine-tuning with Tülu 3 / Qwen2.5 models, the recommended 7B mixtures match or beat those found by RegMix, DML, SVM, DoReMi, and random search while using substantially less simulated training cost, with reported speed-ups over 500% on the largest experiments. The paper also releases 460 full instruction fine-tuning runs so others can study mixture effects without repeating 13,000 GPU-hours of training.

What carries the argument

A Gaussian-process surrogate with a product kernel: an RBF kernel over the mixture simplex (similar mixtures yield similar performance) times a downsampling kernel over normalized parameter counts (larger models generally score better). Single-fidelity search uses Expected Improvement; multi-fidelity search uses Max-value Entropy Search with a cost-normalization factor so the algorithm can weigh the value of a cheap small-model run against a costly large-model run. At each step, the recommended mixture for the target model is the point on the high-fidelity candidate set with maximum posterior mean, so the GP supplies both prediction and uncertainty for exploration.

What would settle it

Take a small fixed corpus, define a synthetic validation score as a simple analytic function of the mixture weights so the true optimum is known and off-grid, and run the same acquisition loop restricted to a coarse Dirichlet sample. If the recommended mixture stays far from the true optimum no matter the budget, the finite-candidate restriction is deciding the result. A complementary check is to retrain the best-found 7B mixture from scratch and compare it against a mixture discovered by continuous simplex search under the same GPU budget.

Watch

Extended reading notes

Core claim

The central claim is that the relationship between training-data mixture weights and final validation performance can be modeled directly as a black-box function, making data-mixture selection a sequential decision problem. The authors fit a Gaussian process whose kernel separates two intuitions: an RBF kernel over the simplex of domain weights says similar mixtures give similar scores, while a downsampling kernel over normalized parameter counts encodes the expectation that larger models perform better. Expected Improvement drives single-fidelity search, and Max-value Entropy Search normalized by training cost drives the multi-fidelity variant, which selects both the next mixture and the ne

Load-bearing premise

Appendix A.1 says each acquisition step selects the point with the highest acquisition value from the pre-computed candidate mixtures in the training set, so the claimed optimal mixture is only optimal among those candidates; if the true best mixture is not in that initial sample, no amount of sequential search can reach it.

Editorial extensions

If this is right

  • Mixture optimization becomes a standard sequential decision problem: each new training run is selected to reduce uncertainty where it matters, so fewer runs are needed to find a good mixture.
  • Multi-fidelity scheduling automatically concentrates early budget on small proxy models, making data-mixture search for 7B-class models feasible on modest compute.
  • Because the surrogate is nonparametric, it can capture mixture-to-performance relationships that fixed linear or exponential law forms miss.
  • Zero-shot transfer from small to large models works best when the proxy is not too small; the measured transferability pattern motivates combining several fidelities instead of trusting a single proxy.
  • The released 460-run instruction fine-tuning dataset lets other groups test mixture-learning methods without rerunning 13,000 GPU-hours of training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit: the same product-kernel surrogate could treat training steps, context length, or data-quality filters as additional fidelities, since the downsampling kernel only needs a monotone scale feature.
  • A natural stress test would compare the finite-candidate acquisition used here against continuous optimization on the simplex; if continuous search improves the final recommendation, the density of the initial Dirichlet sample is the main bottleneck rather than the GP itself.
  • The ARD importance matrices suggest the number of influential source domains shrinks as model size grows; if that pattern holds, mixture search for very large models could restrict itself to a sparse active set of domains, reducing search dimensionality.
  • The released runs could be used to build a benchmark where the true optimal mixture is known by exhaustive enumeration on a small model, letting future methods measure regret directly rather than relative speed-up.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper casts LLM data-mixture selection as a black-box optimization problem and proposes Gaussian-process-based Bayesian optimization (ADMIRE-BayesOpt) and a multi-fidelity variant (ADMIRE-MFBO). The method models validation performance as a function of domain mixture weights, uses expected improvement or max-value entropy search to propose new mixtures, and recommends a mixture for a target model after a budget of training runs. Experiments are conducted on the RegMix Pile pre-training dataset and on a newly released Tülu-3 instruction-finetuning dataset of Qwen 0.5B/3B/7B runs. The authors report speed-ups over regression and random-search baselines, including a '>500%' speed-up claim in the abstract, and release the ADMIRE IFT Runs dataset.

Significance. If the claims were fully established, the paper would make a useful practical contribution: it applies a mature class of sequential decision-making algorithms to an expensive LLM problem, proposes a principled multi-fidelity formulation, and releases a substantial dataset of 460 full training/evaluation runs. The experimental protocol is mostly transparent, with five seeds and cumulative-best curves. However, the central claim of 'determining the best data mixture' is load-bearing and is not supported by the experiments as described, because all acquisition and recommendation are restricted to a pre-computed finite candidate set. The paper's value as a dataset release and as a demonstration of offline ranking over a fixed portfolio is solid, but the continuous-optimization claim needs to be either implemented or carefully rescoped.

major comments (3)
  1. [§4.2, Appendix A.1] The paper defines the optimization problem over the continuous simplex (§3) and describes acquisition by optimizing over S with gradient ascent (§4.2, Eq. (6) ff.). However, Appendix A.1 states that the acquisition function is optimized over the training set and the point with the highest acquisition value is selected; final recommendations are made by maximizing the posterior mean over the training set at the highest fidelity. Thus the experimental search space is the finite set of Dirichlet-sampled mixtures — 76 for 7B Tülu, 48 for 1B Pile, etc. — not the continuous simplex of §3. The abstract's 'determining the best data mixture' and the >500% speed-up claims are therefore established only over this fixed lookup table. If the initial sample misses the optimal region, no BO step can recover it. This is a load-bearing mismatch between the method as described and as evaluated. The author
  2. [§6 Implementation] The finite-set restriction applies to all methods: 'all methods recommend their best-performing data mixture within the candidate set of the target model.' This makes the comparison internally fair, but it changes the meaning of the results. The paper is comparing the order in which methods select from a precomputed set and the quality of their final selection, not the ability to discover a genuinely new mixture. The 'optimal mixture' language in §6.2 — 'the only algorithm to successfully find the true optimal mixture' — is unjustified, because 'optimal' is defined only relative to the candidate set. Please state this limitation prominently in the abstract, introduction, and conclusions.
  3. [§6.1.2 and Figure 5] The quantitative speed-up claims are reported as multiples (1.86x, 2.36x, 19x, 500%) but depend on the cost model and the stopping rule. For the Pile, cost is assumed to scale linearly with model size; for the Tülu dataset, average wall-clock time is used. The speed-up numbers also depend on the size and composition of the candidate set. A sensitivity analysis — e.g., varying the number of Dirichlet samples or using different random seeds for candidate construction — is needed to know whether the relative ranking of methods and the reported speed-ups are robust. Without this, the headline quantitative claims are fragile.
minor comments (5)
  1. [§5 and Appendix A.1] There is an inconsistency in the reported dataset size. §5 says 256 mixtures at 0.5B, 128 at 3B, and 76 at 7B, totaling 460. Appendix A.1 reports 256 + 128 + 60 = 444 samples. The abstract says 460, while the introduction mentions 'over 900 IFT runs'. Please reconcile these numbers.
  2. [§6 and Figure 2] The speed-up percentages in the introduction (195%, 1500%, and 'over 500%') do not obviously match the multipliers reported in §6.1.2 and §6.2. Please add a table that defines the cost metrics and the exact speed-up calculations, or align the prose.
  3. [§4.2, Eq. (9)] The cost-normalization factor c_x in the MES acquisition function is introduced but never defined. Please specify how it is computed for the three model sizes in the experiments.
  4. [§6] The text says results are averaged over 5 independent runs, but Figures 5 and 7 do not appear to show error bars or confidence intervals. Please add them, or state explicitly that only means are shown.
  5. [§5.2.1] The description says 256 mixtures were sampled with a Dirichlet prior using the Tülu-3-SFT weights, but the paper does not report the Dirichlet concentration parameter or the sampling procedure's seed. This information is needed for reproducibility of the candidate set.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GP predictions are evaluated against external benchmark runs; the finite candidate-set restriction is a scope/external-validity caveat, not a circular reduction.

full rationale

Walking the derivation chain: Section 3 defines the objective as minimizing true validation error f(x, M_T); Section 4.1 fits a GP surrogate to observed run outcomes; Section 4.2 selects queries by EI/MES; Section 4.3 recommends by maximizing posterior mean for the target fidelity; Section 6 evaluates the recommended mixture by looking up external benchmark performance from RegMix's Pile runs and the Tulu 3 runs. None of these steps is defined in terms of the quantity it is supposed to predict. The GP hyperparameters (length scales, noise, alpha, beta) are estimated by marginal likelihood from the run data, and the predictions are for target-fidelity mixtures not yet observed in zero-shot transfer or for recommending among candidates; the success metric is an external evaluation, not a fitted constant. The finite-candidate-set restriction in Appendix A.1 ('The acquisition function is optimized using ... over the training set, from which the point with the highest acquisition value is selected') is a genuine scope/external-validity caveat: the claimed optimum and >500% speed-ups are relative to the precomputed 76/48-point candidate sets, not the continuous simplex. But this is not circularity: within that finite set, the algorithm still predicts from one subset of runs to another, and the final benchmark scores are independent of the surrogate. Self-citations in related work (e.g., Brandfonbrener et al. 2024, Evans et al. 2024, Schwarz et al. 2018) are not load-bearing for the central claim. No circular step found.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

No new physical entities, mediators, or forces are introduced. The ledger instead lists the GP kernel hyperparameters, acquisition cost weights, and the candidate-set assumption that the benchmark results rest on.

free parameters (4)
  • GP covariance hyperparameters (output scale, mixture lengthscale, noise) = fitted via marginal likelihood
    The RBF kernel on the simplex and observation noise are learned from the collected runs; they are standard GP fit parameters but are not given a priori.
  • Downsampling kernel parameters alpha and beta = inferred by GP
    Equation 3 assumes concave monotone performance growth with parameter count; alpha and beta are learned from the data, so the cross-scale transfer behavior is partly a fitted model rather than a derived law.
  • MES acquisition cost-normalization factor = not specified in text
    Equation 9 divides information gain by cost; the paper does not give a concrete value or tuning procedure for this cost-weighting, which affects the exploration-exploitation-cost balance.
  • Dirichlet prior weights for sampling the 256 mixtures = centered on Tulu 3 optimal mixture weights
    Section 5.2.1 samples mixtures from a Dirichlet parameterized by the human-optimized Tulu 3 weights; this choice shapes which candidate mixtures exist and therefore what 'optimal' means in the benchmark.
assumptions (6)
  • standard math Gaussian process regression posterior formulas are correct and applicable
    Equations 4 and 5 use standard GP conditioning, treated as background.
  • standard math Expected Improvement and Max-value Entropy Search are valid acquisition functions
    Taken from the cited BO literature and used as black-box tools.
  • domain assumption Validation performance is smooth in mixture weights
    The RBF kernel in Equation 2 encodes the belief that similar mixtures give similar validation scores.
  • domain assumption Larger models perform better, with performance a concave monotone function of parameter count
    The Downsampling kernel in Equation 3 imposes this shape; it is a modeling assumption, not a derived result.
  • domain assumption Mixture preferences transfer across model scales
    Zero-shot transfer and MFBO both depend on small-model observations being informative for the target model.
  • domain assumption The pre-computed candidate set contains a near-optimal mixture
    Appendix A.1 restricts acquisition to existing dataset points, so the benchmark optimality claim is conditional on the initial mixture sampling procedure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimization." pith.science (2026). https://pith.science/paper/XQBSJW46

@misc{pith2026250811551,
  author       = {Pith},
  title        = {Pith review of: ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XQBSJW46}},
  note         = {Machine review of arXiv:2508.11551}
}
read the original abstract

Determining the optimal data mixture for large language model training remains a challenging problem with an outsized impact on performance. In practice, language model developers continue to rely on heuristic exploration since no learning-based approach has emerged as a reliable solution. In this work, we propose to view the selection of training data mixtures as a black-box hyperparameter optimization problem, for which Bayesian Optimization is a well-established class of appropriate algorithms. Firstly, we cast data mixture learning as a sequential decision-making problem, in which we aim to find a suitable trade-off between the computational cost of training exploratory (proxy-) models and final mixture performance. Secondly, we systematically explore the properties of transferring mixtures learned at a small scale to larger-scale experiments, providing insights and highlighting opportunities for research at a modest scale. By proposing Multi-fidelity Bayesian Optimization as a suitable method in this common scenario, we introduce a natural framework to balance experiment cost with model fit, avoiding the risks of overfitting to smaller scales while minimizing the number of experiments at high cost. We present results for pre-training and instruction finetuning across models ranging from 1 million to 7 billion parameters, varying from simple architectures to state-of-the-art models and benchmarks spanning dozens of datasets. We demonstrate consistently strong results relative to a wide range of baselines, resulting inspeed-ups of over 500% in determining the best data mixture on our largest experiments. In addition, we broaden access to research by sharing ADMIRE IFT Runs, a dataset of 460 full training & evaluation runs worth over 13,000 GPU hours, greatly reducing the cost of conducting research in this area.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Data Mixing for Large Language Models Pretraining: A Survey and Outlook

    cs.CL 2026-03 accept novelty 4.0 of 10

    A survey that taxonomizes data mixing strategies for LLM pretraining into static rule-based, learning-based, and dynamic adaptive families while highlighting transferability challenges and evaluation gaps.

Reference graph

Works this paper leans on

28 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    Icml 2024 workshop on data-centric machine learning research,

    work page 2024

  2. [4]

    org/abs/2402.16827,

    URL https://arxiv. org/abs/2402.16827,

    arXiv

  3. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

    1901

  4. [7]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186,

    2019

  5. [8]

    Combining hyperband and bayesian optimization

    Stefan Falkner, Aaron Klein, and Frank Hutter. Combining hyperband and bayesian optimization. InNIPS 2017 Bayesian Optimization Workshop (Dec 2017),

    work page 2017

  6. [11]

    A tutorial on bayesian optimization.arXiv preprint arXiv:1807.02811,

    Peter I Frazier. A tutorial on bayesian optimization.arXiv preprint arXiv:1807.02811,

    arXiv

  7. [13]

    Bayesian optimization with inequality constraints

    Jacob R Gardner, Matt J Kusner, Zhixiang Eddie Xu, Kilian Q Weinberger, and John P Cunningham. Bayesian optimization with inequality constraints. InICML, volume 2014, pp. 937–945,

    work page 2014

  8. [15]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    arXiv

Show all 28 references
  1. [17]

    Adam: A method for stochastic optimization

    18 Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    arXiv

  2. [18]

    T�" ulu 3: Pushing frontiers in open language model post-training

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T�" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124,

    arXiv

  3. [19]

    Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar

    Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Efficient hyperparameter optimization and infinitely many armed bandits.CoRR, abs/1603.06560,

    arXiv

  4. [21]

    When less is more: Investigating data pruning for pretraining llms at scale.arXiv preprint arXiv:2309.04564,

    Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining llms at scale.arXiv preprint arXiv:2309.04564,

    arXiv

  5. [24]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805,

    arXiv

  6. [26]

    Less: Selecting influential data for targeted instruction tuning.arXiv preprint arXiv:2402.04333,

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning.arXiv preprint arXiv:2402.04333,

    arXiv

  7. [27]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    arXiv

  8. [28]

    Data mixture optimization: A multi-fidelity multi-scale bayesian framework.arXiv preprint arXiv:2503.21023,

    Thomson Yen, Andrew Wei Tung Siah, Haozhe Chen, Tianyi Peng, Daniel Guetta, and Hongseok Namkoong. Data mixture optimization: A multi-fidelity multi-scale bayesian framework.arXiv preprint arXiv:2503.21023,

    arXiv

  9. [1998]

    Simopt: A library of simulation optimization problems

    Raghu Pasupathy and Shane G Henderson. Simopt: A library of simulation optimization problems. In Proceedings of the 2011 Winter Simulation Conference (WSC), pp. 4075–4085. IEEE,

    2011

  10. [1999]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027,

    arXiv

  11. [2007]

    Alexander IJ Forrester, András Sóbester, and Andy J Keane

    doi: 10.1098/rspa.2007.1900. Alexander IJ Forrester, András Sóbester, and Andy J Keane. Multi-fidelity optimization via surrogate modelling. Proceedings of the royal society a: mathematical, physical and engineering sciences, 463(2088): 3251–3269,

    2007

  12. [2013]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    arXiv

  13. [2014]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  14. [2017]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    2001 arXiv

  15. [2018]

    Regmix: Data mixture as regression for language model pre-training

    Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492,

    arXiv

  16. [2020]

    Alexander I. J. Forrester, András Sóbester, and Andy J. Keane. Multi-fidelity optimization via surrogate modelling. Proceedings of the Royal Society of London Series A, 463(2088):3251–3269, December

  17. [2021]

    Practical bayesian optimization of objectives with conditioning variables.arXiv preprint arXiv:2002.09996,

    Michael Pearce, Janis Klaise, and Matthew Groves. Practical bayesian optimization of objectives with conditioning variables.arXiv preprint arXiv:2002.09996,

    2002 arXiv

  18. [2022]

    Perplexed by perplexity: Perplexity-based data pruning with small reference models.arXiv preprint arXiv:2405.20541,

    Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L Leavitt, and Mansheej Paul. Perplexed by perplexity: Perplexity-based data pruning with small reference models.arXiv preprint arXiv:2405.20541,

    arXiv

  19. [2023]

    Phi-4 technical report.arXiv preprint arXiv:2412.08905,

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report.arXiv preprint arXiv:2412.08905,

    arXiv

  20. [2024]

    Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos

    URL��������������������������� . Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication.arXiv preprint arXiv:2303.09540,

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.