Pith. sign in

REVIEW 7 minor 53 references

Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives

T0 review · 0 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single taxonomy now organizes every LLM data-mixture method.

desk verdict A taxonomy survey that earns its keep: the fine-grained categorization and formal problem statements are genuinely useful, and the paper's honesty about the lack of aligned experiments keeps the evaluative claims appropriately modest. read the letter →

arxiv 2505.21598 v1 pith:D7TYCXAR submitted 2025-05-27 cs.CL

Yajiao Liu , Congliang Chen , Junchi Yang , Ruoyu Sun This is my paper
classification cs.CL
keywords datamixturedomainreweightinglargelanguagemodelsmin-maxoptimizationbi-levelmixinglawsonlinelearningmulti-domain
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a review that claims the field of data-mixture methods—how to set sampling weights across data domains when training large language models—can be organized into one fine-grained taxonomy. Offline methods, which fix weights before training the target model, split into heuristic-based, algorithm-based, and function fitting-based approaches; online methods, which adapt weights during training, split into online min-max optimization, online mixing laws, and other approaches. The paper argues that most online methods are online analogues of offline ones, and it frames algorithm-based offline methods through two optimization templates: min-max and bi-level optimization, with function fitting as a way of reducing bi-level problems to single-level ones. The value of the claim is a shared map: researchers and practitioners can compare methods by their objective, see which optimization machinery applies, and understand when a method family is likely to fail.

What carries the argument

The load-bearing object is the domain-weight vector $w$ in the probability simplex, together with two optimization templates that give the taxonomy its structure: the min-max problem $\min_\phi \max_{w\in\Delta_k} \sum_i w_i f_i(\phi)$ and the bi-level problem $w^*\in\arg\min_w \ell_{\mathrm{val}}(\phi^*(w))$ subject to $\phi^*(w)\in\arg\min_\phi \sum_i w_i \ell_{\mathrm{train},i}(\phi)$. For function fitting-based methods, the carrying object is a mixing law $f_j$ that predicts validation loss for domain $j$ from domain weights, turning the bi-level problem into a single-level fit-then-minimize problem. These templates determine what each method optimizes, which algorithms apply, and which online analogue it has.

What would settle it

Find an online data-mixture method whose weight-update rule is neither a min-max objective, a fitted mixing law, nor a bandit or credit-assignment scheme; if such a method proves effective, it falls outside the proposed taxonomy and the claim that most online methods are online variants of offline ones is incomplete. For the comparative claims, a controlled benchmark re-running representative methods from each family under identical model, tokenizer, compute, and data configurations would show whether the stated advantages and disadvantages hold; if gains over uniform sampling disappear under aligned settings, the survey's guidance loses force.

Watch

Extended reading notes

Core claim

The central claim is that the apparent zoo of data-mixture algorithms is generated by a small number of underlying objectives. The paper proposes a categorization in which offline methods are divided into heuristic-based methods, algorithm-based methods built on min-max or bi-level optimization, and function fitting-based methods that learn a 'mixing law' mapping domain weights to validation loss. Online methods are then divided into online min-max optimization, online mixing laws, and residual approaches, with most shown to be online variants of the offline families—obtained by replacing the proxy model with the target model itself and updating weights during training. The paper also identifies the key difference between min-max and bi-level formulations: min-max defines the best weights by worst-case training loss across domains, while bi-level defines them by validation performance of the model trained under the weights; function fitting converts the latter into a single-level regression-and-optimize problem. If the taxonomy is right, the field's methods are not disconnected heuristics but instantiations of a few shared optimization structures.

Load-bearing premise

The survey's comparative guidance assumes the experiments in the papers it reviews are reliable and directly comparable, but the authors state these experiments differ in model type, model scale, and training-data scale.

Editorial extensions

If this is right

  • A practitioner can classify any new data-mixture method by asking whether it fixes weights first (offline) or adapts them during training (online), and which objective it optimizes; this tells them which family's known failure modes to expect.
  • Optimization results from multi-task learning—min-max algorithms such as gradient descent ascent and bi-level solvers—carry over to data-mixture problems, since the paper argues the fields share the same formal structure.
  • Function fitting-based methods are the only offline family designed to extrapolate weights across model scale and data scale, so the taxonomy identifies them as the response to the model-scale and data-scale transfer problems.
  • Online mixing-law methods are the only online family that can naturally use out-of-domain validation data, making them the relevant choice when generalization beyond training domains is the goal.
  • The distinction between min-max and bi-level objectives clarifies why some methods target worst-case domain performance while others target validation accuracy, a difference that is hidden by the offline/online split.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The taxonomy implies a design recipe not stated in the paper: any offline method can be converted into an online variant by replacing the proxy model with the target model, so future work can generate new online methods systematically rather than ad hoc.
  • The paper's limitation—that cited experiments vary in model type, scale, and data scale—suggests that a standardized benchmark with aligned settings would be the next bottleneck; until then, comparative claims between families should be read as provisional.
  • The taxonomy predicts hybrid opportunities, such as using a function-fitted mixing law to initialize an online min-max method, which would combine the extrapolation power of offline fitting with the adaptation of online weight updates.
  • Because the paper notes that mixture gains over uniform sampling are often small or negative, a fair reading is that the practical payoff of the field depends on replacing loss surrogates with downstream-task-aware objectives, not on weight-optimization algorithms alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 7 minor

Summary. The manuscript is a survey of data mixture (domain reweighting) methods for LLM training. It proposes a fine-grained taxonomy that extends the offline/online split of Albalak et al. (2024): offline methods are subdivided into heuristic-based, algorithm-based (min-max and bi-level optimization), and function fitting-based (mixing-law) approaches; online methods are subdivided into online min-max, online mixing law, and other approaches. For each branch the paper gives problem formulations (Eqs. 1-3), representative methods (Tables 1-4), and qualitative advantages and disadvantages, and it closes with open challenges. The paper reports no new experiments and explicitly states in the Limitation section that the original methods are not compared under aligned experimental settings.

Significance. The taxonomy is the main contribution and it is a useful one. The connection between offline optimization frameworks (min-max and bi-level) and online update rules makes the survey more than a list of methods, and the tables give practitioners a quick way to position new work. I checked the problem formulations and table entries against the cited methods; aside from the typos noted below, the summaries are faithful. The explicit Limitation about unaligned experimental comparisons does undercut the strength of the comparative claims in Sections 3.4 and 4.4, but it does not undermine the taxonomy, which is the central claim. With the local corrections below, the paper is a solid contribution.

minor comments (7)
  1. [Table 4] In both rows of Table 4, the denominator of the update rule for w_t uses w^{t-1}_i while the numerator uses w^{t-t_update}_i; since these methods update every t_update > 1 steps, the denominator should also be w^{t-t_update}_i.
  2. [Section 4.2] In the displayed definition of the online mixing law, the summation is over i but the weight is written as w^t_j; it should be w^t_i for the expression to be well-defined.
  3. [Section 3.3] After Eq. (3), the optimal weights are written as w* = min_{w in Delta_k} g(...); this should be argmin, since the objective returns a scalar rather than a weight vector.
  4. [Section 3.2.2] The notation Dout_val is used for out-of-domain validation data although Section 2 defines Dood_val; please unify the two notations.
  5. [Table 1] DoReMionline is discussed in Section 4.1 and appears in Table 3, but it is missing from Table 1's overview of online methods; either add it explicitly or add a note that it is an online variant of the offline DoReMi entry.
  6. [Table 1] The method name 'Aoili' in Table 1 should be 'AIOLI' (or 'Aioli') to match the text and the cited paper.
  7. [Limitation] The Limitation section correctly discloses that the surveyed methods are not compared under aligned experimental settings; consider moving or echoing this caveat in Sections 3.4 and 4.4 where relative advantages are stated, so readers do not interpret those statements as quantitative comparisons.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the survey's fine-grained taxonomy organizes external results, and the only self-citation is peripheral.

full rationale

This paper is a taxonomy survey, not a derivation. Its central claim is the fine-grained categorization in Sections 1, 3, and 4, which is defined by observable features of the cited methods: offline methods are grouped by how w* is obtained, and online methods by how w_t is updated. Tables 1-4 apply that scheme to prior published algorithms, and Equations 1-7 summarize problem formulations from those algorithms rather than predicting new quantities from parameters fitted in this paper. The paper conducts no empirical studies, fits no parameters, and makes no predictive claim of its own, so the fitted-input-called-prediction pattern does not arise. The only self-citation is Zhang et al. (2022), mentioned in Section 3.2.1 as an optional GDA variant ('There are other variants of GDA, such as Optimistic GDA (Daskalakis et al., 2018) and Smoothed GDA (Zhang et al., 2022)'); this is peripheral and not load-bearing. The Limitation section explicitly concedes that the underlying experiments 'are not aligned in terms of experimental settings, such as model type, model scale, and training data scale,' which weakens any comparative guidance but does not make the taxonomy circular. No equation in the paper is defined in terms of its own output, and no claimed result reduces by construction to a fitted input or to a self-citation chain. Score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The survey introduces no fitted parameters, no new mathematical objects, and no invented entities. The taxonomy rests on background definitions (probability simplex, loss functions) and on two domain assumptions from the surveyed literature: that fixed static weights can be optimal for a corpus, and that validation loss is a useful surrogate for downstream performance. These assumptions are discussed openly in Sections 5.1 and 5.3.

assumptions (4)
  • standard math Domain weights are constrained to the probability simplex: nonnegative and summing to one.
    Used throughout Section 2 and in Equations 1-3 to define the feasible set for w.
  • domain assumption Offline methods assume the existence of a fixed optimal static weight vector w* for a given corpus.
    The survey states this assumption in Section 3 (introduction) and problematizes it in Section 5.1, where it notes a static optimum may not exist and that curricula can be better.
  • domain assumption Validation loss is a meaningful surrogate for downstream task performance when optimizing domain weights.
    The survey uses validation loss in Equations 2-3 and in the online method descriptions; Section 5.3 explicitly questions this surrogate, citing work that the optimal mixture by loss is only slightly better than uniform, so this assumption is load-bearing for many surveyed methods.
  • domain assumption Data mixture can be viewed as a subfield of Multi-Domain Learning, so optimization frameworks from Multi-Task Learning (min-max and bi-level) apply.
    Section 3.2 states this connection and uses it to structure the entire taxonomy of algorithm-based and online methods.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives." pith.science (2026). https://pith.science/paper/D7TYCXAR

@misc{pith2026250521598,
  author       = {Pith},
  title        = {Pith review of: Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D7TYCXAR}},
  note         = {Machine review of arXiv:2505.21598}
}
read the original abstract

Training large language models with data collected from various domains can improve their performance on downstream tasks. However, given a fixed training budget, the sampling proportions of these different domains significantly impact the model's performance. How can we determine the domain weights across different data domains to train the best-performing model within constrained computational resources? In this paper, we provide a comprehensive overview of existing data mixture methods. First, we propose a fine-grained categorization of existing methods, extending beyond the previous offline and online classification. Offline methods are further grouped into heuristic-based, algorithm-based, and function fitting-based methods. For online methods, we categorize them into three groups: online min-max optimization, online mixing law, and other approaches by drawing connections with the optimization frameworks underlying offline methods. Second, we summarize the problem formulations, representative algorithms for each subtype of offline and online methods, and clarify the relationships and distinctions among them. Finally, we discuss the advantages and disadvantages of each method and highlight key challenges in the field of data mixture.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 11 canonical work pages

  1. [1]

    Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. 2024. https://arxiv.org/abs/2402.16827 A survey on data selection for language models . Preprint, arXiv:2402.16827

    arXiv 2024

  2. [2]

    Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. 2023. https://arxiv.org/abs/2312.02406 Efficient online data mixing for language model pre-training . Preprint, arXiv:2312.02406

    arXiv 2023

  3. [3]

    Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. 2002. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(1):48--77

    work page 2002

  4. [4]

    Lior Belenki, Alekh Agarwal, Tianze Shi, and Kristina Toutanova. 2025. https://arxiv.org/abs/2502.15950 Optimizing pre-training data mixtures with mixtures of data expert models . Preprint, arXiv:2502.15950

    work page Pith review arXiv 2025

  5. [5]

    Yoshua Bengio, J \'e r \^o me Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41--48

    2009

  6. [6]

    Chen, Michael Y

    Mayee F. Chen, Michael Y. Hu, Nicholas Lourie, Kyunghyun Cho, and Christopher Ré. 2024. https://arxiv.org/abs/2411.05735 Aioli: A unified optimization framework for language model data mixing . Preprint, arXiv:2411.05735

    arXiv 2024

  7. [7]

    Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré

    Mayee F. Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. 2023. https://arxiv.org/abs/2307.14430 Skill-it! a data-driven skills framework for understanding and training language models . Preprint, arXiv:2307.14430

    arXiv 2023

  8. [8]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping H...

    arXiv 2022

Show all 53 references
  1. [9]

    Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. 2018. https://arxiv.org/abs/1711.00141 Training gans with optimism . Preprint, arXiv:1711.00141

    2018 arXiv

  2. [10]

    Constantinos Daskalakis and Ioannis Panageas. 2018. https://arxiv.org/abs/1807.03907 The limit points of (optimistic) gradient descent in min-max optimization . Preprint, arXiv:1807.03907

    2018

  3. [11]

    Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hell...

    2022 arXiv

  4. [12]

    Jean-Antoine Désidéri. 2012. https://doi.org/10.1016/j.crma.2012.03.014 Multiple-gradient descent algorithm (mgda) for multiobjective optimization . Comptes Rendus Mathematique, 350(5):313--318

    2012 doi

  5. [13]

    Simin Fan, Matteo Pagliardini, and Martin Jaggi. 2023. https://openreview.net/forum?id=qiKqsqwYXm DOGE : Domain reweighting with generalization estimation . In Second Agent Learning in Open-Endedness Workshop

    2023

  6. [14]

    Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. 2017. https://arxiv.org/abs/1703.01785 Forward and reverse gradient-based hyperparameter optimization . Preprint, arXiv:1703.01785

    2017 arXiv

  7. [15]

    Dimakis, Gabriel Ilharco, Pang Wei Koh, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt

    Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Luca Soldaini, Alexandros G. Dimakis, Gabriel Ilharco, Pang W...

    2024 arXiv

  8. [16]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. https://arxiv.org/abs/2101.00027 The pile: An 800gb dataset of diverse text for language modeling . P...

    2020 arXiv

  9. [17]

    Ce Ge, Zhijian Ma, Daoyuan Chen, Yaliang Li, and Bolin Ding. 2024. https://arxiv.org/abs/2405.14908 Bimix: Bivariate data mixing law for language model pretraining . Preprint, arXiv:2405.14908

    2024 arXiv

  10. [18]

    Smith, and Luke Zettlemoyer

    Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. 2023. https://arxiv.org/abs/2303.14177 Scaling expert language models with unsupervised domain discovery . Preprint, arXiv:2303.14177

    2023 arXiv

  11. [19]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  12. [20]

    Zico Kolter

    Yiding Jiang, Allan Zhou, Zhili Feng, Sadhika Malladi, and J. Zico Kolter. 2024. https://arxiv.org/abs/2410.11820 Adaptive data optimization: Dynamic sample selection with scaling laws . Preprint, arXiv:2410.11820

    2024 arXiv

  13. [21]

    Feiyang Kang, Yifan Sun, Bingbing Wen, Si Chen, Dawn Song, Rafid Mahmood, and Ruoxi Jia. 2025. https://arxiv.org/abs/2407.20177 Autoscale: Scale-aware data mixing for pre-training llms . Preprint, arXiv:2407.20177

    2025

  14. [22]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. https://arxiv.org/abs/2001.08361 Scaling laws for neural language models . Preprint, arXiv:2001.08361

    2020 arXiv

  15. [23]

    Jeongyeol Kwon, Dohyun Kwon, Stephen Wright, and Robert Nowak. 2023. https://arxiv.org/abs/2301.10945 A fully first-order method for stochastic bilevel optimization . Preprint, arXiv:2301.10945

    2023 arXiv

  16. [24]

    Tianyi Lin, Chi Jin, and Michael Jordan. 2020. https://proceedings.mlr.press/v119/lin20a.html On gradient descent ascent for nonconvex-concave minimax problems . In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learn...

    2020

  17. [25]

    Bingchang Liu, Chaoyu Chen, Cong Liao, Zi Gong, Huan Wang, Zhichao Lei, Ming Liang, Dajun Chen, Min Shen, Hailian Zhou, Hang Yu, and Jianguo Li. 2023 a . https://arxiv.org/abs/2311.02303 Mftcoder: Boosting code llms with multitask fine-tuning . Preprint, arXiv:2311.02303

    2023 arXiv

  18. [26]

    Bo Liu, Yihao Feng, Peter Stone, and Qiang Liu. 2023 b . https://arxiv.org/abs/2306.03792 Famo: Fast adaptive multitask optimization . Preprint, arXiv:2306.03792

    2023 arXiv

  19. [27]

    Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. 2021 a . Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34

    2021

  20. [28]

    Dong C Liu and Jorge Nocedal. 1989. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1):503--528

    1989

  21. [29]

    Liyang Liu, Yi Li, Zhanghui Kuang, Jing-Hao Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. 2021 b . https://openreview.net/forum?id=IMPnRXEWpvr Towards impartial multi-task learning . In International Conference on Learning Representations

    2021

  22. [30]

    Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. 2024. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492

    2024 arXiv

  23. [31]

    Jonathan Lorraine, Paul Vicol, and David Duvenaud. 2019. https://arxiv.org/abs/1911.02590 Optimizing millions of hyperparameters by implicit differentiation . Preprint, arXiv:1911.02590

    2019 arXiv

  24. [32]

    Zheheng Luo, Xin Zhang, Xiao Liu, Haoling Li, Yeyun Gong, Chen Qi, and Peng Cheng. 2024. https://arxiv.org/abs/2411.14318 Velocitune: A velocity-based dynamic domain reweighting method for continual pre-training . Preprint, arXiv:2411.14318

    2024 arXiv

  25. [33]

    Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, and Mohammad Rastegari. 2024. https://arxiv.org/abs/2404.14619 Openelm: An efficient language model family with open tr...

    2024 arXiv

  26. [34]

    Hashimoto, and Percy Liang

    Yonatan Oren, Shiori Sagawa, Tatsunori B. Hashimoto, and Percy Liang. 2019. https://doi.org/10.18653/v1/D19-1432 Distributionally robust language modeling . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Join...

    2019 doi

  27. [35]

    Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. 2024 a . https://arxiv.org/abs/2403.17919 Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning . Preprint, arXiv:2403.17919

    2024 arXiv

  28. [36]

    Rui Pan, Jipeng Zhang, Xingyuan Pan, Renjie Pi, Xiaoyu Wang, and Tong Zhang. 2024 b . Scalebio: Scalable bilevel optimization for llm data reweighting. arXiv preprint arXiv:2406.19976

    2024 arXiv

  29. [37]

    Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, Xu Tan, Jie Fu, Wenbo Su, Jiamang Wang, Lin Qu, and Bo Zheng. 2024. https://arxiv.org/abs/2406.01375 D-cpt law: Domain-specific continual pre-training...

    2024 arXiv

  30. [38]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners

    2019

  31. [39]

    Meisam Razaviyayn, Tianjian Huang, Songtao Lu, Maher Nouiehed, Maziar Sanjabi, and Mingyi Hong. 2020. https://doi.org/10.1109/msp.2020.3003851 Nonconvex min-max optimization: Applications, challenges, and recent theoretical advances . IEEE Signal Processing Magazine, 37(5):55–66

    2020

  32. [40]

    Hashimoto, and Percy Liang

    Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. 2020. https://arxiv.org/abs/1911.08731 Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization . Preprint, arXiv:1911.08731

    2020 arXiv

  33. [41]

    Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, and Eric Xing. 2024. https://arxiv.org/abs/2309.10818 Slimpajama-dc: Understanding data combinations for llm training . Preprint,...

    2024 arXiv

  34. [42]

    Xingyou Song, Qiuyi Zhang, Chansoo Lee, Emily Fertig, Tzu-Kuo Huang, Lior Belenki, Greg Kochanski, Setareh Ariafar, Srinivas Vasudevan, Sagi Perel, and Daniel Golovin. 2024. https://arxiv.org/abs/2408.11527 The vizier gaussian process bandit algorithm . Preprint, arXiv:2408.11527

    2024 arXiv

  35. [43]

    Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J

    Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, E...

    2020

  36. [44]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. https://arxiv.org/abs/2109.01652 Finetuned language models are zero-shot learners . Preprint, arXiv:2109.01652

    2022 arXiv

  37. [45]

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2024. https://arxiv.org/abs/2310.06694 Sheared llama: Accelerating language model pre-training via structured pruning . Preprint, arXiv:2310.06694

    2024 arXiv

  38. [46]

    Le, Tengyu Ma, and Adams Wei Yu

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. 2023. Doremi: Optimizing data mixtures speeds up language model pretraining. arXiv preprint arXiv:2305.10429

    2023 arXiv

  39. [47]

    Hospedales

    Yongxin Yang and Timothy M. Hospedales. 2015. https://arxiv.org/abs/1412.7489 A unified perspective on multi-domain and multi-task learning . Preprint, arXiv:1412.7489

    2015 arXiv

  40. [48]

    Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. 2024. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952

    2024 arXiv

  41. [49]

    Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06782

    2020 arXiv

  42. [50]

    Jiawei Zhang, Peijun Xiao, Ruoyu Sun, and Zhi-Quan Luo. 2022. https://arxiv.org/abs/2010.15768 A single-loop smoothed gradient descent-ascent algorithm for nonconvex-concave min-max problems . Preprint, arXiv:2010.15768

    2022 arXiv

  43. [51]

    Yihua Zhang, Prashant Khanduri, Ioannis Tsaknakis, Yuguang Yao, Mingyi Hong, and Sijia Liu. 2023. https://arxiv.org/abs/2308.00788 An introduction to bi-level optimization: Foundations and applications in signal processing and machine learning . Preprint, arXiv:2308.00788

    2023 arXiv

  44. [52]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  45. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.