REVIEW 4 major objections 4 minor 56 references
AutoMixAlign: Adaptive Data Mixing for Multi-Task Preference Optimization in LLMs
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims a generalist LLM can match task specialists by minimizing the maximum clipped excess loss across tasks, and that adaptive reweighting or resampling of this objective beats standard DPO mixing and model merging.
desk verdict Useful empirical recipe for adaptive data mixing in DPO, but the convergence proof is for a simplified objective and algorithm, so the theory section overclaims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the clipped excess loss, defined per sample as $E(\theta,\theta_i,z)=\max\{L(\theta,z)-L(\theta_i,z),0\}$, where $\theta_i$ is the specialist trained on task $i$. This quantity carries the argument because it turns 'match the specialist' into a per-sample threshold: once the generalist's loss on a task falls to the specialist's level, the clipped term stops contributing, so optimization effort shifts to tasks with remaining room for improvement. The second mechanism is the minimax formulation over the task simplex, which lets the same objective be realized either as adaptive weights in the loss (AMA-R) or as adaptive sampling probabilities in the stochastic gradient (AMA-S). For the theory, AMA-S is analyzed as a zero-sum game in which the $\theta$-player is $C$-low-regret and the $\alpha$-player runs EXP3, producing an $O(1/\sqrt{T})$ convergence rate.
What would settle it
Take a task pair known to trade off in a small model, for example maximizing Toxigen non-toxicity and maximizing AlpacaEval win rate in the same 7B model. If AMA is run on those two datasets and the final generalist's clipped excess loss on one task remains bounded away from zero while the algorithm keeps assigning that task increasing weight as $T$ grows, the feasibility assumption has failed and the generalist cannot actually match both specialists.
Extended reading notes
Core claim
The central claim is that multi-task preference alignment can be recast as finding a single model whose task losses do not substantially exceed the losses of separately trained specialists. The paper formalizes this as minimizing the maximum over tasks of the average clipped excess loss, $E(\theta,\theta_i,z)=\max\{L(\theta,z)-L(\theta_i,z),0\}$, and gives two algorithms that solve it: AMA-R updates task weights by exponentiated gradient ascent on this excess loss, and AMA-S samples tasks with EXP3-adjusted probabilities. Both are shown to converge at $O(1/\sqrt{T})$ in the convex setting, with AMA-S's proof coming from framing the procedure as a zero-sum game between a low-regret $\theta$-player and an adversarial $\alpha$-player. In experiments on two- and three-task combinations of helpfulness, coding, and harmlessness, both algorithms produce the highest average scores and, the paper reports, are the only methods tested that never degrade any individual benchmark relative to the specialists.
Load-bearing premise
The load-bearing assumption is that some single model can reach every specialist's loss at the same time; if no one model can be simultaneously as good at every task as the specialists are individually, then the minimax target is unattainable and the balancing behavior has no well-defined optimum.
Editorial extensions
If this is right
- AMA-R and AMA-S both inherit a standard $O(1/\sqrt{T})$ convergence rate in the convex setting, so adaptive task prioritization does not trade away a convergence guarantee.
- Because the excess loss is clipped at zero, tasks already at specialist-level loss stop receiving training pressure; the paper's ablations show this clipping is what prevents the generalist from overfitting beyond specialist performance.
- AMA improves average benchmark scores by up to 9.42 percentage points over standard DPO data mixing and outperforms uniform parameter averaging in the tested setups, while never degrading any single benchmark.
- AMA needs the same number of model updates as two runs of standard DPO, independent of the number of tasks, making it a budget-comparable alternative to running many data-mixing ablations.
- When task weights become extreme, AMA-S concentrates computation on the sampled tasks instead of wasting gradients on near-zero-weight tasks, which the paper shows lets it catch up faster on an undersampled safety dataset than AMA-R.
Reading between the lines
- Going beyond the paper, the same clipped-excess-loss minimax could be applied to supervised fine-tuning or on-policy RLHF, since the reweighting and resampling mechanisms do not depend on the specific DPO loss.
- Going beyond the paper, if tasks genuinely conflict, the minimax objective may endlessly upweight an unattainable task; a testable extension would add a small regularization that interpolates between matching specialists and optimizing the average loss.
- Going beyond the paper, the $O(1/\sqrt{T})$ theorem is for convex losses, so a natural check is to run AMA-R and AMA-S on a synthetic convex multi-task problem and confirm the empirical gap to the minimax value shrinks at roughly $1/\sqrt{T}$; the paper's experiments do not include such a check.
- Going beyond the paper, because the specialist losses are the reference scale, one could quantify how much generalist quality degrades when specialist losses are approximated by smaller models or Hessian estimates, which the paper names as cheaper alternatives but does not evaluate.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes AutoMixAlign (AMA), a two-phase method for multi-task preference optimization with DPO. In the first phase, specialist models are trained on each task dataset; in the second phase, a generalist model is trained by minimizing a minimax objective over clipped excess losses, i.e., max over tasks of the average max{L(θ,z)-L(θ_i,z),0}. Two optimization algorithms are presented: AMA-R, which adaptively reweights the task objective, and AMA-S, which adaptively resamples from task datasets. The paper claims O(1/sqrt(T)) convergence for both algorithms in the convex case, with the AMA-S result proved via online learning; empirically, AMA is evaluated on helpfulness, coding, and harmlessness benchmarks and reported to outperform standard DPO training and model merging. The paper also includes ablations on the clipped excess loss, fixed-weight variants, and weighted model merging.
Significance. If the theoretical guarantee applied to the actual algorithm, AMA would be a useful step toward principled automatic data mixing for LLM alignment, and the excess-loss formulation is well motivated: it prevents optimization from continuing on already-learned tasks. The paper has several concrete strengths: the two-phase specialist/generalist design is clearly described; the ablations in Appendix C show that clipping and adaptive weighting matter; the resampling-vs-reweighting experiment in Section 4.4 illustrates a real efficiency issue; and the weighted model merging ablations strengthen the comparison. However, the central theoretical claim as written is not established: the convergence theorem in Section 3.4 is proved for a simplified raw-loss objective and a different update rule, not for the clipped-excess-loss algorithm actually implemented. The empirical comparison also rests on single training runs without error bars. These issues undermine the 'theoretically-grounded' claim, though the empirical idea remains plausible and worth further work.
major comments (4)
- [Section 3.4, Theorem 3.2, Appendix A.2 vs. Algorithm 1] The O(1/sqrt(T)) theorem is proved for a different objective and a different update rule than Algorithm 1. The proof explicitly 'abstract[s] away from the excess losses' and analyzes min_θ max_α Σ_i α_i (1/|D_i|) Σ_{z∈D_i} L(θ,z). Algorithm 1, however, updates q by exponentiating 1/q_i times the average clipped excess loss E(θ,θ_i,z)=max{L(θ,z)-L(θ_i,z),0}, with no learning-rate η and no 1/α_i importance weighting, and updates θ on the clipped excess loss. The proof's α-player instead uses the importance-weighted estimator z_t=-L_{i_t}(θ_t)/α_{t,i_t}e_{i_t}, an EG step with η=1/(4k), and a single sampled task per round. No argument connects this theorem to Problem 4 or to Algorithm 1, so the abstract's claim that AMA is 'theoretically-grounded' is unsupported for the implemented algorithm.
- [Section 3.4, proof of Theorem 3.2] Even accepting the raw-loss abstraction, the proof does not cover Algorithm 1's smoothing choice. The proof states α_t,i = 1/(2k)+q_t,i/2, which corresponds to smoothing parameter c=1/2, and uses this to conclude 1/α_t,i ≤ 2k and η z_t,i ≥ -1. Algorithm 1 uses α_t,i=(1-c)q_t,i+c/k, and the experiments set c=0.1, so 1/α_t,i can be as large as 10k, violating the bound required for the EG lemma (Lemma A.1). Additionally, Algorithm 1's q update has no explicit η, while the proof's EG step uses η=1/(4k). The theorem therefore does not imply a convergence rate for Algorithm 1 under the experimental hyperparameters.
- [Section 3.2, footnote 3, and Section 6] The claimed O(1/sqrt(T)) guarantee for AMA-R is also not substantiated. Footnote 3 concedes that Sagawa et al. (2019) optimizes the loss L(θ,z), whereas AMA-R optimizes the clipped excess loss E(θ,θ_i,z); the conclusion repeats the convergence claim without a proof for this different objective. Since the clipped excess loss is convex when L is convex, a GroupDRO-style proof might go through, but the paper does not supply the required conditions or argument. The theoretical claims should be either proved for the clipped-excess objective or explicitly re-scoped to a simplified objective.
- [Appendix H.1, Tables 1-3] The empirical superiority claim is based on a single training run per method and setup, with no standard errors or repeated-seed results for the final benchmark scores. Several reported differences are small (e.g., Table 2: Standard Uniform 53.21 vs. AMA-R 53.50 and AMA-S 53.81; Table 3: AMA-R 54.38 vs. AMA-S 53.18), so it is not possible to assess whether AMA's improvements are statistically reliable. The checkpoint selection procedure uses evaluation confidence intervals, but this is within a single run and does not account for run-to-run variance. Please provide multiple seeds or an error analysis, or soften the claims accordingly.
minor comments (4)
- [Section 4.1, first paragraph] The text says 'Our first experiment focuses on balancing helpfulness and harmlessness,' but the experiment is Helpfulness + Coding; the next sentence and Table 1 confirm this, so the first clause should say 'helpfulness and coding.'
- [Algorithm 1, line 8] The symbol T is used both for the total number of training steps and for the multinomial sample set ('T ∼ Multinomial(α_1,...,α_k,b)'), which is confusing; use a different symbol such as S for the sampled set.
- [Theorem 3.2] The assumption max_{θ∈Θ} L(θ,z) ≤ 1 is not satisfied for the DPO loss, which is unbounded above; please state how this condition is met or normalized in the LLM setting, or weaken the theorem accordingly.
- [Section 3.1] The feasibility assertion 'LLMs are typically sufficiently expressive so that there exists a solution to feasibility problem 2' is an unproved empirical claim; if it fails, the minimax objective has a positive optimum and the 'matching all specialists' interpretation does not follow. This should be flagged as an assumption or supported.
Circularity Check
No significant circularity found: the minimax objective is defined against precomputed specialist losses but the empirical claims are tested on external benchmarks, and the theory/algorithm mismatch is an unsupported transfer rather than a circular reduction. One minor background self-citation is not load-bearing.
full rationale
The derivation chain is: train specialist models, freeze their per-sample losses, then train a generalist by minimizing the maximum clipped excess loss E(θ,θ_i,z)=max{L(θ,z)-L(θ_i,z),0}. The target values L(θ_i,z) are computed once and are external to the generalist optimization; no model parameter is fitted to the evaluation benchmarks, so the empirical comparisons in Tables 1-3 are not circular. The central objective is explicitly an aim ('this optimization aims to train a generalist model that matches the performance of all specialist models'), not a prediction derived from the objective by construction. Two admitted gaps affect the theory but are not circular reductions. Section 3.2 footnote 3 concedes: 'we note that AMA-R optimizes a different objective; Sagawa et al. (2019) optimizes the loss L(θ,z) whereas AMA-R optimizes the clipped excess loss E(θ,θ_i,z)', so the claimed O(1/sqrt(T)) convergence for AMA-R is imported from a theorem for a different objective. Section 3.4 states 'For simplicity, we abstract away from the excesses losses' and proves Theorem 3.2 for the raw-loss minimax, then calls it 'a standard O(1/sqrt(T)) convergence rate for the AMA-S algorithm'; Algorithm 1's EXP3 update also differs from the proof's importance-weighted EG update (no eta, no 1/alpha weighting). These are unsupported-transfer flaws, not cases where an output equals an input by construction. The feasibility assertion in Section 3.1 ('LLMs are typically sufficiently expressive so that there exists a solution to feasibility problem 2') is an assumption, not a derived consequence, and does not create circularity. The only self-citation is the excess-loss background (He et al., 2024, with overlapping authors), which is also attributed to the external DoReMi (Xie et al., 2024) and is not load-bearing for the empirical results. Section 7's stated specialist-training overhead is a cost caveat, not a circular step. Overall, no quoted equation or fitted parameter is recycled as a prediction, so the circularity score is 2 only for the minor background self-citation.
Assumptions & free parameters
free parameters (3)
- task weight learning rate eta =
1
- smoothing parameter c =
0.1
- number of training epochs =
3
assumptions (3)
- domain assumption LLMs are sufficiently expressive that a single model can achieve losses no worse than every specialist on its own task (feasibility of Problem 2)
- standard math Convexity and boundedness: L is convex, max_theta L <= 1, and a C-low-regret theta-player exists
- domain assumption Specialist losses L(theta_i, z) are accurate proxies for strong task performance
invented entities (1)
-
N/A
Cite this review
Pith. "Pith review of AutoMixAlign: Adaptive Data Mixing for Multi-Task Preference Optimization in LLMs." pith.science (2026). https://pith.science/paper/7Y5BE2D4
@misc{pith2026250600569,
author = {Pith},
title = {Pith review of: AutoMixAlign: Adaptive Data Mixing for Multi-Task Preference Optimization in LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/7Y5BE2D4}},
note = {Machine review of arXiv:2506.00569}
}
abstract
When aligning large language models (LLMs), their performance on various tasks (such as being helpful, harmless, and honest) depends heavily on the composition of their training data. However, selecting a data mixture that achieves strong performance across all tasks is challenging. Existing approaches rely on large ablation studies, heuristics, or human intuition, but these can be prohibitively expensive and suboptimal. We study this problem in the setting of preference optimization via DPO and introduce AutoMixAlign (AMA), a theoretically-grounded algorithm that adaptively mixes datasets during training to balance performance across tasks. AMA first trains \textit{specialist models} for each task to determine losses that correspond to strong task performance. Then, it trains a generalist model using a novel minimax optimization that prioritizes tasks for which generalist model losses deviate most from specialist model losses. To optimize this problem, we propose two algorithms: (1) AMA-R, which adaptively reweights the objective to prioritize tasks, and (2) AMA-S, which adaptively adjusts how much data is sampled from each task to prioritize tasks. Both algorithms achieve a convergence rate of $O(1/\sqrt{T})$ in the convex case. AMA-R's convergence result follows from Sagawa et al. (2019), and we provide a convergence proof for AMA-S using online learning techniques such as EXP3. We evaluate AMA on several multitask alignment setups and find that AMA outperforms the standard alignment approach -- which simply optimizes the total loss across all tasks -- and also outperforms model merging methods.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. 2023. Efficient online data mixing for language model pre-training. arXiv preprint arXiv:2312.02406
arXiv 2023
-
[2]
P Auer. 2002. Finite-time analysis of the multiarmed bandit problem
2002
-
[3]
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732
arXiv 2021
-
[4]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901
2020
-
[5]
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374
arXiv 2021
-
[6]
Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang, and Heng Ji. 2024. Scaling laws for predicting downstream performance in llms. arXiv preprint arXiv:2410.08527
arXiv 2024
-
[7]
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794--803. PMLR
work page 2018
-
[8]
Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. 2020. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 33:2039--2050
work page 2020
Show all 56 references
-
[9]
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1--113
2023
-
[10]
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback
2023
-
[11]
Karin De Langis, Ryan Koo, and Dongyeop Kang. 2024. Dynamic multi-reward weighting for multi-style controllable generation. arXiv preprint arXiv:2402.14146
2024 arXiv
-
[12]
Jacopo Desideri. 2009. https://doi.org/10.1007/s10957-008-9504-1 Multiple-gradient descent algorithm for multiobjective optimization . Journal of Optimization Theory and Applications, 142(3):639--656
2009 doi
-
[13]
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547--5569. PMLR
2022
-
[14]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783
2024 arXiv
-
[15]
Yann Dubois, Bal \'a zs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475
2024 arXiv
-
[16]
Xiequan Fan, Ion Grama, and Quansheng Liu. 2012. Hoeffding’s inequality for supermartingales. Stochastic Processes and their Applications, 122(10):3545--3559
2012
-
[17]
Heshan Fernando, Han Shen, Miao Liu, Subhajit Chaudhury, Keerthiram Murugesan, and Tianyi Chen. 2022. Mitigating gradient bias in multi-objective learning: A provably convergent stochastic approach. arXiv preprint arXiv:2210.12624
2022 arXiv
-
[18]
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09509
2022 arXiv
-
[19]
Elad Hazan, Tomer Koren, and Nati Srebro. 2011. Beating sgd: Learning svms in sublinear time. Advances in Neural Information Processing Systems, 24
2011
-
[20]
Yifei He, Shiji Zhou, Guojun Zhang, Hyokun Yun, Yi Xu, Belinda Zeng, Trishul Chilimbi, and Han Zhao. 2024. Robust multi-task learning with excess risks. arXiv preprint arXiv:2402.02009
2024 arXiv
-
[21]
Smith, Yejin Choi, and Hanna Hajishirzi
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hanna Hajishirzi. 2024. https://api.semanticscholar.org/CorpusID:270440774 Unpacking dpo and ppo: Disentangling best practices for learning from preference fe...
2024 arXiv
-
[22]
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. 2023. Camels in a changing climate: Enhancing lm adaptation with tulu 2. arXiv preprint arXiv:2311.10702
2023 arXiv
-
[23]
Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. https://openreview.net/forum?id=g0QovXbFw3 Beavertails: Towards improved safety alignment of LLM via a human-preference dataset . In Thirty-sevent...
2023
-
[24]
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2022. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09849
2022 arXiv
-
[25]
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. 2024. T " ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124
2024 arXiv
-
[26]
Hashimoto
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval
2023
-
[27]
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. 2021 a . Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:18878--18890
2021
-
[28]
Liyang Liu, Yi Li, Zhanghui Kuang, J Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. 2021 b . Towards impartial multi-task learning. iclr
2021
-
[29]
Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. 2024. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492
2024 arXiv
-
[30]
Michael S Matena and Colin A Raffel. 2022. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703--17716
2022
-
[31]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273--1282. PMLR
2017
-
[32]
Paul Michel, Sebastian Ruder, and Dani Yogatama. 2021. Balancing average and worst-case accuracy in multitask learning. arXiv preprint arXiv:2110.05838
2021 arXiv
-
[33]
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2023. Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07124
2023 arXiv
-
[34]
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. 2022. Multi-task learning as a bargaining game. arXiv preprint arXiv:2202.01017
2022 arXiv
-
[35]
Shay Oren et al. 2019. Distributionally robust language modeling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)
2019
-
[36]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...
2022
-
[37]
Rui Pan, Jipeng Zhang, Xingyuan Pan, Renjie Pi, Xiaoyu Wang, and Tong Zhang. 2024. Scalebio: Scalable bilevel optimization for llm data reweighting. arXiv preprint arXiv:2406.19976
2024 arXiv
-
[38]
Rafael Rafailov et al. 2023. https://arxiv.org/abs/2309.09341 Direct preference optimization: Your language model is secretly a reward model . arXiv preprint arXiv:2309.09341
2023 arXiv
-
[39]
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. 2019. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731
2019 arXiv
-
[40]
Ozan Sener and Vladlen Koltun. 2018. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31
2018
-
[41]
Shai Shalev-Shwartz and Yonatan Wexler. 2016. Minimizing the maximal loss: How and why. In International Conference on Machine Learning, pages 793--801. PMLR
2016
-
[42]
Shai Shalev-Shwartz et al. 2012. Online learning and online convex optimization. Foundations and Trends in Machine Learning , 4(2):107--194
2012
-
[43]
Sebastian U Stich. 2018. Local sgd converges fast and communicates little. arXiv preprint arXiv:1805.09767
2018 arXiv
-
[44]
Derek Tam, Mohit Bansal, and Colin Raffel. 2024. Merging by matching models in task parameter subspaces. Transactions on Machine Learning Research
2024
-
[45]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288
2023 arXiv
-
[46]
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Cl \'e mentine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944
2023 arXiv
-
[47]
Martin Weyssow, Aton Kamanda, and Houari Sahraoui. 2024. Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences. arXiv preprint arXiv:2403.09032
2024 arXiv
-
[48]
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasin...
2022
-
[49]
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2023. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694
2023 arXiv
-
[50]
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. 2024. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36
2024
-
[51]
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. 2024. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666
2024 arXiv
-
[52]
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824--5836
2020
-
[53]
P Xing, Hao Zhang, Joseph E
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena . P...
2023 arXiv
-
[54]
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911
2023 arXiv
-
[55]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[56]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.